<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[skai engineering blog - Medium]]></title>
        <description><![CDATA[Anything and Everything from Skai’s Engineering (formerly known as the Kenshoo Engineering Blog) - Medium]]></description>
        <link>https://medium.com/kenshoos-engineering-blog?source=rss----38970fe22728---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>skai engineering blog - Medium</title>
            <link>https://medium.com/kenshoos-engineering-blog?source=rss----38970fe22728---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:28:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/kenshoos-engineering-blog" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Copilot 101]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/copilot-101-5f3950b504f9?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/5f3950b504f9</guid>
            <category><![CDATA[copilot-programming]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[github-copilot]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai-code-assistant]]></category>
            <dc:creator><![CDATA[Yuval Dahari]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 12:31:01 GMT</pubDate>
            <atom:updated>2026-03-31T12:31:01.493Z</atom:updated>
            <content:encoded><![CDATA[<h3>Copilot 101: Choosing the Right Tool for the Job</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gcMWIgG6Z2LQ1IhVDUPWgA.png" /></figure><h3>💬 TL;DR</h3><ul><li>GitHub Copilot isn’t one tool, it’s three distinct surfaces (IDE, Web, CLI), each with different context sources and strengths</li><li>The difference between a mediocre and a great Copilot experience is usually which surface you choose, not how you write the prompt</li><li>The IDE plugin sees your local files, Copilot Web searches your entire committed codebase on GitHub, and the CLI works from your terminal and file system</li><li>For complex tasks, follow a progression: Understand → Plan → Execute -don’t jump straight to code generation</li><li>The biggest unlock comes when your team builds custom tools on top of Copilot, turning generic AI into org-specific superpowers</li></ul><h3>🎯 Why This Matters</h3><p>Most developers install GitHub Copilot’s IDE extension and stop there. That’s convenient, but it also means leaving significant productivity gains on the table.</p><p>GitHub Copilot is not a single tool. It’s a family of AI-powered surfaces, each living in a different environment, with different context sources, strengths, latency characteristics, and limitations.</p><p>Understanding <em>why</em> each Copilot environment was designed the way it was, and <em>what context it actually has access to</em>, is the key to getting dramatically better results. In many cases, the difference between a mediocre Copilot experience and a great one is not the prompt itself, but choosing the right Copilot surface before writing it.</p><p>This guide will help you make a smarter decision every time you reach for Copilot.</p><h3>🧠 The Core Principle: Latency vs. Context Depth</h3><p>There’s an important architectural tradeoff behind every Copilot surface: you can’t have infinite context and instant responses at the same time.</p><ul><li>IDE completions must be extremely fast → narrow, highly relevant local context</li><li>Web Copilot can search entire repositories → but responses take longer</li><li>CLI Copilot focuses only on shell context → precise, executable commands</li></ul><p>Understanding this tradeoff explains <em>why different Copilot surfaces exist at all.</em></p><h3>📝 Quick Decision Guide</h3><p>Before reaching for Copilot, ask yourself: “Where does the context I need actually live?”</p><p>Writing or editing code right now?</p><ul><li>Small change, single file → Inline completion / Inline Chat</li><li>Need to understand before changing → IDE Ask mode</li><li>Planning a multi-file change → IDE Plan mode</li><li>Executing a validated plan → IDE Agent mode</li></ul><p>Understanding or investigating code?</p><ul><li>Code is on your machine, file is open → IDE Chat</li><li>Code is committed on GitHub → Copilot Web Chat</li><li>Cross-repo or org-wide question → Copilot Web Chat</li></ul><p>Reviewing a Pull Request?</p><ul><li>Quick question about a diff → Copilot Web (PR tab)</li><li>Ongoing review conversation → @copilot in PR comments</li></ul><p>Creating a PR or fixing an issue autonomously?</p><ul><li>Well-scoped issue, clear description → Coding Agent (assign to Copilot)</li><li>Follow up on agent’s PR → @copilot in PR comments</li></ul><p>Terminal, DevOps, or shell work?</p><ul><li>Already in the terminal → Copilot CLI</li><li>Quick one-shot command → copilot -p &quot;...&quot;</li><li>Multi-step terminal workflow → CLI interactive mode</li><li>CI build is failing → “Explain error” button on GitHub</li></ul><p>Using org-specific tools?</p><ul><li>Check what’s available → type / in Copilot Chat</li></ul><h3>🧩 Copilot IDE Plugin</h3><h3>❓What It Is</h3><p>The Copilot IDE plugin is the most widely used Copilot surface. It runs as an extension inside VS Code or JetBrains IDEs and gives you three surfaces:</p><ul><li>✨ Inline Completions: Ghost text that completes your code as you type</li><li>💬 Inline Chat: Select code, press Ctrl+I / Cmd+I, transform it in place</li><li>🧠 Copilot Chat: Full conversational interface in the sidebar</li></ul><p>Inside the Copilot Chat panel, you can switch between three interaction modes, and picking the right one makes a huge difference in output quality.</p><p>To install: Search for “GitHub Copilot” in your IDE’s extension or plugins marketplace.</p><h3>🧠 The Three Chat Modes - And When to Use Each</h3><p>Think of the three modes as a progression:</p><p>Ask (understand the problem) → Plan (structure the solution) → Agent (execute the changes)</p><p>Skipping straight to Agent mode, especially for non-trivial tasks, is one of the most common mistakes developers make.</p><h4>❓ Ask Mode - Your Thinking Partner</h4><p>Ask mode is for exploration, understanding, and discovery. It doesn’t touch your code. It helps you <em>think</em> before you act.</p><p>Use it when you need to:</p><ul><li>Understand unfamiliar code or a new codebase</li><li>Brainstorm approaches to a problem</li><li>Debug by reasoning through root causes</li><li>Learn concepts or explore tradeoffs</li><li>Get options before committing to an approach</li></ul><p>Good prompts:</p><blockquote>&quot;Explain how this API works.&quot;<br>&quot;What are 3 different approaches to implement a new DAG operator here?&quot;<br>&quot;What could be causing this race condition? Walk me through the possible scenarios.&quot;<br>&quot;I&#39;m new to this codebase. What is the most important thing to understand before touching this module?&quot;</blockquote><p>💡 Tip: Use Ask mode to build <em>mental context</em> before generating any code. The better your understanding, the better your subsequent prompts will be.</p><h4>🧭 Plan Mode - Your Architect</h4><p>Plan mode is for medium and large tasks where jumping straight into code generation is risky.</p><p>The model will break the problem into concrete steps, identify which files will be affected, highlight risks and edge cases, suggest a test strategy, and propose rollout or migration sequencing.</p><p>Use it when you’re about to:</p><ul><li>Refactor across multiple files</li><li>Migrate from one pattern, library, or architecture to another</li><li>Implement a feature that touches several layers of the stack</li><li>Make a schema or API change with downstream impact</li></ul><p>Good prompts:</p><blockquote>&quot;Plan step-by-step implementation for this user story.&quot;<br>&quot;Design a safe rollout strategy for this feature change.&quot;<br>&quot;What&#39;s the safest way to add support with a new publisher for this service?&quot;</blockquote><p>💡 Tip: Plan mode output is a <em>proposal</em>, not a commitment. Edit it, challenge it, and refine it before executing.</p><h4>🤖 Agent Mode - Your Executor</h4><p>Agent mode is where Copilot actually makes changes. It writes code, modifies files, applies refactors, runs terminal commands (with your approval), and implements what you’ve decided.</p><p>Use it when you have:</p><ul><li>A clear, scoped task</li><li>Already validated your approach (ideally via Plan mode)</li><li>Defined what “done” looks like</li></ul><p>Good prompts:</p><blockquote>&quot;Implement step 2 from the plan above.&quot;<br>&quot;Apply this refactor to these services.&quot;<br>&quot;Generate unit tests for the edge cases we identified.&quot;<br>&quot;Add input validation to all the endpoints we discussed.&quot;</blockquote><p>⚠️ Important: Always review Agent mode changes before accepting. It’s powerful, but it acts on your codebase directly. Use small, incremental steps rather than one giant prompt.</p><h3>🏗️ The Recommended Workflow for Complex Tasks</h3><p>For non-trivial engineering work, jumping straight to Agent mode often leads to wrong abstractions, missed edge cases, refactor churn, and hidden conflicts with the existing architecture.</p><p>Here’s a more reliable, battle-tested workflow:</p><ol><li>[Ask mode] Understand the problem space</li><li>[Plan mode] Generate a full implementation plan</li><li>[Ask mode] 🔍 Ask a DIFFERENT model to REVIEW the plan - <em>“What are the risks in this plan? What did we miss?”</em></li><li>[You] Refine the plan manually based on the review</li><li>[Agent mode] Implement step by step, one step at a time</li><li>[You] Validate each step with tests and code review</li></ol><h4>The Multi-Model Review Pro Tip</h4><p>One of the most powerful patterns in AI-assisted development is separating the planner from the executor:</p><ol><li>Use a strong reasoning model (like Claude Opus) in Plan mode to generate a detailed implementation strategy</li><li>Switch to a different model or a fresh chat in Ask mode, and ask it to <em>review the plan</em>:</li></ol><blockquote>&quot;Here&#39;s an implementation plan I was given. What are the risks? What edge cases were missed? Are there better approaches?&quot;</blockquote><p>3. Refine based on the review</p><p>4. Use a faster or cheaper model in Agent mode to execute the validated plan, step by step</p><p>This works because each model has different “blind spots.” Having one model review another’s plan catches mistakes that the original model would likely miss if you just asked it to review itself.</p><p>💡 Pro tip: You can even open two separate Copilot Chat sessions in parallel by sending one or many of them to Copilot Web.</p><h3>⚡ Inline Completions: The Fast Lane</h3><p>Beyond the chat panel, Copilot’s ghost text completions are your constant companion while writing code.</p><p>How to get the most out of them:</p><ul><li>Write a descriptive comment above the function during development, completions use comments as intent signals</li><li>Name variables and functions clearly, naming is context. keywordId generates better completions than kwId</li><li>Keep related files open in tabs, open tabs are part of the context window</li></ul><h3>💬 Inline Chat: Transform Without Leaving Your Flow</h3><p>Ctrl+I / Cmd+I opens Inline Chat, a quick, targeted way to transform selected code without switching to the sidebar.</p><p>Best uses:</p><blockquote>Select a function → Ctrl+I / Cmd+I → &quot;Add documentation to this&quot;<br>Select a block → Ctrl+I / Cmd+I → &quot;Simplify this and reduce nesting&quot;<br>Select a method → Ctrl+I / Cmd+I → &quot;Make this handle null inputs safely&quot;<br>Select a test → Ctrl+I / Cmd+I → &quot;Add edge case coverage for empty instance&quot;</blockquote><p>💡 Tip: Inline Chat is scoped to your selection , keep selections small and focused for best results. Don’t select 300 lines and ask for a refactor.</p><h3>🔧 Slash Commands: Built-in Shortcuts</h3><p>Inside Copilot Chat, there are built-in slash commands that give the model a focused intent:</p><ul><li>/explain - Explains the selected code in plain English</li><li>/fix - Diagnoses and suggests fixes for a problem</li><li>/tests - Generates unit tests for the selected code</li><li>/doc - Generates documentation for the selected code</li></ul><p>These aren’t magic, they’re focused prompts. But they’re a great starting point when you know exactly what you need.</p><h3>🤖 Sub-Agents and Internal Tools: Type / to Discover</h3><p>In addition to the built-in slash commands, your organization can set up custom agents, specialized Copilot extensions built for specific workflows.</p><p>Simply type / in the Copilot Chat panel and browse the list of available agents and commands.</p><p>💡 Before starting any large task, always check what agents are available. Using one can save you hours or even days.</p><h3>🔌 MCP: Model Context Protocol Integrations</h3><p>Copilot can be extended with MCP (Model Context Protocol) integrations, plugins that give the model access to external context it wouldn’t otherwise have. MCP servers can provide access to project management tools like Jira, other repositories, monitoring systems, and more.</p><p>Before tackling a complex problem, ask yourself: <em>Is there an MCP integration that could give Copilot better context here?</em></p><p>💡 Tip: MCP integrations are configured in your IDE settings. Check with your team, someone may have already set up the integrations you need.</p><h3>🧱 Context Is Everything: How Copilot Sees Your Code</h3><p>The IDE plugin’s context window includes:</p><ol><li>Your current file - always included automatically (you can remove it)</li><li>Open tabs - Copilot uses other open files as context</li><li>Your cursor position - Nearby code gets more weight</li><li>Selected code - In inline chat, the selection is the primary context</li><li>.github/copilot-instructions.md- repository-level instructions (if present)</li><li>Your chat history - Previous messages in the current session</li></ol><p>What it does NOT see:</p><ul><li>Files you haven’t opened or attached</li><li>Git history or blame information</li><li>External documentation or APIs (unless MCP-connected)</li><li>Other developers’ local changes</li></ul><p>💡 Tip: Open the files most relevant to your task before asking a question. Context management is an active skill, not a passive one.</p><h3>💡 IDE Quick-Reference</h3><ul><li>✅ Start with Ask mode to understand before generating</li><li>✅ Use Plan mode for anything touching more than 2 files</li><li>✅ Iterate in small, validated steps</li><li>✅ Keep relevant files open as an active context</li><li>✅ Write descriptive comments while working , they dramatically improve completions</li><li>✅ Use a second model or a fresh session to review your plan</li><li>✅ Type / to explore available agents before starting large tasks</li><li>❌ Don’t send 500 lines to the inline chat at once</li><li>❌ Don’t blindly accept large multi-file refactors</li></ul><h3>🌐 Copilot Web</h3><h3>❓What It Is</h3><p>Copilot Web is the AI layer built directly into github.com. No installation. No setup. It’s just there, waiting.</p><p>You’ll find it as the chat icon in the upper-right corner of any GitHub page, inside Pull Requests as a conversation tab or review assistant, inside Issues discussing the problem with full codebase awareness, inside file views asking questions about specific files, and as the Coding Agent, an autonomous AI that takes an issue and opens a PR.</p><p>Unlike the IDE plugin (which lives in your local editor and sees your local files), Copilot Web has read access to your entire committed codebase on GitHub, and can even reason across multiple repos at once. That’s its superpower.</p><h3>🔧 How It Works Under the Hood</h3><p>Before diving into features, it’s worth understanding how Copilot Web <em>retrieves</em> context, because it directly affects what kinds of questions it can and can’t answer well.</p><p>The retrieval pipeline, simplified:</p><ol><li>Classifies your intent: Code question? PR review? Issue investigation?</li><li>Runs code search: both lexical (exact symbol/string matching) and semantic (vector embedding similarity - “what does this <em>mean</em>?”)</li><li>Enriches with page context: if you’re on a PR, it pulls the diff, if on an issue, it pulls the thread</li><li>Pulls repo-level instructions: .github/copilot-instructions.md is always included if it exists</li><li>Assembles and ranks: the most relevant pieces are packed into the LLM’s context window</li><li>Generates and streams the response back to you</li></ol><p>The vector/embedding piece explained: Every function, class, and block in your committed codebase is converted into a mathematical “fingerprint” of its <em>meaning</em>. When you ask “how does authentication work?”, your question is also fingerprinted, and the system finds code whose fingerprint is <em>similar</em>, even if none of the exact words match. This is why you can ask in plain English and get relevant results, rather than needing to know the exact function name.</p><p>💡 This is what makes Copilot Web uniquely powerful for cross-file and cross-repo discovery, something your IDE plugin cannot do.</p><h3>🚀 Key Capabilities</h3><h4>1. 💬 Copilot Chat on GitHub</h4><p>The floating chat panel available on any GitHub page. Use it for understanding an unfamiliar repo without cloning it, asking architecture questions across the entire codebase, or investigating a bug based on an issue or error message.</p><p>Good prompts:</p><blockquote>&quot;Give me a high-level summary of this repository&#39;s architecture.&quot;<br>&quot;Trace the complete flow of a payment transaction, from API endpoint to database write.&quot;<br>&quot;How is error handling done across this codebase? Is there a consistent pattern?&quot;<br>&quot;What are the main external dependencies and what is each one used for?&quot;</blockquote><h4>2. 🔍 PR Review: Ask Copilot About a Diff</h4><p>Inside any Pull Request, open the Copilot tab to have a full conversation about the changes.</p><p>Good prompts inside a PR:</p><blockquote>&quot;Summarize the changes in this PR. What is the intent, and what files are affected?&quot;<br>&quot;Are there any potential bugs or race conditions in this diff?&quot;<br>&quot;Does this PR introduce any breaking changes for API consumers?&quot;<br>&quot;What test coverage is missing? What edge cases weren&#39;t addressed?&quot;<br> &quot;Review this PR&quot;</blockquote><p>💡 Tip: You can also tag @copilot directly in a PR comment thread, just like you&#39;d mention a teammate. Copilot will respond as a participant in the review.</p><h4>3. 🤖 @copilot: Mention It Anywhere in a PR</h4><p>Once Copilot’s Coding Agent creates a PR, you can continue the conversation by tagging @copilot in any comment on that PR. This is one of the most underused features.</p><blockquote>@copilot, the tests are failing. Can you investigate and fix? <br>@copilot, add input validation to the new endpoint you created.<br>@copilot, write unit tests for this PR.</blockquote><p>When Copilot sees your mention, it opens a new agent session with the full context of the PR, shows 👀 on your comment to signal it’s working, makes the requested changes, and pushes new commits to the branch.</p><p>⚠️ Note: @copilot only responds to users with write access to the repository. Also: batch your comments rather than tagging @copilot multiple times one by one, each mention starts a new agent session, and grouped feedback is handled more efficiently.</p><h4>4. 🤖 The Coding Agent: Assign an Issue, Get a PR</h4><p>This is where Copilot Web shifts from <em>assistant</em> to <em>autonomous collaborator</em>.</p><p>The workflow:</p><ol><li>You write a clear issue description</li><li>Assign the issue to Copilot (in the Assignees panel)</li><li>Copilot reads the issue + analyzes the codebase</li><li>Copilot creates a branch</li><li>Copilot writes code, runs tests, and iterates</li><li>Copilot opens a draft PR and requests your review</li><li>You review, leave comments (or @copilot), and request review from another team member</li></ol><p>How to trigger it: Open any GitHub Issue → click Assignees in the right sidebar → select Copilot → optionally add specific instructions → click assign. Copilot starts immediately.</p><p>You can also trigger it from the Copilot Agents panel on GitHub, Copilot Chat on GitHub, your IDE’s Chat panel in Agent mode, the GitHub CLI, or any tool with GitHub MCP server support.</p><p>What makes a great Coding Agent task:</p><p>✅ Good candidates: Well-scoped bug fixes, dependency version upgrades, adding tests to existing code, simple refactors with clear rules</p><p>❌ Poor candidates: Vague open-ended features, tasks requiring UI/UX judgment, tasks involving secrets or credentials, tasks spanning 10+ tightly coupled services</p><p>💡 Pro tip: You can select a custom agent when assigning (using the agent picker in the assign dialog). This lets you use org-specific agents that have specialized instructions, tools, or access patterns instead of the default Copilot behavior.</p><p>⚠️ GitHub Actions note: By default, workflows don’t auto-run on Copilot’s commits (for security). You’ll see an “Approve and run workflows” button in the PR. Review the changes before approving.</p><h4>5. 🔭 Cross-Repo Discovery: The Biggest Advantage Over the IDE</h4><p>This is what Copilot Web does that nothing else does. Because it uses GitHub’s server-side code search infrastructure, it can reason across multiple repositories in your organization, the full commit history and file graph, issues, PR discussions, and code comments.</p><p>Real-world examples:</p><blockquote>&quot;Compare the API response schema in the backend repo with the TypeScript types in the frontend repo. Are there any mismatches?&quot;<br>&quot;If I change the UserProfile interface in repo A, which other repos in this org would be affected?&quot;<br>&quot;How is this specific error code handled across all our microservices?&quot;<br>&quot;Which team owns the payment processing logic -find the relevant repos and files.&quot;<br>&quot;Show me every place in the org where we&#39;re using a deprecated version of this API.&quot;</blockquote><p>💡 Tip: Anchor your cross-repo questions to specific repositories explicitly. Instead of “find where X is used”, say “find where X is used across org/repo-a, org/repo-b, and org/repo-c.&quot; The more specific, the more precise the retrieval.</p><h4>6. 🧩 Custom Agents: Pre-Loaded Expertise for Your Organization</h4><p>Copilot Web supports custom agents that your team can configure. These agents live in .github/agents/ in your repositories and can be specialized for a specific domain, pre-loaded with org-specific instructions and constraints, and shared across your organization.</p><p>When assigning a Coding Agent task or using the Agents panel, the agent picker lets you select from the default Copilot agent, org-level custom agents, or repo-level custom agents.</p><p>💡 Using a pre-loaded custom agent that knows your architecture conventions, coding standards, and workflows produces better results than the generic Copilot agent dramatically, because it already has the context you’d otherwise have to explain from scratch every time.</p><h4>7. 🌌 Copilot Spaces: Curated Context for Teams</h4><p>Copilot Spaces is the answer to a common problem: <em>“How do I give Copilot the institutional knowledge that lives in our heads, our wikis, and our private channels?”</em></p><p>A Space is a curated context collection that you build once and share with your whole team. It can contain linked GitHub repositories (kept in sync automatically), uploaded documentation, specs, and architecture docs, custom instructions and team standards, and links to external references.</p><p>Once a Space is set up, every conversation within it is grounded in your team’s specific context, not generic code knowledge.</p><p>Real use cases:</p><ul><li>Onboarding a new engineer: Architecture overview, coding standards, key repo links, FAQ</li><li>Accessibility standards: Component library, team checklist, WCAG references</li><li>Migration project: Old and new architecture docs, migration run-book, affected repos</li></ul><p>How to create one: Go to github.com/copilot/spaces → click New Space → add repositories, files, and custom instructions → share with your team or org.</p><p>💡 Tip: Think of a Space as a “briefing packet” you give Copilot before every conversation. The better your Space, the less you need to explain in every prompt.</p><h3>⚠️ The Real Limitations: What Copilot Web Cannot Do</h3><p>🚧 It Only Sees Committed, Pushed Code</p><ul><li>❌ Your local changes: Copilot Web cannot see these</li><li>❌ Uncommitted edits: Copilot Web cannot see these</li><li>⚠️ Commits on unmerged PRs: Limited visibility</li><li>✅ Merged, committed code: Fully visible</li></ul><p>💡 Practical tip: Before a deep Copilot Web investigation, push your latest branch and reference it explicitly: “In the branch feature/auth-refactor, I changed X, how does this interact with the existing session logic?&quot;</p><p>🕐 Index Freshness Lag:<br>The code search index is not real-time. After you push a commit, there can be a delay (minutes to hours for large repos) before it appears in search results.</p><p>📏 Context Window Ceiling:<br>Even with server-side retrieval, there’s a limit to how much can fit in the LLM’s context window. Copilot’s retrieval layer ranks and truncates results to fit, which means very large files may only be partially included, cross-repo queries may give some repos less coverage, and the model gets the <em>most relevant</em> chunks, not <em>all</em> the code.</p><p>🤔 It Can’t Run Your Code:<br>Copilot Web is read-and-reason only. It can read your code, explain it, suggest changes, and (via the Coding Agent) write and commit code, but it cannot execute your tests, call your APIs, connect to your database, or reproduce a bug in a live environment.</p><h3>💡 Copilot Web Quick-Reference</h3><ul><li>✅ Anchor questions to specific repos, files, or PRs</li><li>✅ Use cross-repo questions for impact analysis and architectural discovery</li><li>✅ Write clear, specific issues before assigning to the Coding Agent</li><li>✅ Batch @copilot comments in PRs, don&#39;t leave them one by one</li><li>✅ Set up a Space for your team’s repeated context</li><li>✅ Set up .github/copilot-instructions.md in your repos</li><li>✅ Push your latest work before running a Copilot Web investigation</li><li>❌ Don’t paste your entire codebase into a chat message</li><li>❌ Don’t expect Copilot Web to know about uncommitted work</li><li>❌ Don’t assign vague, open-ended issues</li></ul><h3>🖥️ Copilot CLI</h3><h3>❓What It Is</h3><p>Copilot CLI is a terminal AI coding agent that lives entirely inside your command line. No browser tab. No IDE sidebar. Just you, your shell, and an AI that understands your project, your file system, and your tools, all without leaving the terminal.</p><h3>🚨 Important: The old GitHub Copilot is retired</h3><p>If you’ve used Copilot in the CLI before, you likely used gh copilot suggest and gh copilot explain. This extension was retired in October 2025. It no longer receives updates and will eventually stop working entirely.</p><p>The replacement is a completely new, standalone Copilot CLI, not an extension, but its own dedicated tool. It’s a fundamentally different product:</p><p>Before (old gh copilot): An extension you installed via gh extension install. Single-shot, ask, get answer. Could only suggest a command or explain one. No file access, no plan mode, fixed model, no GitHub integration.</p><p>Now (new Copilot CLI): Standalone install (npm/brew/winget). Full conversational multi-turn sessions. Reads and edits your project files (with approval). Built-in Plan mode (Shift+Tab). Choose your model. Create PRs, manage issues, and interact with GitHub directly.</p><h3>📦 Installing the New Copilot CLI</h3><p>Prerequisites: An active GitHub Copilot subscription (any plan).</p><p>macOS / Linux - using <strong>Homebrew</strong> (recommended):</p><p>brew install copilot-cli</p><p>All Platforms - using <strong>npm</strong>:</p><p>npm install -g @github/copilot</p><p>Windows - using <strong>WinGet</strong>:</p><p>winget install GitHub.Copilot</p><p>You can also download executables directly from the copilot-cli releases page on GitHub.</p><h3>🎛️ Two Interfaces: Interactive and Programmatic</h3><p>Interactive Mode - the primary experience:</p><p>copilot</p><p>Drops you into a full conversational session. Switch between ask mode and plan mode with Shift+Tab. Type / to see all available slash commands.</p><p>Programmatic Mode - for scripts and one-shot use:</p><blockquote>copilot -p &quot;How can I apply a commit from another branch?&quot;<br>copilot -p &quot;Show me this week&#39;s commits and summarize them&quot; --allow-tool=&#39;shell(git)&#39;</blockquote><h3>🚀 Core Shortcuts</h3><ul><li>Shift+Tab - Toggle between ask mode and plan mode</li><li>@filename - Mention a specific file to add it to the context</li><li>/ - Show all available slash commands</li><li>? - Open tabbed help</li><li>Esc - Cancel the current operation</li><li>Ctrl+C - Cancel if thinking, clear input, or exit</li><li>Ctrl+L - Clear the screen</li><li>↑ / ↓ - Navigate command history</li><li>/help - Full list of commands and shortcuts</li></ul><h3>🧠 What Context Does Copilot CLI Actually Have?</h3><p>What it sees:</p><ul><li>✅ Files in your project directory</li><li>✅ Shell commands it executes (with your approval)</li><li>✅ Git history, branches, diffs</li><li>✅ GitHub.com (PRs, issues, repos)</li><li>✅ Your shell type and OS</li><li>✅ Your current working directory</li></ul><p>What it doesn’t see:</p><ul><li>❌ Uncommitted changes (unless it reads the file explicitly)</li><li>❌ Environment variables (unless you explicitly show them)</li><li>❌ Processes outside your shell</li><li>❌ Your databases or APIs</li><li>❌ Previous sessions (context doesn’t persist)</li></ul><h3>🔋 What It Can Do: The Full Capability Set</h3><p>Read and Understand Your Project:</p><blockquote>&quot;Give me an overview of this project.&quot;<br>&quot;What are the main entry points of this application?&quot;<br>&quot;Explain how the authentication module works.&quot;</blockquote><p>Write and Edit Code:</p><blockquote>&quot;Add input validation to the createUser endpoint.&quot;<br>&quot;Refactor the PaymentService to use async/await throughout.&quot;<br>&quot;Fix the null pointer exception in UserRepository.java.&quot;</blockquote><p>Run Shell Commands:</p><blockquote>&quot;Find all files larger than 50MB in this directory.&quot;<br>&quot;Show the last 10 commits that touched the auth module.&quot;<br>&quot;Run the tests for the payment service only.&quot;</blockquote><p>Git Operations:</p><blockquote>&quot;Show me what changed between the last two releases.&quot;<br>&quot;Squash my last 4 commits into one.&quot;<br>&quot;Find all commits that touched the pricing logic.&quot;</blockquote><p>GitHub Integration: Create PRs and Issues:</p><blockquote>&quot;Create a PR for this branch with a clear description.&quot;<br>&quot;Open an issue for the bug we just found in the auth flow.&quot;<br>&quot;List the open PRs I need to review.&quot;</blockquote><p>Special Modes:</p><ul><li>Plan Mode (Shift+Tab): Copilot asks clarifying questions before writing any code, builds a structured plan, and lets you review before execution</li><li>Research Mode (/research): Deep dives on technical topics</li><li>Fleet Mode (/fleet): Multiple independent sub-tasks run in parallel (e.g., &quot;Add JSDoc to all exported functions, add missing tests, and update the README&quot;)</li><li>Autopilot Mode: For trusted, hands-free tasks: copilot -p &quot;Fix all compilation errors and run tests&quot; --allow-tool=&#39;shell&#39;</li></ul><h3>🔧 When to Reach for Copilot CLI</h3><ul><li>Already in the terminal: Zero context switch, stay in your flow</li><li>Git surgery (rebase, cherry-pick, bisect): Deep git knowledge, proposes exact commands</li><li>Understanding a new codebase quickly: “Give me an overview” → instant project summary</li><li>Creating a PR without opening a browser: Full GitHub integration from the command line</li><li>Complex shell pipelines: Describe what you want, get the exact command</li><li>Debugging a build failure: Paste the error, get a diagnosis, and fix</li><li>Broad refactor with many steps: Plan mode + Fleet mode for coordinated execution</li><li>Scripting or automating workflows: Programmatic mode with -p flag</li></ul><h3>💡 Real-World Workflows</h3><p>First Day on a New Project:</p><blockquote>&gt; Give me an overview of this project<br>&gt; What are the main dependencies and what does each one do?<br>&gt; Where should I start if I want to add a new REST endpoint?<br>&gt; Show me an example of how tests are written in this codebase</blockquote><p>Build Failure Triage:</p><blockquote>&gt; [paste build error output] What&#39;s causing this, and how do I fix it?<br>&gt; Try the fix and run the tests again<br>&gt; The tests are passing now - create a commit with a good message</blockquote><p>PR Creation From Terminal:</p><blockquote>&gt; I&#39;ve finished the feature on this branch. Create a PR against main with a clear title and description based on the changes.<br>&gt; Add a note about the database migration that needs to run before deploying.</blockquote><p>Quick Command Generation:</p><blockquote>copilot -p &quot;find all files larger than 100MB recursively&quot;<br>copilot -p &quot;undo my last 3 commits but keep changes staged&quot;<br>copilot -p &quot;find which process is using port 8080 and kill it&quot;</blockquote><h3>💡 CLI Quick-Reference</h3><ul><li>✅ Use @filename to explicitly include relevant files</li><li>✅ Switch to Plan mode (Shift+Tab) before complex multi-step tasks</li><li>✅ Use /research for deep technical topics</li><li>✅ Use /fleet for multiple parallel, independent tasks</li><li>✅ Create PRs and issues from the terminal</li><li>✅ Use programmatic mode (-p flag) to integrate Copilot into scripts</li><li>✅ Start a session in your project root</li><li>❌ Don’t paste entire log files, filter to the relevant error first</li><li>❌ Don’t skip the approval step</li><li>❌ Don’t assume it knows your env vars, secrets, or running processes</li></ul><h3>✅ What Happens When a Team Actually Builds on Top of Copilot</h3><p>The previous sections covered what Copilot offers out of the box. But the real magic happens when an engineering organization goes a step further, when teams stop treating Copilot as a generic assistant and start building custom tools on top of it.</p><p>This is the story of what happened at one company when they did exactly that.</p><h3>🚩 It Started With One Pain Point</h3><p>It didn’t begin with a grand AI strategy.<br> It began the way most good engineering tools begin: someone was frustrated.</p><p>I was spending <em>days</em> manually hunting down feature flag references across backend and frontend repositories after a flag had been fully rolled out. Every occurrence looked the same. The logic was predictable. It was exactly the kind of repetitive, error-prone work that screams for automation.</p><p>So I built a Copilot agent for it.</p><p>A simple slash command you could invoke from your IDE that would scan the codebase, locate every reference to a feature flag, and generate the removal changes automatically. You still reviewed the diff, merged, deployed, verified, and only then deleted the flag, in that order, always.</p><p>The first version wasn’t perfect.<br> But the latest version reduced rollout cleanup time from <strong>days to under 10 minutes.</strong></p><p>That was the moment something clicked.<br>If one painful workflow could be automated this way, what else could?<br>So we built another agent.</p><h3>🔁 The Pattern Emerged</h3><p>Something subtle started to change.</p><p>Engineers stopped thinking about AI as a chat interface.<br> They started treating it as a <strong>tooling layer</strong>.</p><p>Every recurring friction point — debugging, migrations, dashboards, retrospectives, cross-repo changes — suddenly looked like a candidate for automation.</p><p>Not every experiment succeeded. Some agents became outdated. Others were too brittle or solved problems that weren’t painful enough. But the successful ones shared the same DNA:</p><ul><li>The task was repetitive</li><li>The logic was structured</li><li>The cognitive load was high</li><li>The creative input was low</li><li>And most importantly — engineers <em>actively avoided doing it</em></li></ul><p>Once that pattern became clear, new tools started appearing across the engineering lifecycle.</p><h3>🩺 The Tools That Followed</h3><p>Teams began turning everyday operational pain into reusable AI workflows:</p><ul><li>🚨 Using AI to triage CI failures before involving DevOps</li><li>🩺 Creating “living codebase assistants” for complex systems</li><li>📊 Investigating production logs directly from the IDE</li><li>📈 Generating monitoring dashboards automatically</li><li>🔄 Opening bulk PRs across dozens of repositories</li><li>🔍 Adding AI reasoning to database migration reviews</li><li>🕰 Generating sprint retrospective reports autonomously</li><li>🥒 Investigating failing tests by suggesting suspicious commits</li></ul><p>What started as a single frustration slowly became a new engineering reflex:</p><p><strong>If it hurts often enough -AI it!</strong></p><h3>📈 The Pattern</h3><p>Looking at these tools together, a pattern emerges:</p><p>Individual wins → Shared tools → Organizational success</p><p>The best Copilot setups don’t stop at installing the IDE extension. They:</p><ol><li>Start with real problems: A failing test. A tedious migration cleanup. Hours lost in monitoring dashboards.</li><li>Build reusable tools: Encode the solution as a slash command or agent skill anyone can invoke.</li><li>Raise the floor for everyone: Instruction files and Spaces mean even the most junior engineer gets the team’s accumulated knowledge automatically.</li></ol><p>The internal toolkit repository became the place where all of this lived. Before starting any large task, engineers would type / in Copilot Chat and see what was already available. Someone on the team may have already solved your problem.</p><h3>🧠 The Myths vs. Reality: Across All Surfaces</h3><p>After everything in this guide, a handful of misconceptions show up everywhere. These are worth addressing directly:</p><p><strong>“One great prompt is all I need.”:<br></strong>Great output comes from iteration, not a single perfect prompt. Start with understanding, plan, execute in steps, and validate. Complex tasks need a progression, not a magic incantation.</p><p><strong>“More context = better results.”:<br></strong>False everywhere. Focused, relevant context consistently beats volume. 15 irrelevant open tabs hurt more than they help. A focused question with specific file references beats a wall of pasted code.</p><p><strong>“I can walk away and let it run.”:<br></strong>Copilot is a collaborator, not an autopilot. The best results come from staying in the loop, redirecting, validating each step, and being the pilot.</p><p><strong>“It knows more than it shows.”:<br></strong>Copilot only knows what’s in its context window. If it’s not there: open files, committed code, explicit prompts, Copilot doesn’t know it. The IDE doesn’t know about files you haven’t opened. The Web doesn’t know about uncommitted changes. The CLI doesn’t know your environment variables.</p><p><strong>“Copilot will catch its own mistakes.”:<br></strong>It won’t, not reliably. That’s your job. Review the output. Run the tests. Be the pilot.</p><p><strong>“Copilot Web and Copilot IDE Chat are the same thing.”:<br></strong>Fundamentally different context sources. Web uses server-side code search across your full repo. IDE uses the local editor state and opens files. They complement each other, they don’t duplicate each other.</p><p><strong>“Plan mode will generate executable code.”:<br></strong>Plan mode generates <em>plans</em>, structured text proposals. Use Agent mode (with the plan as context) to execute.</p><h3>🤝 They’re Complementary, Not Competing</h3><p>A common mistake is treating the three surfaces as alternatives, as if you pick one and stick with it.</p><p>The most effective engineers use all three, often in the same task:</p><ol><li>🌐 Copilot Web: “Which files are related to this bug? Trace the flow.”</li><li>🧠 IDE Ask mode: Open the relevant files, understand the code locally</li><li>🧠 IDE Plan mode: Plan the fix with a strong reasoning model</li><li>🧠 IDE Ask mode: Ask a different model to review the plan</li><li>🧠 IDE Agent mode: Implement the fix step by step</li><li>🖥️ Copilot CLI: Create the PR from the terminal with a clear description</li><li>🌐 @copilot: Ask Copilot to review its own PR and add missing tests</li></ol><p>Each surface hands off to the next naturally. The context flows with you.</p><h3>🏁 Where to Go From Here</h3><ul><li>Get started with the IDE plugin: Install from the Marketplace in your IDE</li><li>Explore Copilot Web: Open any repo on GitHub and click the 💬 chat icon</li><li>Install the new Copilot CLI: brew install copilot-cli or npm install -g @github/copilot</li><li>Discover your org’s internal agents: Type / in Copilot Chat</li><li>Set up a Copilot Space for your team: Go to github.com/copilot/spaces</li><li>Build your own agent: Start with one pain point, encode the solution, and share it</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f3950b504f9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/copilot-101-5f3950b504f9">Copilot 101</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Good Infrastructure + GenAI: How we migrated 300+ E2E tests to Playwright in 1Q]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/good-infrastructure-genai-how-we-migrated-300-e2e-tests-to-playwright-in-1q-b6afe90d750f?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/b6afe90d750f</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[automation-testing]]></category>
            <category><![CDATA[playwrights]]></category>
            <category><![CDATA[ui-testing]]></category>
            <category><![CDATA[e2e-testing]]></category>
            <dc:creator><![CDATA[Liron Arad]]></dc:creator>
            <pubDate>Wed, 28 Jan 2026 21:28:03 GMT</pubDate>
            <atom:updated>2026-01-28T21:28:01.643Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3SpLJD28UeUhV-H-hAPaCg.jpeg" /></figure><p>In the world of fast-paced SaaS development, your testing suite can either be an anchor or a sail. For the past few years, we relied on <strong>Testim.io</strong> for our UI test automation. It served us well during our growth phase, but as our engineering organization matured, we found ourselves hitting the limits of “low-code” solutions and facing the looming shadow of vendor lock-in.</p><p>With our contract set to expire in December, we faced a choice: renew the status quo or rebuild our entire E2E stack under an aggressive deadline. We chose the latter. This is the story of how we migrated our entire E2E testing infrastructure to <strong>Playwright</strong> — and how we did it in less than three months.</p><h3>Why Playwright? The Strategic Shift</h3><p>Playwright is a modern, open-source testing framework designed for web applications. It enables developers to write reliable, fast, and scalable tests across major browsers — including Chromium, Firefox, and WebKit — with a unified API. By providing a code-first, developer-centric experience with intuitive APIs and built-in tools for debugging, Playwright makes it easier to integrate E2E testing directly into the development workflow.</p><p>Our decision to adopt Playwright wasn’t just about saving on licensing costs or technical features. It was driven by a fundamental organizational shift: the closing of our dedicated QA department.</p><p>Today, our developers are the owners of E2E quality. In this new reality, we needed a tool that matched their workflow. Playwright emerged as the winner because it provides a code-first, developer-centric experience. Developers feel more comfortable writing tests in the same IDEs and languages they use for feature code, turning testing from a specialized “extra” task into a core part of the development lifecycle.</p><h3>Technical Foundation: From Dockers to Labs</h3><p>Internally, we operate a set of managed, persistent test environments we call Automation Labs.</p><p>A major part of this migration was moving away from the overhead of managing local Dockerized environments for every test run. While we still support Docker for isolated component testing (using a simple custom annotation that informs the test runner to construct such a dedicated local environment), the heavy lifting for our E2E suite moved to <strong>Automation Labs</strong>.</p><p>These Labs are persistent, managed testing environments that provide a critical advantage: Production-Grade Fidelity. Unlike ephemeral Dockers that often lack the complexity of a live system, Labs provide:</p><ul><li><strong>Production Mirroring:</strong> They allow teams to run tests against environments that accurately replicate production configurations, enabling the reliable reproduction of production-like scenarios and edge cases.</li><li><strong>Consistent Data State:</strong> Unlike ephemeral Dockers that start from scratch, Labs maintain a baseline that is refreshed on a predictable schedule, ensuring a stable foundation for every run.</li><li><strong>Scalability:</strong> They enable the execution of massive test suites across many environments in parallel without the local machine resource drain or performance bottlenecks.</li><li><strong>Build Time Optimization</strong>: The shift to Playwright significantly boosted performance, reducing our overall <strong>build time by 60%</strong>.</li></ul><h3>Our Execution Strategy: From Code to Release</h3><p>A key part of our migration was defining a clear execution lifecycle. We categorized our runs into three tiers to balance developer speed with the stability required for a daily release cycle:</p><h3>1. PR-Level Checks: Fast &amp; Isolated</h3><p>For every Pull Request, we utilize <strong>Dockerized environments</strong>.</p><ul><li><strong>The Goal</strong>: Immediate feedback and total isolation. Every developer gets a fresh instance that is destroyed after the check, preventing any cross-contamination.</li><li><strong>The Data</strong>: We use automated data preparation tools to spin up specific data sets within the container, allowing tests to run in a controlled environment within approximately 30 minutes.</li></ul><h3>Nightly Runs: Preparing the Daily Build</h3><p>The core of our release engine is the Nightly run, executed against the <strong>Nightly Automation Lab</strong>.</p><ul><li><strong>The Goal</strong>: This serves as our primary validation for the <strong>Daily Build</strong>. By running the full suite overnight, we identify cross-team regressions that PR checks might miss, ensuring the codebase is stable enough for the next day’s release.</li><li><strong>The Environment</strong>: A persistent lab that is <strong>refreshed daily</strong> to ensure tests are always running against the most current data baseline.</li></ul><h3>3. Release Sanity: The Final Sign-off</h3><p>Before a version is officially shipped, we conduct final sanity checks in the <strong>Pre-Prod Automation Lab</strong>.</p><ul><li><strong>The Goal</strong>: Production-grade validation.</li><li><strong>The Data</strong>: This is our most robust environment, refreshed weekly, combining automated test data with synced live production data to simulate real-world scale and complexity for a final “Go/No-Go” decision.</li></ul><h3>Solving the Data Problem: The DB Builder</h3><p>One of the biggest friction points in E2E testing is data setup. To solve this, we utilized our <strong>DB Builder</strong> tool. Instead of manually clicking through the UI to create an account and some business entities within it before every test, we use <strong>Cucumber feature files</strong> to generate test data prior to test execution.</p><p>This process runs periodically on the environments before the E2E tests begin, ensuring the required state is “baked in” and ready for the browser.</p><p>By using the @PlaywrightDbBuilder annotation, developers can define their required state in Gherkin:</p><pre>Given I create the following agencies:<br> | name | user feature |<br> | TestAgency | FEATURE_X |<br>And I create the following profiles:<br> | profile id | in agency | name |<br> | 300001 | TestAgency | TestProfile |</pre><p>This tool automatically injects the data into the Lab’s database. This ensures our tests are <strong>atomic and idempotent</strong>, as the DB Builder handles the heavy lifting of backend entity creation before the Playwright script ever opens a browser.</p><p><strong>Navigating Engineering Challenges</strong></p><p>The migration wasn’t without friction. We solved three core technical hurdles:</p><ul><li><strong>Concurrency &amp; Isolation</strong>: To prevent teams from impacting each other, we implemented strict <strong>test isolation</strong>. Every suite runs in a sandboxed state.</li><li><strong>The “False Negative” Trap</strong>: We eliminated phantom failures from previous data remnants by enforcing <strong>strict environment cleanup</strong> for every run.</li><li><strong>DB Schema Coupling</strong>: Since the <strong>DB Builder</strong> generates data directly into the schema, refactors can break tests. We now ensure tests always run against a <strong>matching DB Builder version</strong> to keep data in sync.</li></ul><h3>The AI Secret Sauce: Agentic Workflows with MCP</h3><p>Now that we ensured stable environments and deterministic data, we’re left with the actual migration of hundreds of scenarios. Rewriting 310 tests manually would have been impossible in one quarter. We leveraged <strong>Agentic AI</strong> — specifically using <strong>multiple MCP (Model Context Protocol) servers </strong>giving it access to existing resources<strong> </strong>— to turn our AI agents from mere chat-bots into active participants in our migration.</p><p>The real “magic” happened through a structured instruction set that allowed the AI agent to navigate our internal repositories autonomously:</p><ol><li><strong>Architecture-Aware Generation:</strong> We created a shared library of internal utilities that interact with our specific UI components. We instructed the agents to use these helpers exclusively. This not only ensured the generated code followed our modular architecture but also enforced “Clean Code” principles across all 310 tests.</li><li><strong>Context Discovery:</strong> Using MCP servers, the agents could “pull” documentation for specific UI modules on demand, ensuring they stayed updated with our latest API changes without needing a full model retrain.</li><li><strong>Cross-Repository Data Prep:</strong> The agents weren’t limited to the UI repo. If a test needed a specific DB state, they were prompted to use the GitHub MCP server to read the <strong>DB Builder</strong> docs in our backend repository, generate the necessary Cucumber feature file, and submit it as a PR automatically.</li><li><strong>Local Execution &amp; Self-Healing:</strong> Agents ran the generated tests against the lab using Playwright MCP. If a test failed, the agent analyzed the failure, fixed the script, and produced a summary mapping each fix for future training.<br>To maintain high architectural standards, we implemented a <strong>“human-in-the-loop”</strong> checkpoint here — Instead of letting the agent commit code blindly, developers reviewed the proposed “self-healing” fixes. <br>This step ensured that the agent didn’t just make the test <em>pass</em> (e.g. by adding brittle selectors), but fixed it in a way that aligned with our long-term maintenance goals.</li></ol><h3>The Results: By the Numbers</h3><p>The migration was a total success. By our mid-December deadline:</p><ul><li><strong>Total Tests Migrated:</strong> ~310</li><li><strong>Completion Rate:</strong> 99.2%</li><li><strong>Squads Involved:</strong> 22</li><li><strong>Infrastructure:</strong> 100% of new UI tests now run on Automation Labs using Playwright.</li></ul><h3>Lessons Learned</h3><p>The success of this migration wasn’t just about the tool — it was about <strong>infrastructure readiness</strong>. While it’s tempting to call the process “frictionless,” the reality is that no migration at this scale is. We faced real engineering hurdles, from ensuring test isolation in shared labs to keeping data generation in sync with evolving database schemas.</p><p>However, by leaning into these challenges rather than ignoring them, we reached our main takeaway: <strong>Focus on the infrastructure, and let AI do the repetitive work.</strong></p><p>By providing a stable foundation through our dedicated labs and data generation tools, we created an environment where a well-crafted AI agent could thrive. This combination turned a task that would have required endless hours of grunt work into a feasible engineering undertaking. Today, our UI testing (just like all other testing layers, e.g. unit and component testing) is no longer a separate task performed in a third-party UI; it is version-controlled code and a core part of how we ship software.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6afe90d750f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/good-infrastructure-genai-how-we-migrated-300-e2e-tests-to-playwright-in-1q-b6afe90d750f">Good Infrastructure + GenAI: How we migrated 300+ E2E tests to Playwright in 1Q</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The GenAI Acceleration Day Playbook]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/the-genai-acceleration-day-playbook-8662ebe7bd19?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/8662ebe7bd19</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[change-management]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai-adoption]]></category>
            <dc:creator><![CDATA[Yarin Slivka]]></dc:creator>
            <pubDate>Thu, 08 Jan 2026 19:16:16 GMT</pubDate>
            <atom:updated>2026-01-08T19:16:15.344Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong><em>How We Pushed GenAI Adoption to 95% in 30 Days</em></strong></h3><p>Most companies treat GenAI adoption like a distinct project. They buy Cursor, Claude Code, and Copilot licenses, send a “Welcome” email, and hope for the best. The result? 20% usage and a massive pile of unused seats.</p><p>At Skai, we aimed for <strong>90%+ adoption</strong>. To get there, we didn’t plan an event. We spent 5 months building a foundation, and 30 days executing a sprint. We engineered a <strong>mindset shift</strong>.</p><p>Here is the exact playbook we used to transform 200+ engineers from consumers to creators.</p><h4>Phase 1: The Long Game (The Guild)</h4><p><strong>Timeline:</strong> 5 Months Pre-Event</p><p>You cannot sprint if you haven’t trained. You cannot transform an org without change agents. For 5 months, I led the <strong>GenAI Champions Guild</strong> — a strike team of 12 engineers (one from each development group). At this stage, we didn’t have a “GenAI Acceleration Day” on the calendar. We had a mandate to explore.</p><ul><li><strong>The Goal:</strong> Implement high-impact, low-effort opportunities in the SDLC.</li><li><strong>The Method:</strong> We met weekly to learn, identify, select, plan, implement, and test.</li><li><strong>The Result:</strong> By November, we had a vetted arsenal of tools. When leadership gave us the green light for a Cross R&amp;D event, we didn’t have to scramble for content. We just had to scale what we already knew worked.</li></ul><h4>Phase 2: The “Hype” Campaign (November Rollout)</h4><p><strong>Timeline:</strong> 30 Days Pre-Event</p><p>In early November, we got the green light: <strong>“You have one day in December to stop the org and train everyone.”</strong> We shifted gears from “Exploration” to “Production.” We had 4 weeks to create FOMO.</p><ul><li><strong>Cadence:</strong> We rolled out <strong>1 new tool per week </strong>from our list.</li><li><strong>The Content:</strong> Short video demos focusing purely on the “Wow Moment.”</li><li><strong>The Timing</strong>: Scheduled precisely when devs were active on Slack.</li></ul><p>By the time the event arrived, the hype was organic. The Guild had primed the pump for months; the marketing campaign just opened the valve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Mo5LMqNVmQgjVt_KyKNSw.png" /></figure><h4>Phase 3: The Acceleration Day Architecture (Relevance is King)</h4><p>Generic training fails. A Frontend dev doesn’t care about Backend boilerplate. We split the 12 Champions into 3 Specific Tracks:</p><ol><li><strong>Frontend</strong> (React/Wizards)</li><li><strong>Backend</strong> (Java/API)</li><li><strong>Data &amp; Infra</strong> (Python/Pipeline)</li></ol><h4>Phase 4: The Acceleration Day Bootcamp (AM Session)</h4><p>We didn’t let the Champions “wing it.” We designed a strict pedagogical format that every track had to follow. We iterated on this for weeks:</p><ul><li><strong>Part 1: Foundation:</strong> Planning to solve a track-specific problem.</li><li><strong>Part 2: The Unlock:</strong> Implementing the plan using the <strong>relevant tools</strong>.</li><li><strong>Part 3: The “Wow” Workflow:</strong> Combining both to deliver an end-to-end solution.</li></ul><h4>Phase 5: The “Real Work” Mandate (PM Session)</h4><p>This is where most workshops fail: they focus on “Toy Projects.” We banned toy projects.</p><ul><li><strong>The Prep:</strong> We met with the Team Leads forum beforehand.</li><li><strong>The Requirement:</strong> Every single attendee had to arrive with a <strong>real Jira ticket</strong> assigned from their sprint/backlog, while applying the methodology that was learned in the bootcamp.</li><li><strong>The Friction Removal:</strong> We ran “Office Hours” before the event to solve setup/IDE issues for 50+ devs, ensuring day-of productivity was 100%.</li><li><strong>Prizes: </strong>Always good for building extra engagement.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bx2RqSQ9hHJSlqi_GiWxlQ.png" /></figure><h4>The Result: The Shift</h4><p>The day wasn’t just “fun.”</p><p><strong>Velocity:</strong> We closed 37<strong> Jira tickets</strong> on the spot.</p><p><strong>Adoption:</strong></p><ul><li>We hit <strong>95% monthly active usage</strong> post-event.</li><li>Weekly usage increased by 110%.</li></ul><p>But the real win is the shift in identity. Our engineers stopped asking “What can AI do for me?” and started asking “What agent can I build to kill this manual task?”</p><p>The event was just the spark; the 5 months of Guild preparation provided the fuel.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8662ebe7bd19" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/the-genai-acceleration-day-playbook-8662ebe7bd19">The GenAI Acceleration Day Playbook</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Replication: The Backbone of Scalable, Efficient Data Platforms]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/data-replication-the-backbone-of-scalable-efficient-data-platforms-e110ed4184f7?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/e110ed4184f7</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[real-time-streaming-data]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[replication]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Ziv Fridfertig]]></dc:creator>
            <pubDate>Tue, 26 Nov 2024 13:02:15 GMT</pubDate>
            <atom:updated>2024-11-26T13:02:14.064Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>The modern market offers optimized databases for almost every specific need, be it time series data, transactions, geo-spatial, analytics and more. This fascinating collection of technologies offers high flexibility for organizations to pick the right tool for the right job, based on usage patterns, performance, TCO, resiliency, complexity etc. This often results in maintaining multiple databases for different types of data and access.</p><p>This decentralization brings another challenge in data driven organizations: viewing data across different databases! Such unified views might be required for client needs, business needs, billing purposes, and more.</p><p>A traditional and efficient way of addressing this need in big data systems (Tera/Peta/Zeta byte scales) is using the Data Lake / Data Warehouse model: import “operational data” from the various databases into a single robust analytics database where you can leverage heavy compute power for fast processing of all your data in one place.</p><p>Such “import” processes are commonly called “Data Pipelines”, and are usually being implemented by using one of these approaches:</p><ol><li><strong>Replication</strong><br>Copying data from A to B with no / minimal changes along the way, often using out-of-the-box Export/Import/<a href="https://en.wikipedia.org/wiki/Change_data_capture">CDC</a> tools. <br>Usually being used for [near] real time arrival to the destination database.</li><li><strong>Batch copy</strong><br>Copying data from A to B while applying data enrichment, transformation and/or normalization capabilities. <br>Usually based on mediating software and hardware (<a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL</a> engines) for utilizing Export/Import tools and data processing/modeling programs.</li></ol><p>We heavily use <strong>replication</strong> to copy hundreds of tables from our (hundreds of) operational systems into the Data Lake in near real time.</p><p>In this article we’ll cover the replication-based Data Pipeline we built.</p><h3>Building a Replication Pipeline</h3><p>Here are some key questions one must consider before defining and building a replication pipeline:</p><h4>Scale</h4><ul><li>What are the data volumes you expect to transfer (accounting for growth potential)?</li><li>How much segmentation they have in terms of data sources, structure types, schemas, data change rates, etc.</li></ul><p>Some solutions can be sufficient for lower scales but completely non-viable for larger scales, so collecting these numbers in advance is critical for finding the right Data Pipeline implementation.</p><h4>Performance</h4><ul><li>What is the expected SLA for replicating data from A to B?</li><li>What mitigations/compromises are valid for unusual peaks in traffic?</li></ul><p>This section should help choose the right tools and needed observability over each potential bottleneck in the pipeline.</p><h4>Schema Evolution</h4><ul><li>How will schema changes in the source system be applied to the destination system?</li><li>What kind of limitations, if any, are made necessary due to the existence of a replication pipeline?</li></ul><p>It’s possible that some changes, e.g. changing a table’s primary key, will compromise the pipeline integrity. Avoid such cases by documenting the destructive scenarios and add validations to protect your pipeline from breaking changes.</p><h4>Data Trust</h4><ul><li>How reliable is the pipeline?</li><li>What would help users trust the replicated data is identical to the source?</li></ul><p>Your data may be incorrectly transformed, concatenated or missing in any landing point in the flow. Plan in advance for utilizing observability / <a href="https://en.wikipedia.org/wiki/Data_quality">DQA</a> tools, of which there are many in the market.</p><h4>Stability</h4><ul><li>How do we maximize pipeline stability?</li></ul><p>Data Pipelines are often composed of many different components, each with its own programming language, packaging, versioning and dependencies. Use proper CI/CD tools and testing techniques (e.g. unit, integration, E2E tests) for pushing verified version upgrades to each component.</p><h4>Provisioning</h4><ul><li>How and when will the pipeline components be updated?</li></ul><p>When the source databases are modified, the pipeline might have to change too. Provisioning all the components in a single flow will give you better control. Tie the rollout of schema changes to your pipeline provisioning, so that the two never go out of sync.</p><h4>Observability</h4><ul><li>How do you plan to detect and address data correctness issues?</li></ul><p>Data Lakes are known for their offline / batch processing, deceiving us to think they are not as mission-critical as other systems are. In Data Driven organizations, Data Lakes become more and more critical over time and a costly system to maintain. It is critical to design efficient monitors and reports to cover client-side issues, infra issues and cost analysis.</p><h3>Data Pipelines in Skai</h3><p>With the questions above in mind, we started building our in-house pipelines to replicate data from source systems into <a href="https://www.snowflake.com/en/data-cloud/platform/">Snowflake</a> — our choice for a DataLake solution. Our pipeline adheres to the following set of requirements — which none of the existing off-the-shelf products we inspected could:</p><ul><li>Replicate data near real time.</li><li>Apply schema changes (in source database) automatically in the target database.</li><li>Support for hundreds of source databases (RDS / MYSQL), with gradual deployments and client specific customizations. Our pipeline must be able to handle inconsistent source schemas by intelligently unifying them into one destination schema.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/0*5XqwLp7D9sKmqgBX" /><figcaption>Unified Schema Data Lake: schema changes will roll out gradually to Source Databases 1..</figcaption></figure><h3>Implementation Highlights</h3><h4>Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3ADCL6VQJ25SR0zV" /><figcaption>Replication based Data Pipeline</figcaption></figure><ul><li><strong>Kafka</strong> is the main vehicle through which data is transferred from the source to the destination. The fully managed <a href="https://www.confluent.io/lp/confluent-cloud/"><strong>Confluent Cloud</strong></a> solution saves us the effort of managing and maintaining Kafka clusters ourselves.</li><li><a href="https://kafka.apache.org/documentation/#connect"><strong>Kafka-Connect</strong></a><strong> </strong>on top of <strong>AWS EKS</strong> <strong>clusters</strong> runs containers loading data into Kafka and reading data out of Kafka:<br>- <a href="https://debezium.io/documentation/reference/stable/architecture.html"><strong>Debezium</strong></a><strong> connectors</strong> replicate CDC events out of MySQL / MySQL RDS instances into Kafka and keep the replication lean and efficient with AVRO + Schema Registry.<br>- <a href="https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-kafka"><strong>Snowpipe Streaming</strong></a><strong> connectors</strong> replicate data from Kafka into staging tables in Snowflake.</li><li>Lastly, <a href="https://docs.snowflake.com/en/user-guide/tasks-intro"><strong>Snowflake Tasks</strong></a> Merge / Append staged data into result tables.</li></ul><p>Additionally:</p><ul><li>A custom <strong>microservice </strong>(read <a href="https://medium.com/kenshoos-engineering-blog/microcosm-yes-kenshoo-has-also-built-a-paas-17b0ec4c2680">here</a> about Skai’s self-serve microservice platform)<strong> </strong>is used for processing incoming provisioning requests (e.g. a new source database was created; An existing source database schema has changed) and reflecting them via changes in Github-hosted config files, representing the “desired state” of the pipeline.</li><li><strong>Pulumi</strong> is used to translate these config files into instances of resources / components in the cloud, e.g. Kafka topics, connectors, K8s pods and more. This approach implements the concept of “<strong>Infrastructure as Code</strong>” which helps test, manage and audit such changes.</li><li><strong>AWS Lambdas</strong> are used to deploy / alter DB objects in Snowflake (tables, tasks, streams).</li></ul><h4>Provisioning Flow</h4><p>Since our source databases are extremely dynamic (new databases, schemas, tables and columns can be added or removed regularly), automatic provisioning was a key factor in our design. An orchestrated flow is being launched for each request arriving from the source system upon its deployment. Reminder, coupling the two (i.e. deployment of the source DB and provisioning of the pipeline components) ensures pipelines remain in sync.</p><p>This is the flow we implemented:</p><ul><li>Any source-system deployment triggers a provisioning request (via REST API) to the <strong>provisioning service</strong></li><li>The <strong>provisioning service</strong> analyzes the request and creates automated Pull Requests to alter the GitHub-hosted config files.</li><li><a href="https://hubot.github.com/"><strong>Hubot</strong></a> tracks automated PRs and merges them once they pass validations.</li><li>Each merge triggers a <strong>Jenkins</strong> job that runs <strong>Pulumi</strong>, which creates or modifies the actual Cloud resources.</li><li>Hubot polls the Jenkins job’s status and updates the <strong>provisioning service</strong> via API with its results (success/failure).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kyr3HIg6snUlVjB6" /><figcaption>Provisioning flow</figcaption></figure><h4>User Interface</h4><p>The “<strong>Data Mesh</strong>” approach focuses on making it easy to publish and access important data across your organization. To achieve that, we wanted to provide teams building the source systems an easy, self-serve method for replicating it to the widely-accessible Data Lake.</p><p>We build the pipeline based on a simple configuration file within each source “operational system” repo. All that one needs to do in order to get a table replicated reliably into the Data Lake, is to add it to this file (see image below). We added validations as an automated “PR check” to block any abuse / invalid configuration / human error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*z3Z4_EO79qvS_lrs" /></figure><h4>Monitoring</h4><p>We make heavy use of <a href="https://www.datadoghq.com/"><strong>DataDog</strong></a>, our monitoring platform, including its built-in<strong> </strong>integration with AWS services and Snowlake, to expose useful metrics and dashboards to monitor high level bottlenecks, issues and system load. For example, our dashboards expose:</p><ul><li>Long running queries on Snowflake</li><li>Connectors’ health on EKS</li><li>Detection of Kafka traffic peaks (ingress/egress)</li><li>Overall cloud costs of the pipeline</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IflBZOUGF73-JL5tORZM4A.png" /></figure><p>For the critical lag and consistency monitoring, we enriched the data in its destination with these values:</p><ul><li><em>Source_id</em> — The source DB identifier, keeping client/DB context</li><li><em>Source_timestamp</em> — Time of the change in the source DB, helps measuring Source→Snowflake lag</li><li><em>Destination_timestamp</em> — Ingestion time in target DB, helps measuring Kafka→Snowflake lag</li><li><em>Is_deleted</em> — Indication for deleted event, for history analysis purposes. <br>This field and the rest above are being masked to the Data Lake consumers with a view filtering rows with is_deleted=1</li></ul><p>With these additional columns in every replicated table, we’re able to send detailed lag measurements to DataDog and alert on any SLA breach.</p><h4>Code Reuse</h4><p>Replication into the Data Lake is just one of many pipelines we have in Skai. We saw an opportunity to build a generic provisioning platform, rather than investing resources in each pipeline individually. The benefits included improved code reuse, resource savings, and reduced development time. The main challenge, however, was ensuring that the independent pipelines remain loosely coupled to prevent interdependencies. To address this, we designed a simplified provisioning process, breaking it down into clear, manageable steps, enabling customization based on pipeline type while ensuring full isolation between them. The value of this investment became evident when we quickly onboarded the second pipeline onto this platform, which replicates operational data into our analytical database, <strong>SingleStore</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F7XiM1pPG8X0D4TZ" /><figcaption>Two independent Data Pipelines sharing the same provisioning, monitoring and orchestration building blocks</figcaption></figure><h3>Summary</h3><p>Replication has its costs and limitations, but the ROI can be significant when you really put your organization’s data to good use. There are many examples on how to benefit from having a centralized datastore. For instance, we recently delivered a new <a href="https://www.snowflake.com/en/customers/all-customers/case-study/skai/">AI-powered, client-facing feature to production in just two days</a>, thanks to our data pipelines making sure all necessary data is accessible and of high quality.</p><p>Skai’s success is also driven by its fast UI, handling billion-row aggregations in milliseconds, thanks to the real-time replication into SingleStore. This replaced the slower MySQL, which couldn’t meet today’s demands. Read how we <a href="https://www.confluent.io/blog/skai-cloud-migration-topic-reduction/">saved $400,000 annually</a> by making our pipeline architecture more cost-effective at scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e110ed4184f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/data-replication-the-backbone-of-scalable-efficient-data-platforms-e110ed4184f7">Data Replication: The Backbone of Scalable, Efficient Data Platforms</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpsLevel Reviews: how we helped our teams help themselves]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/opslevel-reviews-how-we-helped-our-teams-help-themselves-ee3d257045c6?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/ee3d257045c6</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[software-management-tools]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[it-operations]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <dc:creator><![CDATA[Shahaf Eran]]></dc:creator>
            <pubDate>Mon, 23 Sep 2024 17:34:53 GMT</pubDate>
            <atom:updated>2024-09-30T13:16:34.738Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Background</strong></p><p>R&amp;D teams at Skai are built to be independent and accountable. We expect our R&amp;D teams to fully own their resources, components and processes end-to-end. Aspects like performance, cost, reliability, security, and more, are all within the team’s responsibility. This is a rather demanding requirement. Among other things, it requires powerful tools and infrastructure, which we are committed to providing to our teams, but tools are not enough — teams need the focus and the capacity to actually use them.</p><p>As the teams’ ownership keeps growing, the number of concerns they need to handle/resolve/optimize for grows with it. This results in an inflated backlog with many different items of different sizes and impact. The backlog becomes a heterogenous mix of new features and items concerning security, cost, documentation, quality, performance, resilience, observability — the list goes on.</p><p>The result can easily overwhelm teams:</p><ul><li>How should these tasks be prioritized?</li><li>What’s “good enough”?</li><li>When should we address non-urgent (but important) issues?</li><li>How does one balance reduction of tech debt with ongoing development in each sprint?</li></ul><p>All of these are valid questions that we wanted to address to achieve better tech standards, solid processes and more focused teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/0*nUU_vRmnw33inHLv" /><figcaption>The many responsibilities of a typical R&amp;D team</figcaption></figure><p>When you think about it, we have probably all faced this challenge in our personal lives — there are many things we want to accomplish, responsibilities we must own, long term and short term goals that might conflict, and no clear guidance. And same as in ‘real life’, we needed a process that would allow (or force?) us to pause, reflect, and construct a plan to get better. Iteratively. Just like going to therapy — that periodic session when you’re forced to think about the important and not just the urgent, look beyond the daily noise that keeps us busy, identify your real needs and align your actions accordingly.</p><p><strong>Our Solution</strong></p><p>We started by helping the busy dev teams answer these few simple questions:</p><ul><li>What components do I own? (these can be microservices, schemas, processes, data pipelines etc.)</li><li>Do they meet the organization’s standards?</li><li>If not — what requires improvement?</li></ul><p>First, we wanted a single place where we can see all our components, manage their metadata and visualize their level of quality — in other words, a <em>service catalog</em>. We chose <a href="https://www.opslevel.com/">OpsLevel</a>.</p><p>From there, we started the work of mapping components to teams and setting measurable standards. We used OpsLevel <strong>tags</strong> to manually or automatically tag each component with the following metadata:</p><ul><li><em>Owner</em> — a (single!) owning team</li><li><a href="https://www.opslevel.com/resources/whats-in-a-tier-really"><em>Tier</em></a> — an indicator of how mission-critical the component is and how tolerant your business is to its potential failure</li><li><em>Maturity Level</em> — measures the maturity of the component. The level is determined by a set of automated <em>maturity checks</em> covering a wide range of concerns — such as test coverage, performance, cloud costs, security, scalability, documentation and more</li><li><em>Component Lifecycle</em> — the current stage of this component, from “developing” to “end-of-life”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xhIO7vBxlh_OasxY" /><figcaption>Our tier definitions. This will help us make sure more critical components are addressed first</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*htIQms7lumuVcSbQ" /><figcaption>Our Maturity Level definitions. As a starting point, we defined Gold as the minimum level for Tier 1 and Tier 2 components</figcaption></figure><p>The automatic maturity checks interface with the various systems in the organization such as Git (code management), Jira (task management), DataDog (monitoring), and more. Components can climb the maturity ladder (from Bronze to Gold) by passing all of the maturity level checks.</p><p>This was a big step forward — each R&amp;D team could now know which components they owned, whether they met the standard, and what they should do to improve. A self-serve tool that shows the big picture and helps diving into the details.</p><p>But that wasn’t enough. Providing teams with <em>visibility</em> doesn’t mean they’re going to use it and address the issues it surfaces. If we’ll go back to the real-life analogy, we now had a long list of action items and areas to focus on, but no clear process for prioritizing and handling them.</p><p><strong>Our Process</strong></p><p>The goal was to move from a reactive approach to a place where the team maps, plans and initiates proactively. We decided to set <em>recurring quarterly meetings</em> with every R&amp;D group (3–4 teams) where, together with team leads, architects and group managers, we’ll inspect the current state of each component, and focus on those that do not meet the standard (didn’t achieve Gold level). For those, the team is expected to come up with a solution, and assess:</p><ul><li>What’s the expected effort?</li><li>What do we expect to gain (e.g. 20% performance improvement)</li></ul><p>The output of these meetings is a prioritized and focused plan with effort estimations and deadlines, which is converted into action items we’ll visit in the next few sprints.</p><p>While the meetings themselves are essential, <em>the real benefit is the preparatory work</em> that precedes the meeting. This is where the team goes over its components, maps the weak points, looks for solutions, zooms in and zooms out. The existence of these meetings on the calendar makes this process a priority, so that it no longer gets pushed back over and over again, ad infinitum.</p><p><strong>Results</strong></p><p>The first and immediate impact was a slew of quick wins, made possible merely thanks to the visibility we gained and the sudden focus. Things like wasteful DB queries or missing documentation links could quickly be identified and fixed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VzjGLxCVHLQXO70M" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*LR6vY4jyNJRmvI7k" /><figcaption>Cost savings achieved through this process were shared with the broader R&amp;D group. These announcements motivate other teams to identify similar opportunities</figcaption></figure><p>After a few iterations, deeper and more meaningful changes climbed to the top of the backlog. For example, identifying weak points that are not trivial to measure automatically, finding the root cause of increasing client facing tickets/incidents within the team’s ownership, or even reconsidering the architecture of a specific solution to better support the growth in its demand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nc3uS4XChqoy56NZ" /><figcaption>An example from one of the teams where testing gaps were identified and addressed to reduce risk of incidents</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/0*V-mLOCfnifmBraRi" /><figcaption>A typical example of a team’s analysis of root causes for increase in client tickets, and the team’s plan to address it</figcaption></figure><p>In some cases, teams were even able to identify missing infrastructure and prioritize creation of shared components that would benefit multiple teams. The visibility brought in by these review sessions (which were open to all) made such opportunities easier to detect.</p><p><strong>To Summarize</strong></p><p>We looked for a way to set high standards and help our busy teams meet them, despite the ever-growing list of responsibilities and the demanding workload. We knew that measuring teams on delivery (only) while “asking” them to also take care of a dozen other aspects will not produce the right balance of priorities.</p><p>The solution was constructed 2 main pillars</p><ul><li>A tool to automatically measure and visualize our current status vs expected standards (OpsLevel)</li><li>A process that would guide teams to use this tool effectively and continuously</li></ul><p>Almost 2 years into this process, we’re still seeing a constant improvement in multiple KPIs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee3d257045c6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/opslevel-reviews-how-we-helped-our-teams-help-themselves-ee3d257045c6">OpsLevel Reviews: how we helped our teams help themselves</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Eliminating human error in legacy systems using Configuration as Code and Behavioral Tests]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/eliminating-human-error-in-legacy-systems-using-configuration-as-code-and-behavioral-tests-786e3511addd?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/786e3511addd</guid>
            <category><![CDATA[behave]]></category>
            <category><![CDATA[configuration-as-code]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[legacy-systems]]></category>
            <dc:creator><![CDATA[Tzach Zohar]]></dc:creator>
            <pubDate>Wed, 07 Feb 2024 13:32:27 GMT</pubDate>
            <atom:updated>2024-02-07T13:32:27.431Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, we’ll share a recent success story of migrating a legacy DB-based configuration store to Git. This transition made it safer and easier to introduce changes to this critical developer-generated, often-changed configuration, by auditing it, applying a set of rigorous automated tests to it, and no longer allowing human error to enter the equation.</p><h3>Legacy</h3><p>The system in question is an internal web scraping system, which underwent a drastic change in how it’s used: while originally built to support usage by analysts to produce <em>internal</em> data on a quarterly basis, we decided to employ this useful capability to produce critical <em>client-facing</em> data in short cycles and with high throughput.</p><p>The legacy system, while not robust enough for these new demands, was luckily rather flexible: it allowed users to define a “<strong>Source</strong>” — a complex JSON object describing the page that should be scraped, the way to obtain it, “selectors’’ to extract specific fields, and some formatting and normalization rules. So new Sources could be added to scrape new sites with no code changes. However, these Source definitions were rather sensitive (mistakes would render them invalid) and far from static, as the pages they describe can change often, requiring a change in the Source configuration. Another reason for Sources to change is the not-uncommon addition of a new field selector.</p><p>These Source configurations were stored in a simple model (2–3 tables) in a relational database. The records in these tables were modified via a homegrown user interface providing simple CRUD operations over this model, which gradually became less and less usable, driving developers (with DB WRITE access) to the “bare hands” approach of using UPDATE commands executed directly over the DB.</p><figure><img alt="Configuration used to be updated either manually or via a legacy UI" src="https://cdn-images-1.medium.com/max/998/0*PSjxRJDSpaGT5Z4r" /><figcaption>Configuration used to be updated either manually or via a legacy UI, and read by the scraping system</figcaption></figure><p>As long as this system served its original internal purpose, this wasn’t a critical issue. Mistakes, if they happened, could be fixed once the analyst reviewing the data caught them. But once we started onboarding usages generating critical user-facing data daily, this approach couldn’t hold. We started seeing too many incidents of bad data (or no data) reaching our clients due to misconfiguration. It was clear that the process of adding or updating Sources needed way more guardrails — including robust automated tests applied before changes take effect. What should those guardrails be, and how can we introduce them without overhauling a large legacy system?</p><h3>Alternatives</h3><p>Once the issue of unintentional or erroneous configuration changes became apparent, we considered a few solutions:</p><ul><li><strong>Resurrect the UI</strong>: The old UI included some basic validations, and it made it much less likely for the wrong records to get updated. However, the codebase for the UI was old and hard to manage, making it difficult to add further necessary validations</li><li><strong>Move the configuration out of the DB</strong>: working directly over a relational database isn’t ideal: it doesn’t provide very good auditing, undo capabilities, diff views, etc.. And yet, moving the configuration out of the DB would require a rather large refactoring effort in the legacy system reading it</li><li><strong>Improve the Source data model</strong>: one cause for errors was the fact that the Source configuration data model was itself not optimal — as any model evolved over years tends to become. But changing the data model would be costly for the exact same reason the previous option was disqualified — it would incur significant changes to the legacy system</li></ul><p>These considerations led us to the following “requirements” from our ideal solution:</p><ol><li>Full auditing capabilities (who made what changes, when and why, plus easy rollback)</li><li>Full validation capabilities <em>before</em> applying a change</li><li>Low impact on the legacy system consuming these configurations</li></ol><h3>Git-based Solution</h3><p>All of these pointed to an almost-obvious solution: Source configuration should be <strong>source controlled</strong> (no pun intended). Using source control for <em>configuration</em> (on top being an absolute standard in managing codebases) has become ubiquitous in the relatively recent transition to “configuration as code” and GitOps approaches. As a company, we’ve already transitioned all of our provisioning configuration to code, so it’s only natural to apply it to this case too:</p><ul><li>We’ll create a new GitHub repository to store the Sources configuration (represented as yaml files)</li><li>We’ll block changes to the main branch and only allow changes through Pull Requests (as we do for all of our codebases)</li><li>We’ll build a Jenkins job that tests each pull request (upon creation or change) and validates the change</li><li>We’ll build another Jenkins job that “releases” any change merged to the main branch, by reading the yaml files, identifying changes, and updating the same DB where the Sources are currently stored</li></ul><p>In other words — we’ll build a CI/CD pipeline identical in concept to what we build for any system we deploy — where automated tests enable continuous delivery with no manual testing and low risk. The “release” would update the existing DB from which the legacy system reads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GjsQKptuDRHDs6d4" /><figcaption>Our git repository now becomes the source of truth for Source configuration, and its release job the only way to update Sources.</figcaption></figure><h3>Test Implementation</h3><p>Moving the JSON-based Source configurations to yaml files stored in Git was relatively easy. The tricky part was designing the tests: we want a fast, stable, reliable way to test a changed Source, which won’t be affected by external changes (e.g. the scraped website was modified). On the other hand, we do want some way of catching such external changes when they happen, as they are just as likely to break our code. We’ll need to have some minimal periodic “smoke” test for <em>all</em> Sources, as well as specific and robust tests for every <em>changed</em> Source.</p><p>To account for that, we ended up defining two separate test suites:</p><p><strong>Nightly Tests</strong>:</p><ul><li>A suite of real-world end-to-end tests that scrape a single page per Source, and validate the response’s <strong>structure</strong> — i.e. that all fields are present</li><li>This suite should catch errors arising mostly from changes to the scraped website, not configuration changes</li><li>This suite is executed on all Sources nightly, and failures trigger an alert to the owning team</li></ul><p><strong>Release Tests</strong>:</p><ul><li>A faster and more reliable suite of tests that use stubbed HTML pages in place of actual scraping, thus avoiding slowness and instability</li><li>For each Source, developers must provide such an HTML sample, as well as the detailed corresponding expected scraping result</li><li>The test will run the (real) scraping system while stubbing the HTML page, and verify parsing results (based on the Source configuration) match expectations</li><li>These tests are executed on every Pull Request for modified Sources only, providing the author of the change immediate feedback</li><li>These tests are also executed during release, i.e. just before updating the actual production DB with the new version of a modified Source configuration</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*WVL8K4eD5ALemloMqa8mhQ.png" /><figcaption>Our Jenkins jobs: release and pull-request running “release” suite; Nightly runs the “nightly” suite</figcaption></figure><p>This separation into two test suites — which differ in goal and impact — proved to provide a good balance between coverage and costs. Tests running on PRs are quick enough to provide immediate feedback and don’t break due to external changes; Tests running nightly run often enough to catch issues before they become incidents.</p><p>In practice — both suites are implemented using Python’s most popular behavioral test tool — <a href="https://behave.readthedocs.io/en/latest/">behave</a>, where each Source has a single Feature file with separate scenarios annotated by nightly and release tags. Here’s an abbreviated example of such a test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bbb486ade5ccf9041b32a3ee2f28fe83/href">https://medium.com/media/bbb486ade5ccf9041b32a3ee2f28fe83/href</a></iframe><p>Steps like I expect the following fields to be numeric were implemented once and can be easily reused. Such decoupling of the “what” from the “how” is one of the benefits of behavioral tests, and results in very readable and easy to write tests.</p><p>To run each suite separately, we’ve wrapped the behave command with simple invoke tasks that we can easily call from our Jenkins jobs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b12a8539eebce8f5392cc30088fdb44a/href">https://medium.com/media/b12a8539eebce8f5392cc30088fdb44a/href</a></iframe><h3>Results</h3><p>Transitioning from a quick (and risky) manual approach to the slower-but-steadier fully tested Git-based approach wasn’t without friction. For simple changes, the new approach felt like it added a lot of overhead, especially in the early days when the tests were not as stable.</p><p>However, after stabilizing the tests and getting used to the safety this approach provided, developers on the team could implement Source configuration changes much more freely and confidently. Incidents related to misconfiguration dropped significantly. The ability (in fact, requirement!) to test every change made these changes feel “normal”, like any other code change, which is the exact goal of the Configuration as Code approach. A few weeks’ effort made this burning issue into a thing of the past.</p><h3>Conclusions</h3><p>Our key takeaways from this process:</p><ul><li>We can benefit from Configuration as Code without having to make large-scale changes to legacy systems</li><li>It’s enough that Git becomes the <strong>source of truth</strong> for the configuration, without necessarily replacing the existing configuration store entirely</li><li>Separate failure scenarios justify separate and different testing approaches</li></ul><p>With these conclusions in mind, we’re constantly looking to eliminate the few remaining instances of manual configuration changes across our products.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=786e3511addd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/eliminating-human-error-in-legacy-systems-using-configuration-as-code-and-behavioral-tests-786e3511addd">Eliminating human error in legacy systems using Configuration as Code and Behavioral Tests</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we saved more than $1M with an in-house EBS auto-scaling mechanism]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/how-we-saved-more-than-1m-with-an-in-house-ebs-auto-scaling-mechanism-f1aa6707e3c3?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/f1aa6707e3c3</guid>
            <category><![CDATA[ebs-volume]]></category>
            <category><![CDATA[cloud-migration]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[iop]]></category>
            <dc:creator><![CDATA[Eli Vaknin]]></dc:creator>
            <pubDate>Wed, 20 Dec 2023 15:26:45 GMT</pubDate>
            <atom:updated>2026-07-27T10:51:04.816Z</atom:updated>
            <content:encoded><![CDATA[<p>We’ve recently migrated our decades-old monolith application, internally codenamed “KS”, from our own datacenter to AWS.</p><p>Each KS (of which there are hundreds) is composed of two EC2 servers: one for the database and the other for serving the front-end, with a total of 9 disks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*-CDUKbRN4z4TrdhbgsJ4Fg.jpeg" /><figcaption>KS EC2 servers within our infrastructure diagram</figcaption></figure><p>While in the datacenter we had our physical storage and we could allocate it as needed at no additional cost, on AWS we will pay more to get better disk <a href="https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics">performance characteristics</a> (<a href="https://blog.purestorage.com/purely-technical/an-analysis-of-io-size-modalities-on-pure-storage-flasharrays/#:~:text=Throughput%20is%20a%20measurement%20of,used%20together%20to%20describe%20performance.">IOPS and throughput</a>).</p><p>When we started the migration project, the initial suggestion was to go with IO2 for our database and other performance-critical disks. Since IO2 is very pricey, we thought we would start with the cheaper GP3 while allocating the maximum IOPS allowed, but we knew this too would eventually be a costly overkill for at least some of our servers. To mitigate this, we adjusted the values manually based on our monitoring system as soon as we started the rolling migration to the cloud. Soon enough we realized that the manual approach won’t scale, as we have around 550 instances, which means 4950 disks to adjust.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*4IAu6nOKzA14q5D6asCmhQ.jpeg" /><figcaption>One KS <a href="https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics">performance characteristics</a> dashboard</figcaption></figure><p>It was clear an automated solution was required to optimize our disks’ <a href="https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics">performance characteristics</a>.</p><h4>Home-grown vs. off-the-shelf</h4><p>We started by searching the market for an off-the-shelf solution to tune the IOPS and throughput automatically. There were a few players that started offering solutions for this problem, but they were in the early stages and didn’t meet our requirements, which are summarized below:</p><ul><li>Dynamically setting the value — the system needs to set the values (up or down) based on the actual usage of each disk</li><li>No need to replicate (and then maintain) more EBS volumes, as some of the off-the-shelf tools do</li><li>Accuracy — the system had to be as accurate as possible, as AWS allows only one change for EBS every 6 hours</li><li>Reliability — setting incorrect values can cause degradation of system performance, which we cannot allow for our clients</li><li>Controlling the frequency of changes — as AWS allows only one change every 6 hours, it was important for us to control the frequency of the changes: if we were to change the values too frequently, we would not be able to increase disk size when we need to</li><li>Tuning the sensitivity of the system — we wanted to have the ability to control the system manually (e.g. enforcing a lower IOPs/throughput limit) and put thresholds to limit the amount of configuration changes</li><li>Automated system — manual intervention should be as minimal as possible</li></ul><p>After exhausting the search for a solution that provides all of the above, we decided to build our own solution.</p><h4>Building the solution</h4><p>In light of the requirements described above, we started to design our solution.</p><p>First, we had to decide how and where this system would run. One option was Jenkins, which we’re as the orchestrator for many tasks in our organization. Still, we wanted the system to be faster and more responsive than the typical Jenkins job (with Jenkins’ queues and delays), and as we’re adopting Serverless solutions across the company, we’ve decided to use AWS Lambdas for running this solution.</p><p>The Lambdas will be triggered as a chain, each one focusing on a simple task. Although Step Functions might fit this requirement, we chose to stay with pure async Lambda invocations as we already had robust automation for creating and maintaining Lambdas that supported everything we needed.</p><p>Our KS instances in AWS are provisioned with <a href="https://www.pulumi.com/">Pulumi</a>, where we already have IOPS/throughput settings for each device in the Pulumi stack. That means the new system would have to bypass that value whenever it runs, which will cause drifts between the Pulumi state and the actual state in AWS.</p><p>To avoid such drifts, we decided to set this setting to ignore_changes. This solved another problem — as we’re dealing with a sensitive system we wanted to be able to set a hard low limit for devices, so the Pulumi stack setting will now serve as our low limit for each volume.</p><p>The implementation of the chain of Lambdas can be seen in the following diagram:</p><ol><li>A few Lambdas collect all the current information from various sources (AWS, DataDog, and Pulumi)</li><li>The next Lambda crunches all the data and calculates new values for each volume’s IOPS and t<a href="https://blog.purestorage.com/purely-technical/an-analysis-of-io-size-modalities-on-pure-storage-flasharrays/#:~:text=Throughput%20is%20a%20measurement%20of,used%20together%20to%20describe%20performance.">hroughput</a>. It wasn’t a simple calculation, so we guarded it with maximal thresholds for increasing or decreasing the values. This helped us avoid hitting that once-in-6-hours change limit imposed by AWS</li><li>The last Lambda sets the calculated values for the volumes that need to be changed in AWS and sends a metric to Datadog, providing visibility for changes made by this system</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U083VvChYsTDdUzH" /><figcaption>System Diagram</figcaption></figure><p>To help us detect and resolve issues in this new system, we’ve added a dashboard that visualizes the EBS volume changes that are triggered each day and the status of each change. We’re also monitoring Lambda executions and get alerted if any Lambda fails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6XNUE4CiSKZZbovjJ9_tg.png" /><figcaption>System’s DataDog dashboard</figcaption></figure><h4>Recent improvements and future plans</h4><p>The system has been running in production for 10 months now. One notable enhancement was supporting IO2 as well as GP3 — which we had to introduce once some specific instances outgrew the performance GP3 could provide. This required our system to become aware of the disk type, as these two types differ in how their IOPS and throughput values are set.</p><p>Along the way, we’ve also found that we rely on data that is not accurate: we query max reads and max writes in two separate queries and summarize those two, resulting in an inaccurate max IOPS value. To solve this, we’ve introduced a <em>custom</em> metric that is sent from the instance itself and combines the read and write data simultaneously.</p><p>In the future, we plan to migrate this solution to Step Functions so it will be more reliable and easier to manage.</p><h4>Conclusions</h4><p>The solution described above allowed us to successfully and automatically align our volumes’ settings via a Serverless system, running a tunable algorithm that monitors all EBS <a href="https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics">performance characteristics</a> from Datadog and aligns our volumes; it meets all of our requirements.</p><p>As mentioned above, the original plan was to provision the maximum IOPS (16K) of GP3 EBS type for our DB disks and another disk on the frontend, which we knew would be an overkill and would be too costly. Utilizing this system, we can optimize all our disks and reduce the cost dramatically.</p><p>To demonstrate the savings, we’ve calculated the average IOPS on the relevant disks and performed two calculations on the AWS Cost Calculator — one with the original “static” 16K IOPS setting and one with the actual average IOPS — and found out that we’re saving $93K a month, or a whopping <strong>$1.116M</strong> annually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*cKwiB6iLgvUS_PE6K3aZWg.png" /><figcaption>AWS cost calculation (average IOPS — 4785 against 16k, both at our average throughput of 138 for 1662 volumes) resulting in ~93K$ of savings monthly</figcaption></figure><p>Those savings make this project a clear ROI-positive effort: the cost of development combined with the AWS Lambda execution costs is a fraction of the savings.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1aa6707e3c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/how-we-saved-more-than-1m-with-an-in-house-ebs-auto-scaling-mechanism-f1aa6707e3c3">How we saved more than $1M with an in-house EBS auto-scaling mechanism</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to collaborate with Data Science teams on delivering features to production?]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/how-to-collaborate-with-data-science-teams-on-delivering-features-to-production-371bcc85be9d?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/371bcc85be9d</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Merav Aizenfeld-Green]]></dc:creator>
            <pubDate>Thu, 26 Jan 2023 13:13:33 GMT</pubDate>
            <atom:updated>2023-01-26T13:13:33.185Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>How to collaborate with Data Science teams on delivering features to production? 5 Tips for Dev Managers</strong></h3><p>The value of integrating Machine Learning algorithms into SaaS products has become much clearer in recent years, especially now with the surge of generative AI and its public availability through open APIs.</p><p>Efficient work between the development teams (who bring these capabilities to production) and the Data Science (DS) teams (who are usually the owners of these and other powerful ML/AI capabilities) is important now more than ever in order to bring value to real users in production.</p><p>Productizing DS-powered features can be challenging and even frustrating for development teams and their managers, as DS practices might seem to break common development workflows and best practices.</p><p>As a development manager, you might be tempted to treat the integration with a DS deliverable (trained model, draft algorithm, or pseudocode) just as you would any other integration with a component developed by another team: we’ll define the boundaries between the components, work on these components independently, and eventually integrate them. But this is not necessarily the right approach.</p><p>In reality, the DS deliverables — while extremely valuable — are often not a production-ready component we interact with, but rather a blueprint for building production code. This means that when the time comes and we start integrating it into the product, we’ll find ourselves going back and forth with the DS team until things are working. It is important to understand that the work of those two teams is not a relay race, it’s more like a dance: each time someone leads, and each participant has their solos. While dancing, you must be aware of both yourself and your partner’s challenges and be there to support each other. You both need to be at your best for this show to be a success.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*yT7SwulnMWezv31q" /></figure><p>Below are 5 tips for managing such projects with DS teams.</p><p><strong>1. Be involved in the DS work from day one</strong></p><p>Your engineers don’t need to be in the loop at this stage, but you — as their manager — must be there. Read the relevant design docs and ideas, and if there is no written material, ask for a meeting to understand the planned effort and its goal. Every DS work should have a well-defined goal! Make sure you understand what’s planned and how it achieves it.</p><p>Since DS projects might provide partial results or end up being entirely scrapped (e.g. if there’s simply no correlation where you hoped to find one), you need to get a good feel of how close we are to a solution. How do the DS engineers feel about the odds of success? Do you agree with their estimation? Where are we in the process — is it at the very beginning of the proof of concept or steps away from completion? Are we stumbling around in the dark or is there some theory we are trying to prove?</p><p>This early involvement will give you the critical background for having meaningful discussions and understanding concerns throughout the next phases of the project.</p><p><strong>2. Provide needed engineering support even before the development is officially started</strong></p><p>During data analysis and algorithm development, the DS team might face many engineering challenges that are not algorithm-related (e.g. loading the data from one store to another, accessing internal data APIs etc.). Make sure your team’s engineering expertise is utilized to unblock the critical path efforts in the DS team.</p><p>Assign a dedicated engineer to attend the DS team’s daily meetings and provide the needed support.</p><p>Having even one engineer involved in these early stages allows the DS team to focus on their strength and really make the process efficient. It will also introduce an additional team member to the DS work for a smoother transition later on.</p><p><strong>3. Don’t write even one line of production code until the DS work is summarized</strong></p><p>The summary doesn’t have to be final, but some clear result needs to be presented and agreed upon.</p><p>It doesn’t mean you cannot assign the lead engineer to start documenting, add notes, start discovery or even write an HLD (High-Level Design) whenever you want, but coding too early might tie you to implementations that might turn out to be incompatible with the eventual algorithmic solution.</p><p><strong>4. Involve the DS team in your development milestones</strong></p><p>Once development does start, have a DS representative in your kick-off meetings and any other milestone meeting. Share your HLD, open questions, and timelines with the DS team — it is critical for getting feedback/concerns as early as possible.</p><p>Make sure both teams are aligned on the following:</p><ul><li>All APIs between teams should be clearly defined and documented</li><li>Keep a clear separation between the algorithm delivered from DS and your implementation concerns — the algorithm should be decoupled and have clear boundaries and definition-of-done</li><li>Agree on a clear KPI for the algorithm’s quality, and monitor it. This is a challenging and critical step for understanding and reflecting on the value for ourselves and for every level in the organization.</li></ul><p><strong>5. Last but not least: Reaching production availability is the finish line for both teams!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*iXgPJMDI16HI62mD" /></figure><p>DS teams have their own cycles of trial and error, which might seem endless to delivery-oriented development teams. This is needed and beneficial for the research phase, but once development has begun, help the DS team remain focused on <em>delivery</em>.</p><p>Make sure the relevant checkpoints are being tracked and marked as done throughout the entire development process. Stop and provide mitigation if checkpoints are missed, and take decisions that will allow you to complete the lightest end-to-end flow first.</p><p>You as a development manager responsible for delivery must find a way to keep your development cycles and deliver features to production on time.</p><p><strong>Summary</strong></p><p>working with DS teams is all about collaboration — utilizing each team’s expertise while not forcing one team’s practices on the other. As a manager, it’s your responsibility to get the right people involved at the right time and facilitate effective communication between teams whose vocabularies might be slightly different. Meeting that challenge can produce quick delivery of valuable features for your users.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=371bcc85be9d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/how-to-collaborate-with-data-science-teams-on-delivering-features-to-production-371bcc85be9d">How to collaborate with Data Science teams on delivering features to production?</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“So Pick Me, Choose Me, Love Me” — A 4 Step Guide To Finding Your R&D Execution Metrics’ True North]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/so-pick-me-choose-me-love-me-a-4-step-guide-to-finding-your-r-d-execution-metrics-true-north-5b5cf948a832?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/5b5cf948a832</guid>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[agile-methodology]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Neta Ilany]]></dc:creator>
            <pubDate>Tue, 20 Dec 2022 13:16:48 GMT</pubDate>
            <atom:updated>2022-12-20T13:16:48.571Z</atom:updated>
            <content:encoded><![CDATA[<h3>“So Pick Me, Choose Me, Love Me” — A 4 Step Guide To Finding Your R&amp;D Execution Metrics’ True North</h3><p><em>Neta Ilany | R&amp;D Lead Program Manager at Skai</em></p><p>What’s the best metric to measure R&amp;D teams’ efficiency? I ❤️ this question! It opens the door to an important and mature discussion in the team. Here’s my POV on how to answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/0*o9ypKohLq_d1TjBa" /></figure><p>I’ll take the “classic” engineer approach here, and say — “it depends”. It depends on how “effective” translates into in your org. It depends on what is currently causing “noise” and disturbing the team. And it depends on what you’ll want to balance, so no aspect is compromised over the other.</p><h3><strong>WIIFM</strong></h3><p>Metrics are friends, not foes; they’re first and foremost for the team to consume, and only then for management to review. Part of taking yourself and your team to the next professional level, is establishing the ceremony of looking at these metrics and<em> have them tell the story rather than you</em>. Yes, as data driven as it gets.</p><p>Mindset in place, remember that this is a journey. It might be a bumpy ride, but the process is more valuable than reaching the destination. It raises important questions and discussions such as what’s important for the team, what the team feels is blocking them and what they want to improve. So take the time to get to the roots of it.</p><h3><strong>Finding True North Metrics</strong></h3><p><strong>Step1: You Need To Crystallize What “Being Effective” Means in Your Org</strong><br>Efficiency requires focus, and what to focus on should be first defined by the R&amp;D organization at a much higher level.</p><p>Examples of some focus points and their reflecting metric:</p><ul><li>How fast do we deliver? → Feature cycle time to production</li><li>How many features do we deliver? → Feature development vs production support</li><li>Quality of deliverables → Escaped Bugs, Time to Resolve (TTR)</li><li>Scope maturity → Unplanned tasks ratio in sprint</li><li>Blocker reduction → measure specific pain points such as test environment creation or support tickets per component</li></ul><p><strong>Step 2: Identify Your Pain Points<br></strong>The art of selecting the right metrics relies on 2 pillars: identifying the pain and being able to measure it with reliable data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Vwgt3bojrzdFHZAo" /></figure><p>Each R&amp;D org is unique in its culture, but at the end of the day we face common problems, and can use common solutions. Here is our pain matrix. Some pains have more than one root cause, and more than one metric that can point out the trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*3w8iuid3-whIbCMx1SDuAw.png" /></figure><p>Once you have your pain matrix laid out, it’s easier for the team to start the discussion around which pain they experience, what hurts the most, and what can move the needle in terms of team’s execution. <br>Each team will probably want to choose a different set of pains. It’s important to use more than one metric to ensure balance, and avoid overcorrection.</p><p>At Skai we find it useful to balance the following 3 focus points, so neither one is compromised, and all teams are on top of their <em>throughput</em>.</p><ol><li>Feature ratio vs production support/tech investment</li><li>Feature cycle time</li><li>QA metrics</li></ol><p><strong>The impact</strong>: our development teams decrease feature cycle time in an AVG of 30% YoY, get more control over tech debt investment and gain better visibility of our R&amp;D execution.</p><p><strong>Step 3: Make Sure You Use Data That You Can Trust</strong></p><p>Use out of the box/customized reports that are based on strong R&amp;D team habits, so data reflects reality. E.g, if the team updates the task status on a daily basis, but does not report their actual work on tasks — measure cycle time rather than logged time reported.</p><p>When people see the value they’re getting by learning from these metrics, they’ll be open to implement new habits that will eventually provide new data.<br>At Skai, we took all data from our task management system, and built our own BI views on top of it. This enabled us to apply our own logic. Consuming data is easy, and requests for new metrics/knobs are managed by the program team.</p><p>Be minded to user experience — a dashboard that is also a tool to investigate the above “pains” can be your way to capture your users’ attention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_khd5E0pFphCzf424-bfOA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4yEFDJqBnuTqHyVsh0QEKw.png" /></figure><p><strong>Step 4: Adoption &amp; Stickiness</strong></p><p>You need to work your way in a Pincer Movement here — both bottom up and top down;</p><p>Include the metrics review routine in your team retrospective. This is where team members ventilate how they feel and validate it with the metrics. <br>From the other end, managers need to show interest and provide professional feedback based on the execution trend.</p><p>At Skai, we initiated a monthly group retrospective for team leads with their manager, and a quarterly with the directors. These meetings focus on reviewing the trend and the underlying issues, and discussing what we need to do to improve.</p><p>By implementing this framework, the trend is reviewed from the team up.</p><h3><strong>3 Tips Before You Hit the Road</strong></h3><ul><li><em>Dance like nobody’s watching</em> — think of the right metric for <strong>you</strong> that reflects your pain, without overthinking what if other people see it.</li><li>Compare You to you — compare yourself to who you were yesterday, not to other teams</li><li>Amplify every small success with the team. We rely on feelings and tangible feedback, like “this feature is amazing”. But feedback backed up by data — that’s golden.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b5cf948a832" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/so-pick-me-choose-me-love-me-a-4-step-guide-to-finding-your-r-d-execution-metrics-true-north-5b5cf948a832">“So Pick Me, Choose Me, Love Me” — A 4 Step Guide To Finding Your R&amp;D Execution Metrics’ True North</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Should I use Spark? It’s complicated]]></title>
            <link>https://medium.com/kenshoos-engineering-blog/should-i-use-spark-its-complicated-1297feb46532?source=rss----38970fe22728---4</link>
            <guid isPermaLink="false">https://medium.com/p/1297feb46532</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[distributed-computing]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Alexander Moshan]]></dc:creator>
            <pubDate>Tue, 29 Nov 2022 13:02:03 GMT</pubDate>
            <atom:updated>2022-11-29T13:02:02.991Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/566/0*k7efzylIN1rBO1vh" /></figure><h3>How do you know when you should use Spark?</h3><p>Apache Spark has positioned itself as one of the leading tools for large-scale data processing. However, when facing a specific data-processing problem, it’s difficult to know whether Spark would be the most suitable solution. This becomes even more challenging if your company doesn’t have much existing experience or infrastructure around Spark. Using Spark requires adjusting your algorithms, learning the library, handling resource management, managing clusters (or paying for platforms to do it for you) and most importantly, developer time. Is it worth the investment? Will it solve your problem?</p><p>Skai has recently centered its focus towards data. We’ve always had a lot of data in Skai — it’s time to use it to generate more meaningful insights for our customers. Most of our <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL</a> flows are simple <a href="https://airflow.apache.org/docs/apache-airflow/stable/index.html">Airflow DAGs</a> with steps that run simple or complex SQL queries. Very quickly we reached a point where there was a need for heavy processing in order to generate specific datasets for all our products. That is where our story begins.</p><p>In this post I’ll explain the journey we took to solve a seemingly simple problem (that turned out not to be so simple after all), starting with a naive implementation and gradually moving to the full blown solution implemented in Spark.</p><h3>The Problem</h3><p>As part of building our knowledge base over our data, we faced a common online-retail problem of merging product sets across various retailers — i.e. identifying identical products sold on different platforms (e.g. Amazon, Walmart) even though each platform might identify products using a different subset of globally-unique identifiers. In our case, each product has three types of IDs: Each source has its own mandatory ID (“source ID”) and two additional global IDs that are optional but are used across sources.</p><p>The mission was to create a unique ID for each <em>group of product instances</em> that represent a single product spread across sources.</p><p>A simple rule was used to unite products: if they shared the same ID of the same type of ID, then they were the same product. Which in turn can create long chains of IDs that keep on merging product instances. e.g:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tvzXXDl4KKgP8XNG" /></figure><ol><li>Product1 and Product2 are connected via Source ID SID_111</li><li>Product3 and Product1 are connected via Global ID 1 GID1_222</li><li>Product4 and Product3 are connected via Global ID 2 GID2_222</li><li>All the steps above should result in the first 4 products merged into a single product with one unique ID</li></ol><p>The output we’re looking for has the following structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NObSyVnGBR3cPPWv" /></figure><p>The job had to run on a weekly basis, meaning that the process should not run more than a week.</p><h3>Iteration 1: Draft algorithm in Python</h3><p>With this requirement came a draft algorithm that was written in a Python notebook. It completed processing a small dataset of about 1 million products in about an hour, and had 2 issues:</p><ol><li>The algorithm was non-deterministic, meaning different order of the rows resulted in different grouping, which was obviously unacceptable.</li><li>The algorithm wasn’t really scaleable: at worst case scenario it was running in O(N²) requiring O(N) memory, which would result in horrible results on our actual datasets.</li></ol><p>The algorithm was just a draft, but it gave us the first step to see the bigger picture. With a lot of thought put into it we came up with a new improved algorithm which was both deterministic and more scalable — with an O(N) complexity and requiring O(3N) memory (which is still O(N)). The improved algorithm was inspired by the standard <a href="https://en.wikipedia.org/wiki/Depth-first_search">DFS algorithm</a> that starts from a single product ID and digs its way through the entire chain of IDs until the group is complete. Now all that was left to do was to implement and use it at scale.</p><h3>Iteration 2: Plain SQL</h3><p>Since our data is stored in <a href="https://www.snowflake.com/en/data-cloud/workloads/data-warehouse/">Snowflake</a> and due to the frequent table access required by the algorithm, our first instinct was to implement the algorithm in plain SQL — keeping the computation as close to the data as possible.</p><p>Such an approach would also fit right into our ETL flows as just another SQL step in an Airflow DAG. Unfortunately, due to long ID chains and the “loopy” nature of our algorithm, it would require an extremely complex SQL query which we would probably have a hard time reading and maintaining.<br>The idea of implementing it in plain SQL was therefore dropped, but the hope of using Snowflake as the processing engine was not lost just yet!</p><h3>Iteration 3: JavaScript stored procedure — in-memory processing</h3><p>We knew that Snowflake supports writing stored procedures in multiple languages, and decided that JavaScript would be the quickest way to implement the algorithm. We implemented and tested the following naive JavaScript stored procedure on a small table to verify that it does the job correctly:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28e4e96d97782b4d84941eeaee50793b/href">https://medium.com/media/28e4e96d97782b4d84941eeaee50793b/href</a></iframe><p>After a successful test, we started running it on bigger datasets. Within a few hours we started seeing failures over out-of-memory errors. The Snowflake functions couldn’t handle all the data in-memory and failed.</p><h3>Iteration 4: JavaScript stored procedure — IO-based processing</h3><p>We realized relying on memory in the function wasn’t going to cut it. Since we were already running within the database, we could just let Snowflake handle the memory while we focused on the processing.</p><p>We rewrote the JavaScript function to repeatedly query Snowflake for the data instead of loading much of it into memory. After successfully verifying the validity on a small dataset, we moved to test it on bigger datasets.</p><p>This was one tedious part with a lot of waiting. The solution was so slow that eventually Snowflake automatically terminated the process *insert frustration*. Our Snowflake instance is configured so that not a single process would run over 48 hours, because that indicates a wasteful process and is therefore not recommended. <br>We could change that configuration for a specific user, but we had no idea if we were close to being done with the job and needed just a few more hours, or, so far away that it would take longer than a week.</p><h3>Iteration 5: JavaScript stored procedure — Bulk processing</h3><p>We had to understand how close we were to finishing the job. We decided to try processing the data in bulks — which would reduce the number of queries and give us visibility into our process’s progress. The key was to come up with the right bulk size. If the bulk was too small, the processing would take too long and be terminated. If the bulk was too big, we would run out of memory.<br>Unfortunately nothing was good enough — we were either too slow or consuming too much memory. We had to figure out a different way. But we learned a lot about the nature of our problem and the requirements of our algorithm.</p><h3>Iteration 6: Spark POC</h3><p>Separately from our project — a group of architects were leading a process of enabling Spark adoption in the company. We had a staging account in <a href="https://www.databricks.com/spark/getting-started-with-apache-spark">Databricks</a> (a platform to work with Spark that provides automated cluster management) that was being tested with a few POCs.</p><p>I met with them to discuss the algorithm, which fortunately lended itself nicely to being distributed across nodes: The way the algorithm works allows each group of IDs to be discovered separately, without sharing any information between the groups. Spark’s distributed architecture of multiple nodes and multiple threads in the nodes is already abstracted and ready to use, which works to our advantage. They suggested that Spark was just the thing that could fit our algorithm and solve our problem. It was decided, we would try Spark.</p><p>I started self learning and writing a Scala application using Spark on a local machine, as a POC to verify that Spark could indeed solve our problem. I probably should have started with a course or two in Spark before I got into the process of writing the application and learning Spark as I go, but time was crucial and things had to be done quickly. The result was a lot of Spark problems I learned how to solve on the fly. I wholeheartedly recommend you complete at least a basic course about Spark before you begin to actually develop the application, because Spark uses a unique distribution model that’s hard to understand without some formal introduction.</p><p>Here’s a rough representation of what this algorithm looks like when implemented using Spark:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5be99102868df78073dbd1dd3ca151c0/href">https://medium.com/media/5be99102868df78073dbd1dd3ca151c0/href</a></iframe><p>The local Spark POC app was doing well on a selected subset of data — so once again we went on to test our solution at scale.</p><h3>Iteration 7: Spark over Databricks</h3><p>We started testing the application in our staging Databricks account on a selected dataset. We immediately had to do some Databricks-specific tuning, such as configuring the clusters and their supported language versions, their size, memory, CPU, external credentials etc., while trying to maintain low expenses.</p><p>After successfully processing staging datasets, it was time to move to the final stage, full production data. The result was that the Spark application completed after roughly <strong>60–90 minutes</strong>, a very delightful surprise that blew past the 1-week limit we started with, which in turn changed the requirements for the process to be run daily instead weekly.<br>Was that the end?</p><h3>Iteration 8: Spark fine tuning</h3><p>Once we had a more-or-less working solution, new requirements and some last-minute changes set the execution time at close to 2 hours. With more potential future enhancements in mind, we decided this wasn’t good enough, as it doesn’t give us enough leg room to keep the 1-day SLA in case volumes increase further.</p><p>We decided to go into another optimization cycle and managed to reduce execution time to <strong>12 minutes</strong> — mostly by moving some of the pre-processing into Spark, tuning partitions, and hashing some long string IDs used as comparison keys. A year later, we still use the same configuration maintaining a 15 minute SLA, while the data has more than doubled in the meantime.</p><h3>Conclusion</h3><p>This long process of trial-and-error wasn’t easy, but it was very educational when it comes to choosing the right tool for a large-scale data-processing job. Here are some of the conclusions:</p><h4>What are the requirements?</h4><p>Everything starts with requirements. How fast does it have to be? How much data is involved, and how fast will it grow? Is it a one-time process or will it run on a daily basis? How much are you willing to spend on resources? How many hours of development time can you spare? Will it need to be maintained?<br>There are plenty of questions you need to answer before you decide on a course of action.</p><h4>Measurement is key</h4><p>Always measure, calculate and make estimations based on these measurements. Start small, scale it up and see how things work and if they fit your needs. Try to deal with conditions as close as possible to the real thing. What will the final size be? How long will it run? Is your test set indicative of real-world data or will you need to change your code entirely? All those measures could help you decide if what you are using will solve the problem or not.</p><h4>No premature optimization</h4><p>Optimization and refactoring is time and effort consuming. If not done properly, the final results might not even be optimized towards the right goals. Make sure you actually need this optimization. With our application completing successfully within 2 hours, we only went into the last optimization after realizing datasets will grow fast enough to put us dangerously close to our 1-day limit.</p><h4>Don’t jump on buzzwords</h4><p>Your company has its own best practices, its own technology stack and experience. Multiple teams have faced multiple challenges before, which is a valuable resource on its own. Before you jump on what’s “hot right now”, consider the assets (knowledge, licenses, infrastructure) that already exist in your company. Have people faced similar issues before? Was the scale similar? Which existing technologies might help? How well your company handles new technologies and who will maintain them? Will it help others in the future? Every buzzword has a price.</p><h4>Draw your own conclusions</h4><p>Starting small and optimizing gradually teaches you a lot about your problem. Use that knowledge and determine the best practices for the future. We tested boundaries and learned the limits of our technologies, and roughly know what problems each technology can solve. Now we can advise on future problems whether the solution is worth the effort. We have baked some of that knowledge into our own Spark application “template” that resolves multiple technical issues that future teams will no longer need to face. We learned a lot, made our own conclusions, and improved things for the next people in line, and I hope you did too :).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1297feb46532" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kenshoos-engineering-blog/should-i-use-spark-its-complicated-1297feb46532">Should I use Spark? It’s complicated</a> was originally published in <a href="https://medium.com/kenshoos-engineering-blog">skai engineering blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>