<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yonatan Katz on Medium]]></title>
        <description><![CDATA[Stories by Yonatan Katz on Medium]]></description>
        <link>https://medium.com/@katzyonatan?source=rss-1bff84bc532b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*1azTzjcAaCaCdtXRxilaPw.jpeg</url>
            <title>Stories by Yonatan Katz on Medium</title>
            <link>https://medium.com/@katzyonatan?source=rss-1bff84bc532b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:57:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@katzyonatan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My Journey Developing with LLMs. tl;dr: Go Gemini!]]></title>
            <link>https://medium.com/@katzyonatan/my-journey-developing-with-llms-tl-dr-go-gemini-c9242f1de86e?source=rss-1bff84bc532b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9242f1de86e</guid>
            <category><![CDATA[mashlom]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[gemini]]></category>
            <dc:creator><![CDATA[Yonatan Katz]]></dc:creator>
            <pubDate>Mon, 07 Apr 2025 18:13:19 GMT</pubDate>
            <atom:updated>2025-04-07T18:13:19.376Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*g091AnMpuocgWD5iCSTLDg.png" /></figure><p>I work for Google. About a year ago, I founded mashlom.me, a project aiming to improve the Israeli healthcare system by developing tools and accessories to automate and ease the work of healthcare professionals.</p><p>Since it’s an off-hours project, I needed to be highly efficient. Early on, I started incorporating LLMs into my development process.</p><p>I began with ChatGPT. Once I exhausted the free plan, I subscribed. While it improved my efficiency, it wasn’t quite enough. Sometimes, I found myself wrestling with ChatGPT for hours to get the desired results.</p><p>So, I switched to Claude. This marked a significant step up in my development process. It felt like a mutual tango: Claude provided good solutions and working code, and in return, I structured the project to make it easy for an LLM to read and update (e.g., duplicating code where necessary, minimizing black boxes, keeping files short to reduce token counts).</p><p>Meanwhile, as a Googler, I had access to and used Gemini internally. To say I wasn’t impressed, especially with its coding capabilities, would be an understatement. I found it more useful for language-related tasks like writing Hebrew poems, translating text, or understanding phrases in context. It was also decent at summarizing text. But for code generation? Not really. Classification tasks were hit-or-miss, often depending on the context length.</p><p>I also experimented with end-to-end coding tools like windserf, loveable, and base44. They were useful for quickly creating mockups to show doctors. However, developing production-ready code with them proved challenging due to factors like integrating with an existing codebase, managing the size of the codebase, achieving pixel-perfect design, and complying with the specific UI/UX requirements set by mashlom.me’s experts.</p><p>Overall, though, I remained satisfied with Claude’s performance for my needs at the time.</p><p>Then, about a month ago, Google released its new model, Gemini 2.5 Pro. Although my previous experiences with Gemini hadn’t been great, social media buzz and benchmarks suggested it was worth another look, even for code generation. When a teammate suggested trying it with cline.bot “just to generate a readme file,” I gave it a shot. The VS Code extension was straightforward, and getting an API token without needing a credit card was convenient. The result was… okay. Functional, but not particularly impressive — after all, it was just a README.</p><p>When we initially started Mashlom, we used AngularJS. AngularJS? Seriously? Well, yes… but that’s a story for another post. Eventually, we reached a point where migrating to React became necessary. As often happens during refactoring, we built new features in React, leaving the old code with the promise of “we’ll migrate it at some point.”</p><p>About two months ago, I tasked Claude with this migration. The result was messy: the code didn’t work, the UI didn’t look right, and it was essentially unusable. So, the migration went back on the shelf, filed under “at some point.”</p><p>Until today.</p><p>Today, using Gemini and cline.bot, I successfully migrated almost all the AngularJS code to React — only some styling adjustments remain. It was AMAZING! But wait… does it actually work? Especially the complex mathematical parts written by my partner?</p><p>Well… turns out, that wasn’t a problem either! Gemini also migrated the corresponding tests from Jest to Playwright!</p><p>It’s hard to change old habits, especially for a self-proclaimed “conservative guy” like me when it comes to tools, but Gemini succeeded.</p><p>It had been frustrating working for Google and not being able to effectively use its flagship LLM for my own challenging coding tasks. It was particularly vexing knowing I worked for the company that published the disruptive “Attention Is All You Need” paper — the foundation for modern LLMs — yet feeling like our own tools weren’t quite leading the pack in practice.</p><p>Now, it feels like Google and Gemini are truly back in the game.</p><p>Google — you made my day!</p><p>(Oh, and thanks to Gemini for editing this post and generating the cover image :) )</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9242f1de86e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Code Styling in the Age of LLMs]]></title>
            <link>https://medium.com/@katzyonatan/code-styling-in-the-age-of-llms-6151889df00d?source=rss-1bff84bc532b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6151889df00d</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[github-copilot]]></category>
            <category><![CDATA[code-style]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Yonatan Katz]]></dc:creator>
            <pubDate>Wed, 02 Oct 2024 07:53:44 GMT</pubDate>
            <atom:updated>2024-10-06T06:26:52.248Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="LLM" src="https://cdn-images-1.medium.com/max/937/1*JDPLwQDSQ8MkpEm-4Ib9cw.png" /></figure><blockquote>As a developer with no prior knowledge of React, I recently started to develop an ambitious project with Claude, an AI language model: building a complete application for managing hospital protocols. Meanwhile, a fellow volunteer at our organization created an impressive framework for managing hospital applications. At first glance, anyone comparing our code would likely conclude that my colleague’s work appears more professional. However, this experience led me to an important realization: my code, while perhaps less conventionally elegant, was far more conducive to collaborative development with an LLM. Unlike my colleague’s work, which required providing extensive background information to the AI, each page in my application stood independently. This allowed me to simply copy, paste, receive results, and integrate them back into my project with ease. This experience highlighted a crucial point: while code styling remains important, it needs to evolve. What we once considered “clean code” may need to be written differently in the era of LLM-assisted development. In this post, I’ll share my insights on how our approach to coding might need to change to better align with this new paradigm.</blockquote><p>Today, we consider clean code and consistent styling important things. There are several reasons for this, one of which is simply to appease those code-styling-obsessed programmers. But there are other compelling reasons: it makes code easier to read, reduces cognitive load for developers, and allows for better use of IDE tools. In other words: Consistent code styling is important for a maintainable code.</p><p>This is still true, but in the era of LLMs the styles, and what we consider as “clean code” should change.</p><p>The way we develop is changing. Large Language Models (LLMs), GitHub Copilot, and auto-code generators are writing more and more code. There’s no longer a need to memorize Dijkstra’s algorithm or manually add new endpoints.</p><p>Recently, I developed a full-featured, production-ready application — not just a one-day student project. I used React and Node.js, despite having little knowledge of either. But Claude knew them well.</p><p>Claude generated everything step by step, including tests. My process was simple: define requirements in a prompt, copy-paste Claude’s changes, perform QA, and ask for fixes. I think we can all agree that this is the new way of writing code.</p><p>This brings us back to styling. While it remains important, it needs to adapt to this new coding paradigm. Coding effectively with LLMs means:</p><ol><li>Minimizing manual changes. The goal is to simply copy-paste the LLM’s output. Let the LLM generates everything, not just a single function.</li><li>Achieving more with minimal context. Long contexts are not only costly and time-consuming but also less accurate.</li></ol><p>Here are my styling takeaways from my recent project:</p><ol><li>Keep files and components small: In the past, if I had three dialogs, each 10 lines long, I would have put them all in the same file. Now, I prefer creating three separate files.</li><li>Reduce dependencies, make code self-contained: You want to minimize the context you give to the LLM. The fewer imports you have, the easier it is for the LLM to modify the code. This requirement now outweighs avoiding code duplication. Think of it this way: when asking an LLM for a change, how many files should you attach to the prompt?</li><li>While I don’t recommend this yet, we should consider eliminating folder hierarchies and keeping everything in the same directory. It sounds odd, I know. But consider this: development is an iterative process. We’ll likely ask the LLM to change the code 3–4 times until it meets our requirements and is bug-free. At each iteration, you need to integrate the code into your app and run it. If you need to make manual adjustments, like the exact path for the imports, you’ll either have to instruct the LLM to do it or apply them manually each time. Wouldn’t it be easier to just put everything in the same folder, and let the LLM set the import to ./MyComponent.js ?</li><li>Avoid internal acronyms and don’t be afraid of long names: Let’s say you have a knowledge base stored in the database, and you want to query one article by its title. Which function name will be easier for the LLM to understand? getKnowledgeBaseArticleByArticleTitle or getKBByTitle? Remember, the LLM is the newcomer that hasn’t gotten used to your acronyms or your internal language. It’s like a new team member who hasn’t been indoctrinated into your company’s jargon yet. Clarity trumps brevity when working with LLMs, so opt for descriptive, self-explanatory names even if they’re longer.</li></ol><p>Finally, I want to share some prompts and phrases I frequently used. Some of the suggestions above relate to these prompts:</p><ul><li>“I have a web application based on Node.js and React. These are the main entities:” (I often had to list the entire model when starting a new context / chat)</li><li>“Can you please write the full file? So I can just copy-paste.” (I prefer to copy-paste rather than find the exact place myself. I’m lazy)</li><li>“Attached is the final version of MyFile.js” (I often made minor changes and wanted the LLM to continue development from where I left off)</li><li>“Please generate the test file. The test should test the controller. As a reference, you can see the attached file that shows you how to set up the server, connect to MongoDB, etc.” (Similar examples are encouraged, and LLMs can be very useful for writing tests)</li></ul><p>While I’ve provided some styling guidelines, the main takeaway is that you should consider the future of development. “Clean code” as we knew it may — and should — change. Being overly generic and having numerous abstractions isn’t ideal when developing with LLMs.</p><p>Structure your code so that you can take small pieces, paste them as-is into the LLM, and expect to copy-paste the result directly back into your codebase.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6151889df00d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The KPIs of KPIs]]></title>
            <link>https://medium.com/@katzyonatan/the-kpis-of-kpis-815ba9a60c77?source=rss-1bff84bc532b------2</link>
            <guid isPermaLink="false">https://medium.com/p/815ba9a60c77</guid>
            <category><![CDATA[kpi]]></category>
            <category><![CDATA[management-and-leadership]]></category>
            <category><![CDATA[management-software]]></category>
            <dc:creator><![CDATA[Yonatan Katz]]></dc:creator>
            <pubDate>Wed, 08 Jul 2020 15:48:47 GMT</pubDate>
            <atom:updated>2020-07-12T07:23:33.539Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9rtvAXs6zDkZonlFXiorFQ.jpeg" /></figure><p>I am running a bootstrap startup. The code is in good shape. The testing coverage is so good that I can deploy at any time without QA cycle. The product is stable, eventhough the code is extremely complex.</p><p>I have all of that, without settting KPIs for number of bugs or code coverage.</p><p>The story of the KPIs (key-performance indicators) in software corporates is funny and painful.</p><p>Have you ever felt you are enforced to meet some numbers, that everyone knows it has nothing to do with the success of the company? Are you required to work on KPIs, just because they are marked in red over a dashboard at the middle of the office? Are you familiar with some weird blitzes at the end of the quarter / year, for fixing numbers?</p><p>In corporates, managers see these numbers as a good recipe for a successful company. I am not so sure they right.</p><p>Here are two examples for KPIs I had to meet during my career:</p><p><strong>Opened / closed bugs ratio</strong>: I’m still waiting for the executive manager to say “amazing! What a great impact did you bring by reducing this annoying ratio”.</p><p><strong>Static code analysis warnings</strong>: I’m always happy to come back home, and tell my wife “you have no idea how important my role is! We now have only 132 minor warnings in our code analysis tool”.</p><p>Let’s list some rule of thumbs to know whether your KPI will ever contribute to the overall performance of the company:</p><ol><li><strong>You are embarrassed</strong> <strong>to talk about the KPIs with senior management</strong>: It will be a bit weird to show the CEO your KPI dashboard, filled with “number of P1 bugs” per development group.</li><li><strong>Amorphic bazzwords</strong>: Are you willing to pay for someone that promise you to deliver “quality”?</li><li><strong>Task is missing from the backlog</strong>: Have you ever sat in a planning meeting, and the manager / product manager said — “hey, but if we will implement all of that we won’t have time for the KPIs”? KPI needs planning, effort, trial and error. It shouldn’t be just an annoying task upon angry manager email.</li></ol><p>What is a KPI?</p><p><strong>KPI is the way to measure the results of the actions you take from a goal defined by higher management level </strong>(I’m not sure it’s the definition in the dictionary, but the dictionary never run a company, and certainly didn’t set KPIs...).</p><p>Your manager set you a goal. In order to meet this goal, you need to break it down into actions. Then, you need to set metrics that will measure whether you succeed to meet the goal.</p><p>For example, Chris, the CEO, set a goal of reducing customers churn rate. Obviously, reducing the churn rate will increase company growth. What are the action items required to gain that? After talking to customers and account managers, it turns out that the product is not intuitive enough, and the response time for the customer’s inquiries takes too long. These insights match the NPS survey made last year, that shows how customers are disappointed.</p><p>So the CEO set the following KPIs table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*dWywsrCsnLXhVb1w29UjOQ.png" /></figure><p>Great! We know what the strategy is, what needs to be done, and how we measure ourselves! Not only that, Now the R&amp;D knows what its goals are: Improve the product UX and give quicker respond to customers. But how can they achieve them? What actions the R&amp;D needs to take in order to say “we improved the product UX in a way that will cause the churn rate to reduce”?</p><p>The product and the UX team sat on this problem and discovered, with massive usage of Google Analytics, that it takes around 3–5 minutes to complete the setup, and only 3 main features are being used, out of 12 in the “processor configuration” screen. Also, they found that the loading time of the main page is around 30 seconds, which makes users to give up.</p><p>So the R&amp;D managers, set the following KPIs table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*iHCc5QH37pgpHQvAtX5D3Q.png" /></figure><p>When John, a team leader in the R&amp;D stand up and said — “ ‘ Shorten the setup time up to 3 minutes‘ is the measurement, not the action”, his manager answered — “Nope. ‘Setup time less than 3 minutes’ is the definition of done for the action you take. It says nothing about achieving the goals you set”.</p><p>See what we are doing here?</p><p>We are not setting KPIs just because. We are starting from strategy, and we go down the pyramid and verify the strategy is being implemented. Furthermore, in every level we know how to measure a success.</p><p>Let’s go back to the “Opened / closed bugs ratio” metric mentioned above. Isn’t it a good metric? Can’t we put it on the table in the same way?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*GiA4DLODgFtaJ2hKorsZOw.png" /></figure><p>Well, let me ask you this: Once you meet this KPI number — does it mean the product was improved? If I’ll ask the customer — will he say “WOW, you are much better now”? In other words: Do you measure the <strong><em>action</em></strong><em> </em>you take or measure the <strong><em>goal</em></strong> you set?</p><p>Wait a minute. Maybe the “make product more stable and less buggy” should be a goal for a team within the R&amp;D?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*ydsR7Jh_WAt2MHciYYKXOg.png" /></figure><p>This is much better. The company goal is to reduce churn; The way to do that is by improving product UX, which become the goal for the R&amp;D; The way to do that, is by making the product more stable and less buggy, which become a goal for a specific team. The way to make the product more stable is by closing the bugs — which can be measured by opened /closed bugs ratio.</p><p>Bingo? not so fast.</p><p>Indeed, this is a logical thinking. In some cases, this is how KPI structure should look like. Nevertheless, please consider the followings:</p><ol><li>Do you have a reason to believe that the problem is the bugs? Did you check that the bugs that were found in lab and were not solved are coming again from the field? Maybe there is lack of testing? In other words: Does the ratio measures the success of “closing bugs” or “product less buggy”?</li><li>Do you believe that the customers will see the impact of your action?</li><li>And most importantly — did the KPI represent the way you think about breaking goals into actions, or it is just a metric that you saw over the internet, and you’re trying to justify it by building this table bottom-up?</li></ol><p>The problem with how managers set KPIs is that they confuse between <strong>goals</strong> and <strong>means</strong>. They are mixing between measuring a success and how to drive people to work on something.</p><p>During the process of building KPIs as described above, we gain these benefits:</p><ol><li><strong>We distinguish between goals and means:</strong> Reducing churn rate is a goal. Reducing number of bugs is a mean. Measuring a success is measuring whether we are meeting the goals we set. Not the actions we took do implement that.</li><li><strong>You need to plan KPIs:</strong> Achieving the goal of reducing churn is a huge, tough task. It needs a plan: Which things impact churn? What are the main customer’s concerns? How much effort do we need to put in order to reduce this number? “Quick respond for customer’s inquires” should be a result of research and analysis. This research is super important and, unfortunately, less considered.</li><li><strong>Working on essence, not on number:</strong> It’s not just a task at the end of the quarter, it’s the way of thinking. It’s the way we build the company. Every developer knows that he needs to respond fast for escalations, because that’s what bothers the customers. This will reduce the churn — which will make the company to become successful.</li><li><strong>Aligning the company with the strategy:</strong> It doesn’t matter whether you are executive manager, or a developer at the bottom of the pyramid. You know what you are doing, why are you doing it, what are the impacts in company level, and what will be considered as a success.</li></ol><p>Back to the rule of thumbs for bad KPIs.</p><p>Let’s examine this table again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*UtSEbEh1quQJoBvx76tREQ.png" /></figure><p>Can you show this table during the all-hands session? Is it something that is interesting to show during the monthly status with the CEO?</p><p>Is it amorphic? Are you willing to pay for outsourcing service, that will ensure the next NPS survey will produce results that are better in 20% than previous survey?</p><p>Is it a real task that needs design, product and thinking? Do you need a story in your project management tool that describes “shorten the setup time up to 3 minutes”?</p><h4>Summary</h4><p>KPIs are not just numbers out of the blue; They are a way of thinking, the way you break goals down into tasks and measure how much you reach the goals.</p><p>KPIs are hierarchical. Every management level set goals to its direct reports. Together they break the goals into tasks. Every task become a goal to the next managerial level, down stream.</p><p>After understanding the model, it’s easy to see how KPIs can drive the company forward. How it makes everyone aligned to the strategy, how it is effectively measures the performance.</p><p>Back to my startup.</p><p>My strategy was to <strong>keep the churn on 0</strong>. No customer should leave. It takes too much time to secure new customers, I cannot afford myself to let someone leave. In addition, I wanted the <strong>time-to-market to be as fast as possible</strong>, so I can move fast with new leads. Luckily for me, I managed to be where I planned. How did I do that?</p><ul><li>I have great testing coverage</li><li>Every field bug is fixed the day after, with a new automated test that covers it.</li><li>I have full CI/CD. I can deploy my code at any time.</li></ul><p>I never set quality as a goal. It wasn’t interesting enough.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=815ba9a60c77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testing Automation — Making It Right: #3 Test Machine Learning Based Applications]]></title>
            <link>https://medium.com/@katzyonatan/testing-automation-making-it-right-3-test-machine-learning-based-applications-388d7ae5d55c?source=rss-1bff84bc532b------2</link>
            <guid isPermaLink="false">https://medium.com/p/388d7ae5d55c</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[automation-testing]]></category>
            <category><![CDATA[machine-lear]]></category>
            <dc:creator><![CDATA[Yonatan Katz]]></dc:creator>
            <pubDate>Mon, 13 Apr 2020 19:07:21 GMT</pubDate>
            <atom:updated>2020-04-16T19:51:18.774Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N33pWVL_GjrrcIAiROAkrA.jpeg" /></figure><h3>Testing Automation — Making It Right #3: Test Machine Learning Based Applications</h3><p>This is the third part of the series.<br>#1 — <a href="https://medium.com/@katzyonatan/testing-automation-making-it-right-1-theory-a5c694ab2c22">Theory</a><br>#2 — <a href="https://medium.com/@katzyonatan/testing-automation-making-it-right-2-test-types-and-integration-tests-e782bdeee2a4">Test Types and Integration Tests</a></p><p>Why do I dedicate a post for machine learning testing? Isn’t it just like any other piece of code?</p><p>Well, in some aspects it’s like a regular app that needs to be tested, but in some aspects — it’s different.</p><p>(a) All machine learning algorithms are based on statistics. They don’t guarantee that a specific input will work, and therefore — it’s hard to unittest it. Writing a test that you are not sure it will be “green” in the future is problematic.</p><p>(b) It’s hard to estimate the effect of a new ML model on the final product. Even if the accuracy increases it doesn’t mean a specific customer won’t be badly affected.</p><p>(c ) Running a test that checks the ML statistics (accuracy, false positives, precision / recall, etc.) on large input may take time. We want to acheive an efficient dev environment, and we cannot afford every tests run to take one hour.</p><p>In this blog post, I would suggest several things to consider, from my experience. The goal is clear: <strong>We need to have one gateway that tells us whether we can deploy a new version or not.</strong></p><ol><li><strong>Separate the model and the application</strong></li></ol><p>Since checking the ML statistics may be slow to run during development, or even as part of CI — it should be in a different deliverable. Its tests should run in a separate CI pipeline. The statistics test should be a gateway only for the ML model, and not for the entire application.</p><p><strong>2. Define “improvement” of your statistics</strong></p><p>You need to define what is considered as an improvement of your model, and break the CI if you have degradation. I am not defining here what is a “degradation”, as it is a project specific decision. In some projects the total errors is the main number, in others the false positives and not the false negative, etc. You first need to define it, then you need to enforce it.</p><p><strong>3. Do your best to have consistent results</strong></p><p>In many cases I encountered, two successive training runs can produce two different results, even when code and data have not been changed. Obviously, this can make the tests “flicker”. One solution for that is to allow degradation of X % from previous run. But that’s not the best option. First because after several runs you can have bigger degradation. Second, because you can allow releasing app version even though you don’t really know what is the cause for the degradation, or what’s its impact.</p><p>The better option (although not always possible) is to make sure you have consistent results: If you have data-set — it should be sorted, so every run on every machine will take the input in the same order; If you run over files — sort them by name; Avoid iterating non sorted data structure (e.g. sets); Make sure all random generated numbers have constant seed (even if you need to hack the ML library a bit. True story); You understand my point.</p><p><strong>4. Write “regular” tests for demos and sanity scenarios.</strong></p><p>When the sales person is going to a customer, his demo must work. Excuses like “its ML based algorithm, that’s why it doesn’t work for this input. For other inputs it will work” — look bad.</p><p>Test any input that is used for demo. That’s a good advice for every project, but mainly in ML project.</p><p><strong>5. Try to engage with your customers to integrate their data into your tests.</strong></p><p>ML training assumes that your data set represents the real distribution of data. But what happens when you have a customer that his main data does not have enough representation in your data set? Delivering new model can affect his flow dramatically even though your accuracy is higher in the newer version.</p><p>Therefore, try to ask your customers to provide you with a “sanity data set”. Add them to your tests. Your customers will thank me.</p><p><strong>6. Test the code around the ML logic.</strong></p><p>From time to time, when you have system tests you skip writing integration tests, as they are already included. I can understand that, although not agree.</p><p>In machine learning testing, don’t assume that if training passes and accuracy increased — the code doesn’t require further testing. Test every piece of code separately.</p><p>If you have features extraction code — test it. If you have pre-processing, post-processing, or anything related — test it separately. Don’t assume the final ML statistic numbers are enough.</p><p>One of the reasons for that, is because ML can hide bugs. ML can work well with bugs, because the bug may be consistent. Yet, the model can be much improved if the code will work.</p><p>If I need to summarize this post, the summary would be: Make sure you’re comfortable with shipping your new model if you have “green light” in your tests.</p><p>You’re afraid of customer use case? add it to the test. Main flows can be affected? test it.</p><p>Finally, test your stats.</p><p>Now, you’re ready for deployment!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=388d7ae5d55c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testing Automation — Making It Right: #2 Test Types and Integration Tests]]></title>
            <link>https://medium.com/@katzyonatan/testing-automation-making-it-right-2-test-types-and-integration-tests-e782bdeee2a4?source=rss-1bff84bc532b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e782bdeee2a4</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[unittest]]></category>
            <category><![CDATA[system-testing]]></category>
            <category><![CDATA[integration-testing]]></category>
            <dc:creator><![CDATA[Yonatan Katz]]></dc:creator>
            <pubDate>Mon, 13 Apr 2020 17:36:10 GMT</pubDate>
            <atom:updated>2020-04-13T19:16:22.767Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Up8vHcKFDOymCr_HEPu3AA.jpeg" /></figure><h3>Testing Automation — Making It Right #2: Test Types and Integration Tests</h3><p>This is the second part of the series.<br>#1 — <a href="https://medium.com/@katzyonatan/testing-automation-making-it-right-1-theory-a5c694ab2c22">Theory</a><br>#3 — <a href="https://medium.com/@katzyonatan/testing-automation-making-it-right-3-test-machine-learning-based-applications-388d7ae5d55c">Test Machine Learning Based Applications</a></p><p>There are lots of terms for testing types: e2e (end to end), unit, integration, sanity, smoke, component, contract…</p><p>In order to keep it simple, and avoid fighting on syntax, I would list three of them, and sketch their boundries:</p><ul><li>Unit test — testing an algorithm, a function. It doesn’t test a flow, integration with any 3rd technology (like database) or any other services.</li><li>Integration test — testing a flow: http request, processing of item, complete single cli command.</li><li>End to end (e2e) test — testing the whole system: all components up and running, UI and server, full sequence of cli commands that represent a real world scenario, etc.</li></ul><p><strong><em>Unit test:</em></strong></p><p>Pros: Fast, code branches (if… else…) can be easily tested</p><p>Cons: In web applications, you need to work hard to mock services. Therefore, writing tests become hard and slow task, with lots of maintenance. You are testing the internal of the codes, not integration points. By nature, integration points have less changes than method signatures.</p><p>The most problematic part with unit tests is the confidence in the code you deliver. You don’t know whether you can ship your code even if tests pass successfully. For example, an algorithm can relay on a DB query. The algorithm is perfect, but the query is incorrect. In such case, you have a bug in the code you just wrote, but you cannot discover it using the unit tests.</p><p><strong><em>Integration test:</em></strong></p><p>Before analyzing this type of test, which is my favorite, we need to better define it: What is “integration”? What does it include?</p><p>Different engineers will tell you different things. Here is my definition for integration test:</p><p>Testing a flow. We test a single component (usually, an OS process). If needed, other components / processes can run as well, assuming they meet the following preconditions:</p><p>a. We can assume they work.</p><p>b. They return consistent results.</p><p>For example:</p><ul><li>DB can be run as part of the integration test, as we can assume it works, and for a certain input — it returns consistent output.</li><li>Integration with Google search API must be mocked, as it doesn’t return a consistent output for a certain input.</li><li>If your company develops two microservices, and one calls another — the other microservice must be mocked, as we cannot assume it works.</li></ul><p>The idea behind these rules is to get green light whenever you didn’t break the code, and red light when you are the one who broke the code. Dealing with endless “why it doesn’t work” questions makes the processing of writing and running tests very cluttered.</p><p>That’s why Google search API must be mocked: we cannot guarantee the test will pass. If we test several services we develop at once — one service can break the test, and it will be hard to find why it is broken, or who broke it. It won’t always be related to the code you just wrote.</p><p>Now, the pros of integration tests are — you have pretty good confidence in the flow; you are capable of developing each component separately, which makes the coding more efficient; integration between teams become easier.</p><p>Cons: Inability to test nuances in the code on the one hand, and you don’t know that your system as a whole works on the other hand.</p><p><strong><em>E2E test:</em></strong></p><p>Sounds like the dream of every manager: In end-to-end tests, you are spinning up the whole system, and start running complete scenarios. Happy happy joy joy!</p><p>In practice, its a VERY expensive solution for complex systems: —</p><p>(a) It takes time to run all tests. That means you cannot run all of them before deploying critical fix.</p><p>(b) Since the developer didn’t develop the whole system, it is not the developer who write the tests. Therefore, there are test engineers that are responsible for the tests, and that makes the feedback process on the code quality much longer. See details about the feedback loop in the <a href="https://medium.com/@katzyonatan/testing-automation-making-it-right-1-theory-a5c694ab2c22">first blog post</a>.</p><p>(c) Much more people are involved in testing, reviewing and investigating bugs.</p><p>(d) The resources to run the tests are often very expensive.</p><p>(e) Environments where the tests run on become buttlenecks, and from time to time hard to maintain.</p><p>Therefore, my recommendations for testing are:</p><ul><li>Use unittest for complex algorithm. Not for web request flows.</li><li>Use integration tests as much as possible, especially for web application. Make sure every developer can run them at any time, from their IDE; Make sure the “green light” for shipping doesn’t depend on other services to work.</li><li>Use end to end testing only for basic sanity and extremely important flows (e.g. signup)</li></ul><p>You can read this great <a href="https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html">blog post</a>, describing google testing approach. I completely agree about the e2e tests part. As oppose to them, I would take unit/integration test approach according to the nature of the tested element, and not according to the pyramid rule of thumb.</p><p>Next: <a href="https://medium.com/@katzyonatan/testing-automation-making-it-right-3-test-machine-learning-based-applications-388d7ae5d55c">Test Machine Learning Based Applications</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e782bdeee2a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testing Automation — Making It Right: #1 Theory]]></title>
            <link>https://medium.com/@katzyonatan/testing-automation-making-it-right-1-theory-a5c694ab2c22?source=rss-1bff84bc532b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5c694ab2c22</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[automation-testing]]></category>
            <dc:creator><![CDATA[Yonatan Katz]]></dc:creator>
            <pubDate>Mon, 13 Apr 2020 17:06:10 GMT</pubDate>
            <atom:updated>2020-04-13T19:09:45.590Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LbOCB_exHFau_gs9fleq4Q.jpeg" /></figure><h3>Testing Automation — Making It Right #1: Theory</h3><p>This is the first part of the series.<br><a href="https://medium.com/@katzyonatan/testing-automation-making-it-right-2-test-types-and-integration-tests-e782bdeee2a4">#2 — Test Types and Integration Tests</a><br><a href="https://medium.com/@katzyonatan/testing-automation-making-it-right-3-test-machine-learning-based-applications-388d7ae5d55c">#3 — Test Machine Learning Based Applications</a></p><p><strong>About the series</strong></p><p>Over the years I have worked on lots of projects — as a developer, as a manager and as an entrepreneur, from small and medium startups to large enterprises, across the fields of machine learning, big data and web applications. Its a good time to reflect and share tips and insights I have learned in creating effective testing automation. I would like to discuss testing from a methodology point of view, and debunk some misconceptions.</p><p>I never had QA engineers in my greenfield projects. Nevertheless, these projects’ were much higher quality than projects I was involved in that have QA engineers.</p><p>Not only I feel that the projects without QA engineers have less bugs, regressions and crashes — I much more enjoyed developing these projects. The development was fast and effective, and I could concentate in the code.</p><p>By “projects without QA engineers” I don’t mean there was no testing. In these project, there were two rules for testing: (1) All tests must be automated, and (2) all tests must be written by the developers, and not by dedicated QA engineers.</p><p>Let’s first start with setting goals for testing automation. What exactly are we trying to acheive by having automation tests?</p><p><strong>Setting goals</strong></p><p>Engineers usually find automation beneficial because it increases quality, reduces regression testing time and allows you to deliver fast.</p><p>I agree, but I would like to tackle it from a different angle. What I am about to say might seem obvious, but I would like to shift the argument from end results to how we can improve our development processes and adopt more effective development methodologies.</p><p>So, with automation tests…</p><ul><li><strong>The feedback to the developer on the correctness of her code comes much faster.</strong> This is one of the key enablers for fast development.</li><li><strong>Dependencies between teams are dramatically reduced.</strong> Not only that you don’t need to coordinate between the dev team and the QA team; With automation tests you’re capable of developing the server independently from the UI; You don’t need to wait for other team’s microservice to be ready; You don’t need other teams to analyze a bug or a support case, before it comes to your desk. You can easily add a new test that reproduces a field case using the same input + output.</li><li><strong>Customers won’t get the same bug twice. </strong>Sounds trivial, ha? tell that to enterprise customers.</li><li><strong>New or inexperienced developers are more limited in the damage they can do to your production environment.</strong> Not only it will be hard for them to ruin the production, when I had a [very] weak developer, I just made a massive code review on his tests. Shitty code, but works.</li><li><strong>Development considerations are written explicitly</strong>. That’s the best way to document things. Developers cannot ignore them. They will see immediately why their code contradict the ideas behind the base code.</li><li><strong>You have a clear gateway before production deployment.</strong> Think of a big refactor you did. How much were you afraid before deploying the code to production?</li></ul><p>When these are your goals, along with the common ones, your whole methodology may change. It’s not just that you’ll add few QA automation engineers that write tests on top of the existing flow. The goals above put the responsibility mainly on the developers:</p><p>If a QA engineer will write the tests, the feedback loop to code quality is longer; The ability to write code independently from other developers / teams doesn’t exist; QA engineers are not familiar with code considerations, and therefore they cannot leverage the testing as a documentation tool.</p><p><strong>Misconceptions</strong></p><p>When I am speaking about the need of developers to write testing automation, for some organization its trivial (OK Google). But for some, I get strong resistance. The main two claims are:</p><ol><li>The development become much slower.</li><li>Its frustrating developers.</li></ol><p>I am not going to argue about #1, because if you are writing it correctly — the development becomes faster. Not slower. Those who claim #1 didn’t work on a project where the testing automation was effective. The reason for the fast development is simple: The main time consumer is bureaucracy and dependencies between teams. Testing reduces both. There are more reasons why testing makes the development faster, but as I said — I’m not going to convince you why its true. Before claiming the development is slower, you should work in a place where there are good testing.</p><p>Regarding #2 — “Its frustrating developers” — its true. It is frustrating developers, but mainly the poor ones. What I can say is that in my experience, the dependencies between teams, the inability to know in advance whether the code works, deploying patches in the middle of the night — all of these, are much more frustrating. Therefore, I prefer writing tests.</p><p><strong>Do we need manual QA engineers?</strong></p><p>This question must be asked: Do we need manual QA engineers? Should the developers do everything?</p><p>In my opinion — in most cases we don’t need them. They usually become bottlenecks, and it prevents the project to develop in the right way. Having manual QA engineers reduces responsibility from the developers, the quality feedback to the developers arrives a while after the code has been completed, and… its very expensive. More engineers, more people that has to be involved in code &amp; bugs, more bureaucracy…</p><p>BUT,</p><p>There are few cases where I would take manual QA engineers (in addition to automation tests written by devs):</p><ol><li>On-prem products: fixing on-prem product problems costs the organization a lot more than in SaaS. Coordinating with customers, getting the logs, inability to control the environment, deploy debug patches, etc. Therefore, we want QA engineers to be “safety net” before going GA.</li><li>You may want to have QA engineer to independently test the customer experience, or test the global system experience as a whole. Its not a must, but it can bring value. You need very few QAs for that, and they shouldn’t be a gateway before deploying the code in production.</li></ol><p>Next: <a href="https://medium.com/@katzyonatan/testing-automation-making-it-right-2-test-types-and-integration-tests-e782bdeee2a4">Test types</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5c694ab2c22" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>