<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by breakingtesting on Medium]]></title>
        <description><![CDATA[Stories by breakingtesting on Medium]]></description>
        <link>https://medium.com/@nikolayn87?source=rss-2863de45312e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BIV4C-Cze5eNtGV_EXXiyQ.png</url>
            <title>Stories by breakingtesting on Medium</title>
            <link>https://medium.com/@nikolayn87?source=rss-2863de45312e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:08:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nikolayn87/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI claims to have evidence but cannot prove]]></title>
            <link>https://medium.com/@nikolayn87/ai-claims-to-have-evidence-but-cannot-prove-795d9a1dd8ac?source=rss-2863de45312e------2</link>
            <guid isPermaLink="false">https://medium.com/p/795d9a1dd8ac</guid>
            <dc:creator><![CDATA[breakingtesting]]></dc:creator>
            <pubDate>Wed, 23 Apr 2025 08:40:43 GMT</pubDate>
            <atom:updated>2025-04-23T08:40:43.861Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*CCflJMW7se0ZlXS0-MtrjA.png" /></figure><p>Initially, I was going to write an article about my experience using logs from functional automated tests to detect performance degradations (yes, sounds uncommon). Before starting, I decided to use a well-known AI instead of Google Search — just to check if anyone had already written about this approach. The conversation I had with the AI led me to write something entirely different.</p><p>Here, I want to show you the kind of miserable results you get when you blindly trust the hyped claims that AI can “analyze,” “write code,” and do all sorts of bla bla. Sure, it can — to some extent — with questionable quality.</p><p>All the screenshots below are from the actual conversation. If you need proof, DM me or leave a comment — I’ll share the link to the chat, so graciously offered by the AI service itself.</p><h3>Original request</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_0ZVXclvbMC_PnNM8GTMg.png" /></figure><p>AI’s first answer says clearly — there’s evidence and suggests to share links with me:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jDY-tLS9_uU5riV_qQUxug.png" /></figure><h3>Analyzing the first “evidence”</h3><p>AI claims the “Performance Testing” article in Google Testing Blog shows proofs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*msHG8-oUiXIN5DmfJEOatg.png" /></figure><p>I read the whole article, cool. However, it does not actually had evidence I was asking for:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A60yvRpM_iG0azn4AHiM4Q.png" /></figure><p>I decided to clarify, whether the link really had no evidence. AI continued hesitating:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CnBkiH65H6F1lDcza2kk0g.png" /></figure><p>Finally, got a clear confirmation that AI’s first link was garbage:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XFOyeHirdOg9V-FamZzNHg.png" /></figure><h3>Moving on to the second “evidence”</h3><p>Here’s the second link provided by AI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muswCNhcYlT2psDeYy0DGQ.png" /></figure><p>Clarifying:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yk4nMAJ_BD-tnk2Pi6c51g.png" /></figure><p>Not only did it confirm the answer was garbage (again), but managed to confuse the links accuse me:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9cJrZoKJRVAFnm3eT1XI-w.png" /></figure><h3>Clarifying the third “evidence”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5haKr5g6HvjYjL_VGJxTpA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnmXou9Nwv2Wbseiwk2g6A.png" /></figure><h3>Conclusion</h3><ol><li>Three times in a row AI gave 100% wrong answers.</li><li>Used wrong links from the context or silently confused them.</li></ol><blockquote>Trust, but verify.</blockquote><p>I wasn’t going to check the fourth link, given how bad the first three were. However, to be fair, the AI eventually managed to provide correct links after several iterations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=795d9a1dd8ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use case for self-healing tests with a local LLM]]></title>
            <link>https://medium.com/@nikolayn87/use-case-for-self-healing-tests-with-a-local-llm-dc1bb24f42ca?source=rss-2863de45312e------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc1bb24f42ca</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[self-healing]]></category>
            <category><![CDATA[qa]]></category>
            <dc:creator><![CDATA[breakingtesting]]></dc:creator>
            <pubDate>Wed, 12 Mar 2025 21:51:28 GMT</pubDate>
            <atom:updated>2025-03-12T21:51:28.062Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*8M685LRFwBNinKyKQl0iyw.png" /></figure><p>In this article, I will discuss the practical application of large language models (LLMs) in combination with traditional automation tools like Python/Selenium to improve test reliability.</p><p>The article consists of the following sections:</p><ol><li>What are self-healing tests</li><li>Hardware configuration</li><li>Software configuration</li><li>Verifying the local LLM API</li><li>Integrating with tests</li><li>Limitations</li><li>Future prospects</li></ol><h3>What are self-healing tests</h3><p>Automated tests must be reliable to prevent false positives. Such tests increase trust in their results and enable deeper integration of automated testing into processes. Test reliability can be improved by addressing the key issues that arise during their execution. All these issues lead to false positives:</p><ol><li>Changing properties of elements in the tested application.</li><li>Unstable infrastructure.</li><li>Excessive speed of the automation tool.</li></ol><h3>Hardware configuration</h3><p>Based on these common issues, we define self-healing tests as those that can automatically adapt their behavior when a problem occurs. The focus of the current implementation is on solving issue #1.</p><p>The table below provides measurements of two model performance parameters on our configurations, along with a description of the configurations themselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MUpgIf04JnWXlvQwCy2c7g.png" /></figure><p>Due to limited hardware availability and experimental usage, we use only one server in the first configuration</p><h3>Software configuration</h3><ol><li>The model lmstudio-community/Qwen2.5–7B-Instruct-MLX-4bit from HuggingFace (convenient to download via LMStudio — see item 3 below).</li><li>The OpenAI pip package.</li><li><a href="https://lmstudio.ai/">LMStudio</a> — an “IDE” that is convenient for prompt debugging, model parameter configuration, offline execution, and setting up a “model server.”</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YnT33IE4MURbMB-nToB96w.png" /><figcaption>Debug prompts</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3JM7ef4jF61B2ph52pTs9w.png" /><figcaption>Configure model</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6VfT4oz1frxQTNBBoP3-1g.png" /><figcaption>Configure server</figcaption></figure><h3>Verifying the local LLM API</h3><pre>(venv) user@MacBook-Pro-Admin-2 web2 % cat tttt   <br>from openai import OpenAI<br><br>LLM_URL = &#39;http://&lt;Internal_IP_address&gt;:1234/v1&#39;<br>LLM_MODEL = &#39;qwen2.5-7b-instruct-mlx&#39;<br><br>def call_llm(request):<br>    client = OpenAI(base_url=LLM_URL, api_key=&quot;lm-studio&quot;)<br>    completion = client.chat.completions.create(<br>        model=LLM_MODEL,<br>        messages=[<br>            {<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;content&quot;: [<br>                    {<br>                        &quot;type&quot;: &quot;text&quot;,<br>                        &quot;text&quot;: request,<br>                    },<br>                ],<br>            },<br>        ],<br>    )<br>    return completion.choices[0].message.content<br><br>print(call_llm(&#39;Yes or no?&#39;))<br>(venv) user@MacBook-Pro-Admin-2 web2 % <br>(venv) user@MacBook-Pro-Admin-2 web2 % python tttt<br>I&#39;m sorry, but your question is not clear. Could you please provide more details or context?</pre><h3>Integrating with tests</h3><p>All the logic begins with the get_object method, which is accessible to the final PageObjects through the base PageObject. The green items are new to the standard scheme.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*07hU8Oe6AVCqrPjWIes86g.png" /></figure><p>Thanks to RLS (Run-time Locators Storage), all subsequent tests within the test run will not fail on the problematic element. Another important aspect is the organization of the design-time locator storage. These are separate classes that are connected to functional PageObjects. They look something like this:</p><pre>class LLogin:<br><br>    @staticmethod<br>    def L_I18N_TEXTFIELD_LOGIN(lang=&#39;&#39;):<br>        &quot;&quot;&quot;login input field&quot;&quot;&quot;<br>        return (&#39;xpath&#39;, f&#39;//*[@e2e-id=&quot;I intentionally broke this locator&quot;]&#39;)</pre><p>Thanks to the naming convention for locators (the name starts with the prefix L_) and the convention for working with page objects (access to them is only done through Page.get_object(L_I18N_TEXTFIELD_LOGIN)), we can extract the locator name from the stack trace and save it in the Run-time Locators Storage.</p><p>Thanks to the docstring of the locator method (“””login input field”””), we have an elegant solution for where to store the human-readable description of the locator, which is then used for LLM inference.</p><p>Below is a raw log demonstrating self-healing in action. See WARNING level steps:</p><pre>2025-03-02 23:28:57 STEP WEB Client: Pick language<br>2025-03-02 23:28:57 STEP WEB Client: Authorize<br>2025-03-02 23:29:03 WARNING Web element with locator &quot;(&#39;xpath&#39;, &#39;//*[@e2e-id=&quot;I intentionally broke it&quot;]&#39;)&quot; not found within timeout, trying AI locator<br>2025-03-02 23:29:03 WARNING Problematic locator is L_I18N_TEXTFIELD_LOGIN for pages.login.PVersion2Login<br>2025-03-02 23:29:03 WARNING AI will try to find element locator using description: &quot;login input field&quot;<br>2025-03-02 23:29:36 WARNING Store AI-locator in cache for: &quot;pages.login.PVersion2Login.L_I18N_TEXTFIELD_LOGIN&quot; for subsequent tests<br>2025-03-02 23:29:36 WARNING Using AI-calculated locator &quot;(&#39;xpath&#39;, &#39;//input[@e2e-id=&quot;login-page.login-form.login-input&quot;]&#39;)&quot;<br>2025-03-02 23:29:36 STEP WEB Client: Enter code<br>2025-03-02 23:29:36 STEP Mailbox: Get confirmation code<br>PASSED                                                                                                                                                                                              [ 33%]<br>-------------------------------------------------------------------------------------------- live log teardown --------------------------------------------------------------------------------------------<br>2025-03-02 23:29:40 STEP ~~~~~END test_login_2fa~~~~~<br><br>tests/test_login.py::test_logout<br>--------------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------------<br>2025-03-02 23:29:40 STEP ~~~~~START test_logout~~~~~<br>---------------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------------<br>2025-03-02 23:29:50 STEP WEB Client: Pick language<br>2025-03-02 23:29:51 STEP WEB Client: Authorize<br>2025-03-02 23:29:51 WARNING Using AI-calculated locator from cache: &quot;(&#39;xpath&#39;, &#39;//input[@e2e-id=&quot;login-page.login-form.login-input&quot;]&#39;)&quot;<br>2025-03-02 23:29:55 STEP WEB Client: Open profile<br>2025-03-02 23:29:55 STEP WEB Client: Logout<br>2025-03-02 23:29:55 STEP ASSERT: User sees auth page</pre><p>For now, we are not making automatic commits to replace old locators with new ones generated by the LLM. Instead, we display them in the Allure report:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-FJsOJiNwnBRvRtnZLNi6w.png" /></figure><h3>Limitations</h3><ul><li>Simultaneous requests to the LLM for inference increase the TTFT (Time To First Token) for subsequent requests. For example, if you make three requests at once on our initial configuration (TTFT=35), the response for the first request will arrive in 35 seconds (OK), for the second in ~70 seconds, and for the third in ~105 seconds. This resembles a queue.</li><li>If you decide to use this approach with a configuration that results in a long TTFT, especially when many tests are running or when almost all locators fail, everything will queue up, and your Gitlab-like system will terminate the test pipeline due to a timeout.</li><li>Under limited hardware resources, you cannot simply connect all locators to the LLM — doing so could potentially create a queue due to multi-threaded test execution, even if your product is stable and developers allocate special properties for automated tests (like e2e-id in our case). To minimize this issue, a simple strategy can be employed: automatically count how many times each locator is used in a test run and connect only the most frequently used ones. For instance, when almost all your tests start from the login page, you will get the highest usage count for locators related to the login input field, password input field, and the “Login” button. By connecting these, you can prevent the failure of all tests at the very beginning.</li><li>Context size refers to the number of tokens in the prompt. The larger the context size, the higher the memory consumption. The smaller the context size, the more you will need to optimize to provide the LLM with a suitable portion of the HTML for inference. In our case, we preprocess the data — we remove unnecessary parts from the entire page_source provided by Selenium, such as style and script tags along with their content. We will likely further reduce the size for pages with a large number of elements. Our context size is 10,000 tokens. According to <a href="https://qwen.readthedocs.io/en/latest/getting_started/concepts.html">Qwen’s documentation</a>, 1 token is approximately 3–4 characters of English text.</li></ul><h3>Future prospects</h3><ul><li>Products that are in relatively active development can now have UI automated tests thanks to the self-healing approach.</li><li>Automated testing of relatively stable products will become more reliable due to the adaptability to changes in the application.</li><li>Automated testing of relatively stable products can cover more scenarios (for example, in testing desktop platforms, where access to system windows is sometimes required and limited by automation tools) through the integration of multimodal LLMs or a combination of several simpler models.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc1bb24f42ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>