<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[deTesters - Medium]]></title>
        <description><![CDATA[Test Automation - Medium]]></description>
        <link>https://medium.com/detesters?source=rss----57c73ce6102---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>deTesters - Medium</title>
            <link>https://medium.com/detesters?source=rss----57c73ce6102---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:17:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/detesters" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why you should start using agentic testing]]></title>
            <link>https://medium.com/detesters/why-you-should-start-using-agentic-testing-6b247396d6a7?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/6b247396d6a7</guid>
            <dc:creator><![CDATA[Tim Lolkema]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 14:11:10 GMT</pubDate>
            <atom:updated>2026-10-20T20:35:02.642Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YDBAWlN1U5Y6h21q.png" /><figcaption>Let’s explore!</figcaption></figure><p>You might be using AI coding agents already to write your automated tests.</p><p><a href="https://medium.com/detesters/using-agent-skills-to-write-playwright-tests-d0988aeccc93">In my previous blog, I showed how coding agents combined with a skills setup can help create E2E tests reliably, while following the conventions and style of your project.</a></p><p>But what if we could use these agents not just to write tests, but to actually test our applications?</p><p>It allows us to explore new functionality, investigate edge cases, and get feedback while we’re still developing a feature.</p><p>In this post we’ll explore agentic testing and why it should be part of your testing strategy. I’ll share what I’ve learned so far, and where I think these agentic quality checks can add value.</p><p><strong>It’s time to start adding autonomous quality checks in your project next to the scripted automation.</strong></p><p>Hopefully, it will inspire you to start experimenting with them in your own project as well. 😃</p><h3>What is agentic testing?</h3><p>Scripted test automation is very good at following a path we already know.</p><blockquote><em>Action &gt; Action &gt; Assert</em></blockquote><p>Agentic testing agents don’t need that path. They can inspect what is happening in the application and decide what to do next. They run in a loop, taking actions and evaluating every time until they reach their goal.</p><blockquote><em>Action &gt; Make LLM call, evaluate current state towards goal &gt; Next Action</em></blockquote><p>This makes agentic testing very flexible but also it will never be as <strong><em>fast</em></strong>, <strong><em>reproducible</em></strong> and <strong><em>accurate</em></strong> as predefined scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YB8lqCvUjpjwK6Bh.png" /><figcaption>Which path will you take?</figcaption></figure><h3>Why agentic testing is worth exploring</h3><p>Last year I hosted over 20 workshops on agentic AI. I’ve noticed many people want to get started but don’t know where to begin. Some teams are already using AI coding agents to write their automated tests, but very few actually use agentic AI for testing.</p><p>I think one reason is that it’s not immediately obvious where agentic testing adds value.</p><p>Autonomous agents are particularly useful for testing new functionality while it is still being developed. They can explore a feature, react to what they see, create the required test data, and investigate scenarios that were never explicitly scripted beforehand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kCcviZgdtt1HANt-.png" /><figcaption>Agentic testing to explore new features.</figcaption></figure><p>Agentic testing to explore new features</p><p>Even though an agent is slower than a scripted Playwright test, it can still provide feedback much faster than waiting for a manual check.</p><h3>When to run agentic testing checks</h3><p>I believe this type of agentic testing is especially useful during development itself.</p><p>Shift-left testing is all about getting feedback as early as possible. With AI agents, we can now make end-to-end exploratory testing much easier during development.</p><p>Instead of waiting until a feature is finished, you can use an agent to validate the implementation throughout development.</p><p>I really think this is great for the overall quality. It’s becoming much easier to do this many checks with very little effort before a pull request is even created!</p><p>Scripted automation complements this by providing fast, reliable, and repeatable regression checks in CI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4bDNlqgL7aB4SY9B.png" /><figcaption>Agentic testing during development and scripted automation for fast, repeatable regression checks in CI.</figcaption></figure><p>I also like that testing becomes conversational.</p><p>While developing a user story, I might ask an agent to test the functionality implemented so far. It will usually start with the main happy paths and report what it found. After reviewing the results, I can ask it to explore additional edge cases, focus on a particular scenario, or investigate something that looked suspicious.</p><p>That feedback loop makes exploratory testing much cheaper.</p><p>In the past, limited time often meant choosing which scenarios to test and which ones to skip. With autonomous agents, it becomes realistic to explore many more cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Hcw3VtKyzhkcnOK.png" /><figcaption>The power of agentic testing!</figcaption></figure><p>And this is not limited to functional testing. You can also ask the agent to inspect things such as keyboard navigation, accessibility behaviour, validation states, error handling, or other quality characteristics that might otherwise receive less attention.</p><p>I believe a good modern testing strategy should include both agentic and scripted test automation.</p><blockquote><em>This does not require developers to own all testing. In a more traditional workflow, testers can use the same approach once a feature reaches a test environment. Let agents handle execution and exploration while the tester focuses on directing and judging the evidence.</em></blockquote><h3>Trusting your AI agent</h3><p>But AI agents make mistakes, they hallucinate, fill in the blanks and don’t properly follow instructions.</p><p>How can we ever trust an AI testing agent?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2TckG84bLKFwZocw.png" /><figcaption>Can we trust him?</figcaption></figure><p>After running many agentic tests, I’ve learned that providing good context makes agents much more effective. Clear requirements and constraints help them understand what to test and reduce the number of assumptions they need to make.</p><p>But that does not mean I automatically trust the result.</p><p><strong>Trust comes from evidence.</strong></p><p>I want to see which scenarios the agent executed, which paths it followed, what it observed, and why it concluded that the behaviour was correct.</p><p>That evidence allows me to decide for myself how much I trust the result.</p><p>It also makes it much harder for the agent to take shortcuts. Instead of simply claiming that a scenario passed, it needs to show what it actually did.</p><p>This shifts my role from executing every test myself to reviewing whether the checks were sufficient and whether the evidence supports the conclusion.</p><p>If the checks are not sufficient, I will ask the agent to look at a few more edge cases, test a specific case, or provide more context to do another test.</p><h3>My agentic testing skill</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eM172xOTwJ4gvSWx.png" /><figcaption>Always read the instructions manual 😉</figcaption></figure><p>What does this look like in a real project?</p><p>I have a <strong><em>test-user-story</em></strong> skill that takes a user story or feature description as input and performs the exploratory testing for me. It can also reuse information from our existing scripted E2E setup, such as test data, users, and environment URLs.</p><p>The workflow looks roughly like this:</p><ul><li>It receives a user story or description of the functionality.</li><li>It derives a set of relevant test cases.</li><li>It asks which cases should be executed, against which environment, and what type of evidence should be captured.</li><li>It executes the selected scenarios using <strong><em>playwright-cli</em></strong></li><li>It generates a consistent test report containing the results and supporting evidence.</li></ul><p>The template itself is just HTML, together with a small script that fills it with the details from the test run. Whenever a report needs to be created, the agent calls that script with the collected results.</p><p>Instead of receiving a different style of output every time, I get a predictable report that is easy to scan and review. More importantly, the report leaves behind an artifact showing what the agent actually tested rather than only its final conclusion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0UhaK8tiAZpMv1BNmzRtQg@2x.png" /><figcaption>Nice reports at the end!</figcaption></figure><p>See the full skill at my GitHub</p><p><a href="https://github.com/tlolkema/demo-webshop/tree/main/.agents/skills/test-user-story">https://github.com/tlolkema/demo-webshop/tree/main/.agents/skills/test-user-story</a></p><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AxVUM88lx-gT4CZy.png" /><figcaption>Let’s goooooooooo!</figcaption></figure><p>For me, the biggest lessons with agentic testing are:</p><ul><li>Agentic testing will not replace scripted automation, but it can complement it really well. Scripted tests remain ideal for fast and reliable regression checks in CI.</li><li>Agentic testing adds value by exploring new functionality, testing edge cases, and providing feedback much earlier in development. A modern testing strategy should use both.</li><li>The key is not to blindly trust the agent, but to review the evidence it provides.</li><li>As (quality) engineers, we should embrace agentic testing and the opportunities it brings<strong>.</strong> Software development is evolving rapidly, and with it come exciting new possibilities. Let’s embrace these changes and actively help shape the future of testing!</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b247396d6a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/why-you-should-start-using-agentic-testing-6b247396d6a7">Why you should start using agentic testing</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Fine-tuning to LLM Evaluation: Building a Reliable LLM with MLX and Promptfoo]]></title>
            <link>https://medium.com/detesters/from-fine-tuning-to-llm-evaluation-building-a-reliable-llm-with-mlx-and-promptfoo-6e8cd19cd404?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/6e8cd19cd404</guid>
            <category><![CDATA[promptfoo]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[mlx]]></category>
            <dc:creator><![CDATA[Julian Boot]]></dc:creator>
            <pubDate>Wed, 04 Feb 2026 18:23:30 GMT</pubDate>
            <atom:updated>2026-02-04T18:23:00.677Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J_uIMbpKBq7vyGZXjQLcGg.png" /></figure><p>The most frequently asked question after I wrote my blog on <a href="https://medium.com/detesters/automated-llm-evaluation-with-promptfoo-771bbf1c055a">Automated LLM evaluation with Promptfoo</a> was: “For what AI implementations do we actually need LLM evaluation?”. This blog covered basic examples for testing deterministic and non-deterministic prompt outputs from models of OpenAI and Anthropic. To make it more understandable I would like to cover a practical use case of Large Language Models where LLM evaluation can be valuable.</p><p>Many companies struggle to adopt AI due to data privacy concerns and don’t want to rely on LLMs hosted by external providers. Running an LLM locally, in-house, helps address these issues by keeping sensitive data within the organization. Using a local open-source LLM also enables greater control over what the model knows, as domain-specific data can be incorporated through fine-tuning or other training methods. When using an open-source model and applying fine-tuning, it is important to validate that the LLM produces correct responses and that the domain data has been properly embedded. LLM evaluation plays a key role in this validation process. In this blog, I will show how to fine-tune and evaluate a local model for use as a chatbot.</p><h4>What is LLM fine-tuning?</h4><p>LLM fine-tuning is a machine learning method that is used to teach a pre-trained LLM information that the LLM did not learn before. This is done by training the LLM on a domain-specific dataset that is relevant to a particular application. This process is resource-intensive and can be time-consuming. So fine-tuning is most suitable when the contextual information that the LLM will need for a correct output does not change much over time. But what if the information does change a lot over time? In this case using RAG (Retrieval-Augmented Generation) might be better but will come at a cost of higher latency.</p><h4>Meet The Model Oven Bakery</h4><p>For LLM fine-tuning we need a dataset that will be used for training. The dataset that I will be using is a dataset of a fictitious bakery company called The Model Oven Bakery. The dataset contains information about the company and the products it sells, such as its location, contact details, and bakery offerings. During the fine-tuning process the model will learn domain-specific knowledge about the bakery and will be able to answer questions related to the business, much like a customer service chatbot.</p><h4>Fine-tuning a open-source LLM with MLX</h4><p>How can we fine-tune a LLM? We will be using <a href="https://github.com/ml-explore/mlx-lm">MLX</a>, an open-source machine learning framework developed by Apple.</p><p><strong>Warning:</strong> MLX is a machine learning framework developed by Apple and for Apple users with machines containing M series chips. As I mentioned before fine-tuning is a resource intensive process so Apple developed it to run as efficiently as possible specifically for Apple M series chip users. If you are not an Apple user or don’t have a Apple machine with a M series chip I would recommend using a different tool like Unsloth.</p><p>Here’s what we need to fine-tune a model:</p><ul><li>MLX</li><li>A open-source model from Hugging Face</li><li>A dataset to train the model on</li></ul><p>Let’s start with installing MLX in a directory of your choosing, we can use the following command:</p><pre>pip install mlx-lm</pre><p>Now we need to download a model from Hugging Face that we want to fine-tune. Hugging Face is an open-source platform where users can open-source LLMs. Kind of like GitHub but for LLMs. From Hugging Face I will be using the <a href="https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3">Mistral-7B-Instruct-v0.3</a> model. This is a 7B parameter model that can run on most modern machines. “7B” stands for “7 billion” and will tell you how big the model is. The more parameters the more demanding it will be to run on your machine. If you have an older machine you might want to use a smaller model of 3B-4B parameters.</p><p>To download the model to a local directory, we can use the Hugging Face CLI tool. Before running this CLI command you will need to have a Hugging Face account and be logged in by using a token. You can get a token by going to your Hugging Face account settings and creating a new token. I recommend downloading the model to a directory where you have installed MLX.</p><pre>hf auth login</pre><pre>hf download mistralai/Mistral-7B-Instruct-v0.3 --repo-type model --local-dir &lt;your-directory&gt;</pre><p>Next we need a dataset to train the model. MLX will need two files with training data: a <strong>train.jsonl</strong> and a<strong> valid.jsonl</strong> file. I recommend creating a “data” directory in the root of your project where you have installed MLX. The <strong>train.jsonl</strong> file will contain the data that will be used to train the model and the <strong>valid.jsonl</strong> file will contain the data that will be used to validate the model during training. The JSONL files will need to contain JSON objects containing prompt and response pairs. This looks like this:</p><pre>{&quot;prompt&quot;: &quot;What is the name of the bakery?&quot;, &quot;completion&quot;: &quot;The name of the bakery is The Model Oven&quot;}<br>{&quot;prompt&quot;: &quot;What is the address of the bakery?&quot;, &quot;completion&quot;: &quot;The address of the bakery is 342 Tasty Street in Bakersfield&quot;}<br>{&quot;prompt&quot;: &quot;What is the phone number of the bakery?&quot;, &quot;completion&quot;: &quot;The phone number of the bakery is 555-1234&quot;}<br>{&quot;prompt&quot;: &quot;What kind of shop are you?&quot;, &quot;completion&quot;: &quot;We are a bakery shop that sells bread, pastries, and baked goods. We also sell coffee and tea.&quot;}</pre><p>I have created a dataset of around 50 prompt and response pairs for the train.jsonl file and 10 prompt and response pairs for the valid.jsonl file. Make sure you use different data for the validation than what is used in the training set.</p><p>Now we can train the model. We will be using a training method called LoRA (Low-Rank Adaptation). LoRA is a method that is used to fine-tune a model by adding a small amount of trainable parameters to the model. This is a good method for fine-tuning local models because it is less resource intensive than fine-tuning the entire model. The following command will train the model:</p><pre>mlx_lm.lora --model &lt;your-model-path&gt; --data &lt;your-training-data-path&gt; --train</pre><p>This command will train the model with default settings but many parameters can be tuned to get the best performance. After the training is complete a adapters directory with adapter files will be created containing the fine-tuning adjustments based on the training. To test these adjustments we can prompt the model with the created adapters:</p><pre>mlx_lm.generate --model &lt;your-model-path&gt; --adapter-path &lt;your-adapters-path&gt; --prompt &quot;What is the name of the bakery?&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*XoG42c5J-Y_GoJRAJM0Klw.gif" /></figure><p>If you are happy with the results you can merge the adapters with the model. This is done with using the “fuse” command:</p><pre>mlx_lm.fuse --model &lt;your-model-path&gt; --save-path &lt;save-path-for-your-fused-model&gt; --adapter-path &lt;your-adapters-path&gt;</pre><p>After this we can test the model again to see if the adapters were merged correctly:</p><pre>mlx_lm.generate --model &lt;your-fused-model-path&gt; --prompt &quot;What is the name of the bakery?&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*v2J8KkCUExEFCBstiW40Dg.gif" /></figure><p>So now we have fine-tuned the model and we can start evaluating it. Before we get there we need to be able to interact with the model. For this we will run the model as a local server with a REST API. There are various ways of doing this. I choose to use <a href="https://lmstudio.ai/">LM Studio</a> because it is open-source, easy to use and supports models that have a MLX format. You can also use the server command from MLX to run the model as a local server but I chose for using LM Studio since it is more convenient for users on other operating systems. Other options are Ollama and llama.cpp, but require models in a “gguf” format. Converting a MLX trained model to “gguf” is possible with the converter script that is included in llama.cpp.</p><p>LM Studio has an easy to use GUI that will allow the user to import and run LLMs. On the developer tab you can find a server switch that will allow you to run the model as a local server. The API interface is similar to the OpenAI API so it is also easy to integrate with Promptfoo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yi8KnV_7G9aaOykKhHXDrQ.png" /></figure><h4>Setting up Promptfoo to evaluate the fine-tuned LLM</h4><p>For installing and understanding Promptfoo I refer you to my <a href="https://medium.com/detesters/automated-llm-evaluation-with-promptfoo-771bbf1c055a">previous blog</a>. In short Promptfoo is a open-source LLM evaluation tool with wide support of options for automated evaluation.</p><p>We start by creating a config for Promptfoo to interact with our fine-tuned LLM running on a local server. Because the API interface is similar to the OpenAI API we can use the same provider config as for the OpenAI API. We only need to change the API base URL to the URL of the local server and add a fake API key. The API key is not used by the LLM but is required by the Promptfoo OpenAI provider config because this is based on the OpenAI API and that always requires an API key.</p><pre>providers:<br>  - id: openai:chat:default<br>    config:<br>      apiBaseUrl: http://localhost:1234/v1<br>      apiKey: fake-key </pre><p>For continuing the setup we will create a prompt template so the user and system prompts can be parsed into a format that can be used by Promptfoo. We can use a json file where we can define a messages object for the user and system prompts where the prompts will be added as content. This looks like this:</p><pre>[<br>    {<br>        &quot;role&quot;: &quot;system&quot;,<br>        &quot;content&quot;: &quot;{{system_prompt}}&quot;<br>    },<br>    {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;{{user_prompt}}&quot;<br>    }<br>]</pre><p>In our promptfooconfig.yaml file we then can use the “file://<strong>” </strong>prefix to load the prompt template:</p><pre>prompts:<br>  - file://prompt.json</pre><h4>Choosing metrics</h4><p>Before we create the tests we are going to determine the metrics that we want to use to evaluate the model. We are going to evaluate the model with graders so we need metrics where we can measure specific aspects of the outputs of the LLM. These graders are LLM-as-a-Judge Promptfoo assertions that will use a LLM to judge the quality of the LLM output.</p><p>The use case of our fine-tuned model is to act as a customer service chatbot that can answer questions about the bakery. So we will be using metrics that are relevant to this use case. There are various metrics that are used for LLM evaluation based on benchmark studies and evaluations by LLM providers such as Open AI and Anthropic. Because implementations of AI varies a lot there is not one set of metrics that is universally used. So I have made a list of relevant metrics based on metrics I found during my research and on experiments I have done with evaluation:</p><ul><li><strong>Factuality</strong>: Is the information correct in the answer?</li><li><strong>Completeness</strong>: Are all parts of the prompt addressed in the answer?</li><li><strong>Coherence</strong>: Is the response logical and well-structured?</li><li><strong>Hallucination: </strong>Does the response contain hallucinations or fabricated information?</li><li><strong>Tone: </strong>Does the answer sound like a helpful and friendly response?</li></ul><h4>Creating the tests</h4><p>We will be using the tests object to define the tests. Each test will contain a <strong>user_prompt</strong> variable that will be used for the user prompt. The system prompt will be the same for each test so it is added to the defaultTest object. Next we add the assertions that we want to use to evaluate the model. These assertions are the metrics that we defined earlier:</p><pre>defaultTest:<br>  vars:<br>    system_prompt: &quot;You are a helpful bakery assistant of the The Model Oven bakery. You will always answer the question in a helpful and friendly tone.&quot;<br><br>tests:<br>  - vars:<br>      user_prompt: &quot;What is the name of the bakery?&quot;<br>    assert:<br>      - type: factuality<br>        value: &quot;The name of the bakery is The Model Oven&quot;<br>        metric: accuracy<br>      - type: llm-rubric<br>        value: |<br>          The response fully addresses the customer&#39;s question.<br>          All aspects of the inquiry are covered without leaving important parts unanswered.<br><br>          The customer&#39;s question: {{user_prompt}}<br>        threshold: 0.8<br>        metric: completeness<br>      - type: llm-rubric<br>        value: |<br>          Customer question: {{user_prompt}}<br><br>          Evaluate whether the response contains hallucinations or fabricated information<br>        threshold: 1<br>        metric: hallucination<br>      - type: g-eval<br>        value: |<br>          Coherence: The response is logically organized, easy to follow,<br>          and presents information in a clear, structured manner.<br>        threshold: 0.8<br>        metric: coherence<br>      - type: llm-rubric<br>        value: |<br>          The response has a helpful, and friendly tone appropriate for customer service.<br>          It is welcoming, professional, and makes the customer feel valued.<br>        threshold: 0.8<br>        metric: tone</pre><p>For the metrics I have used different types of Promptfoo graders. A short explanation what these grader types can do:</p><ul><li><strong>factuality</strong>: Evaluates factual consistency between the LLM output and a reference statement. It uses OpenAI’s public evals prompt to determine if the output is factually consistent with the provided reference value. This is ideal for checking if the model provides accurate information.</li><li><strong>llm-rubric</strong>: Promptfoo’s general-purpose grader that uses an LLM to evaluate outputs against custom criteria or rubrics. Its highly flexible and can be used to assess various aspects of the response by providing specific evaluation criteria. You can set a threshold to determine if the output passes or fails based on the score.</li><li><strong>g-eval: </strong>Uses chain-of-thought prompting to evaluate outputs against custom criteria following the G-Eval framework. This method is particularly effective for evaluating more complex criteria like coherence and structure, as it uses a reasoning process to arrive at its evaluation.</li></ul><p>Some assertions I have allowed a margin of error by setting a threshold that is “0.8”. For the hallucination metric I have set the threshold to “1” because the model should never hallucinate.</p><h4>What prompts to evaluate?</h4><p>When evaluating the fine-tuned model, it is important to test a diverse range of prompts that cover different aspects of the model and its intended use case. For The Model Oven bakery chatbot I have used specific training data that is relevant to the fictitious bakery. So it is important to validate if the domain data is successfully embedded into the model. Besides the data the model is fine-tuned on it should also deal with prompts that are not related to the trained data. The model still needs to be able to handle these prompts correctly and will need to stay within it’s role as a bakery chatbot.</p><p><strong>Basic factual prompts:</strong></p><pre>user_prompt: &quot;What is the name of the bakery?&quot;<br>user_prompt: &quot;What is the address of the bakery?&quot;</pre><p>This evaluates the model’s ability to recall fundamental information from the training data. These are the most basic tests to verify that the fine-tuning process successfully embedded training data into the model.</p><p><strong>Complex factual prompts:</strong></p><pre>user_prompt: &quot;When are you open and where are you located?&quot;</pre><p>A prompt that will require the model to deal with different pieces of information like opening hours, exceptions when the bakery is closed and the location of the bakery.</p><p><strong>Important domain-related safety prompts:</strong></p><pre>user_prompt: &quot;Do you have anything for people with gluten intolerance?&quot;<br>user_prompt: &quot;I have a severe nut allergy, is it safe to eat here?&quot;</pre><p>These prompts are important for the model to respond correctly because they are crucial for the safety of the users.</p><p><strong>Boundary context prompts:</strong></p><pre>user_prompt: &quot;Do you sell computers?&quot;<br>user_prompt: &quot;Can you give me medical advice?&quot;<br>user_prompt: &quot;I hate you!&quot;</pre><p>These prompts verify whether the model stays within its domain knowledge and if it can handle out of context prompts correctly. Also it is important to evaluate whether the model can handle inappropriate requests. You can also consider these prompts as boundary tests.</p><h4>Running the tests and evaluating the results</h4><p>When the Promptfoo tests are created with the desired prompts we can run them by using the following command:</p><pre>npx promptfoo@latest eval</pre><p>After running the results can be displayed in detail by running the web view command:</p><pre>npx promptfoo@latest view -y</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yo2diN3Y3RSDOQgY6HTJ1g.png" /></figure><p>As you can see some of the tests fail. This means that the model is not able to handle some of the prompts correctly. We need to investigate why this is the case. In my case the model scored poorly on the “Tone” and “Hallucination” metrics. Things that can be concluded is that the training data does not cover all the information that is needed for the model to handle the prompts. An easy step that can be taken to improve the model outputs is by improving the system prompt to include missing information or to make the model sound more helpful. For missing information this will only work for small pieces. For more significant amounts of missing information in the training data it is better to add more training data and fine-tune the model again. This is a process that can be repeated until it is production ready.</p><p>Another way to improve the model outputs is to build another layer around the model that will provide missing context information. This can be done by using a RAG (Retrieval-Augmented Generation) system that will retrieve relevant information from a knowledge base and provide it to the model.</p><h4>Conclusion</h4><p>I hope that this blog has given you a good overview of how to use fine-tuning with MLX and LLM evaluation. It is important to understand that LLM evaluation becomes more important when you take a base model and provide relevant context to it. While base models are already evaluated by providers on general benchmarks, the most critical evaluation work happens when you customize the model for your specific context. Just as you wouldn’t deploy a web application to production without testing the code and integrations, you shouldn’t deploy a fine-tuned model without evaluating how well your training data has been integrated. By using the correct metrics and prompts in your evaluations you can systematically identify missing training data or other issues and iterate until your model is production-ready.</p><p><strong>What about security?</strong> Yes, testing security is also very important to consider. Things such as prompt injection might cause your model to ignore its system prompt, reveal confidential training data or give outputs outside its intended scope. Promptfoo supports adversarial testing, which can help identify these vulnerabilities. I have deliberately kept security testing outside the scope because this topic is too extensive to cover in this blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e8cd19cd404" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/from-fine-tuning-to-llm-evaluation-building-a-reliable-llm-with-mlx-and-promptfoo-6e8cd19cd404">From Fine-tuning to LLM Evaluation: Building a Reliable LLM with MLX and Promptfoo</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated LLM Evaluation with Promptfoo]]></title>
            <link>https://medium.com/detesters/automated-llm-evaluation-with-promptfoo-771bbf1c055a?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/771bbf1c055a</guid>
            <category><![CDATA[promptfoo]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <dc:creator><![CDATA[Julian Boot]]></dc:creator>
            <pubDate>Mon, 22 Sep 2025 17:39:56 GMT</pubDate>
            <atom:updated>2025-09-22T17:39:52.651Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HZ60fNyQxeTjRLhD2QDx5Q.png" /></figure><p>With AI being more implemented into applications there are new challenges for Quality Assurance (QA). While building traditional software has well-known testing practices, testing AI applications is something else. Unlike conventional code that follows predictable logic, AI systems are probabilistic, context-dependent, and can produce different outputs for the same input. A prompt that works perfectly in one setup may fail when context is added or a different Large Language Model (LLM) is used. At the same time not every LLM is the same. LLMs are created/trained differently, some are more lightweight, offering faster and cheaper responses at the expense of output quality. Others have more advanced reasoning capabilities to generate more complex outputs but will be more expensive and slower. I noticed this a lot during my experiments building agents, workflows and a project at a client where AI is used to generate metadata.</p><p>So how can we ensure that the AI applications/agents will be reliable, fast and cost efficient?</p><p>In this blog I would like to show some entry level examples on how you can use automated LLM evaluation with Promptfoo to measure and test the outcomes of prompt responses of various LLMs.</p><h4>What is Promptfoo?</h4><p>Promptfoo is an open-source LLM evaluation tool designed to automatically test and measure the quality of AI-generated outputs. Promptfoo offers A/B comparison of prompts and models, cost and latency tracking, and quality scoring using various evaluation methods including LLM-as-a-judge. To provide clear insights into test results, there’s also a simple but informative web view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FnNhq6F-PA-wceINvDzwQQ.png" /><figcaption>Promptfoo web view</figcaption></figure><p>To install Promptfoo:</p><pre>npx promptfoo@latest</pre><h4>Setting up a Promptfoo test</h4><p>Our goal is to set up a test which we can use to input multiple prompts and test the output of various models. Promptfoo tests are written in YAML and here’s a simple test setup:</p><pre>description: &quot;This is a test setup for a promptfoo test&quot;<br><br>prompts:<br>  - &quot;Calculate the sum of {{number1}} and {{number2}}&quot;<br><br>providers:<br>  - openai:gpt-5-nano<br>  - openai:gpt-5<br>  - anthropic:messages:claude-sonnet-4-20250514</pre><p>Promptfoo offers various options to setup tests. First a description can be defined to describe a test scenario. A prompt is also needed to use as input to the LLM’s. The prompt includes two variables that we later can use to parse different numbers for different test cases. Next, we define the providers and their models for our tests. These are well-known models of Anthropic and OpenAI (don’t forget to setup your API keys for these providers).</p><p>Now that we have the first part of the setup we need to add assertions to validate the model outputs. I will divide these validations in two types:</p><ul><li>deterministic validation</li><li>non-deterministic validation.</li></ul><h4>Deterministic validation</h4><p>Deterministic validation uses exact matching rules to verify LLM outputs. With Promptfoo, you can check if responses contain specific keywords, match exact text patterns, or follow predefined formats. This approach is ideal for testing structured outputs such as JSON responses, specific answer formats, or scenarios requiring precise control over model outputs. In this example, we verify that the LLM correctly calculates the sum.</p><pre>description: &quot;This is a deterministic promptfoo test&quot;<br><br>prompts:<br>  - &quot;Calculate the sum of {{number1}} and {{number2}}. Only output the final answer.&quot;<br><br>providers:<br>  - openai:gpt-5-mini<br>  - openai:gpt-5<br>  - anthropic:messages:claude-sonnet-4-20250514<br><br>tests:<br>  - vars:<br>      number1: &quot;1&quot;<br>      number2: &quot;2&quot;<br>    assert:<br>      - type: equals<br>        value: &quot;3&quot;<br>  - vars:<br>      number1: &quot;5&quot;<br>      number2: &quot;4&quot;<br>    assert:<br>      - type: equals<br>        value: &quot;9&quot;</pre><p>In this case we know what the output will be so a “equals” assertion is added that will do a exact string match. Now we are going to run the test:</p><pre>promptfoo eval -c your_promptfoo_test_yaml</pre><p>To open the web view:</p><pre>promptfoo view --yes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O8qzn7204XF0NbX8uRE_kg.png" /><figcaption>Web view of test results</figcaption></figure><h4>Non-deterministic validation</h4><p>We cannot depend on deterministic checks if it is unknown what the exact output will be (non-deterministic). With Promptfoo’s LLM-as-a-Judge feature, you can have one model evaluate another model’s output. This method is suited for automated evaluation of non-deterministic outputs. In this example, we’ll prompt the LLM to write a news article about a topic, which we’ll define as a variable.</p><pre>description: &quot;This is a non-deterministic promptfoo test&quot;<br><br>prompts:<br>  - &quot;Write a short news article about a {{topic}} fact&quot;<br><br>providers:<br>  - openai:gpt-5-mini<br>  - openai:gpt-5<br>  - anthropic:messages:claude-sonnet-4-20250514<br><br>tests:<br>  - vars:<br>      topic: &quot;cat&quot;<br>    assert:<br>      - type: llm-rubric<br>        value: The article should be about a cat fact<br>  - vars:<br>      topic: &quot;dog&quot;<br>    assert:<br>      - type: llm-rubric<br>        value: The article should be about a dog fact</pre><p>In the example above I have setup two testcases with each a “llm-rubric” assertion. This assertion is Promptfoo’s general purpose LLM-as-a-Judge assertion. Promptfoo offers multiple types of LLM-as-a-Judge assertions for specific purposes. The value contains instructions for the LLM-judge on what to evaluate. In the results the LLM-judge will also give a reason for it’s evaluation result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u5ioxjnWdZPrJWAmAzEqyQ.png" /><figcaption>Web view of test results</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rFnA-0vnG8pekPTOc6CoEQ.png" /><figcaption>Evaluation result of llm-rubric assertion</figcaption></figure><h4>Grading outputs</h4><p>When evaluating non-deterministic outputs, there’s typically no clear pass or fail result due to a large gray area where outputs may be partially correct. So when dealing with these outcomes it is best to have a measurement of the output quality. Instead of prompting the LLM-judge to judge if a output is true or false we will give it a prompt to grade the quality of the output and add a threshold on the score so it will fail the test when the score is too low.</p><pre>description: &quot;This is a non-deterministic scoring promptfoo test&quot;<br><br>prompts:<br>  - &quot;Write a short news article about a {{topic}} fact&quot;<br><br>providers:<br>  - openai:gpt-5-mini<br>  - openai:gpt-5<br>  - anthropic:messages:claude-sonnet-4-20250514<br><br>tests:<br>  - vars:<br>      topic: &quot;cat&quot;<br>    assert:<br>      - type: llm-rubric<br>        value: |<br>          Evaluate the quality of the news article about a cat fact. Grade with a score based on the following criteria:<br>          - Relevance to the topic (cat fact) - does it actually discuss cat-related information?<br>          - Article structure - does it follow proper news article format with clear introduction, body, and conclusion?<br>          - Factual accuracy - is the cat fact presented accurate and believable?<br>          - Readability - is the writing clear, engaging, well-written and without spelling errors?<br>          - Completeness - does it provide sufficient detail about the cat fact?<br><br>          Deduct points for each (minor) issue found.<br>        threshold: 0.8<br>  - vars:<br>      topic: &quot;dog&quot;<br>    assert:<br>      - type: llm-rubric<br>        value: |<br>          Evaluate the quality of the news article about a dog fact. Grade with a score based on the following criteria:<br>          - Relevance to the topic (dog fact) - does it actually discuss dog-related information?<br>          - Article structure - does it follow proper news article format with clear introduction, body, and conclusion?<br>          - Factual accuracy - is the dog fact presented accurate and believable?<br>          - Readability - is the writing clear, engaging, well-written and without spelling errors? And it does not contain<br>          - Completeness - does it provide sufficient detail about the dog fact?<br><br>          Deduct points for each (minor) issue found.<br>        threshold: 0.8</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L7soFfjiHR_rPmWLo5bTpA.png" /><figcaption>Web view of test results</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nasOGrqasTmoMaMrGEd4tQ.png" /><figcaption>Evaluation result of llm-rubric assertion</figcaption></figure><h4>What’s next?</h4><p>Based on the examples above the next things you can do with Promptfoo:</p><ul><li>Fine-tune prompts and LLM settings based on test results.</li><li>Add latency and token cost assertions.</li><li>Add custom providers to test agents or workflows. Promptfoo can execute Python or JavaScript files containing agents or workflows.</li><li>Implement in CI/CD pipeline for automated regression testing.</li><li>Automatic model selector for your AI application. If you have made the model that is used configurable you could potentially build a model selector in a CI/CD pipeline which will configure the best model based on the Promptfoo test results for the application deployment.</li></ul><h4>Conclusion</h4><p>Measuring the quality of LLM outputs is challenging but is made far more insightful with Promptfoo. Remember to use different strategies for deterministic and non-deterministic outputs as I demonstrated in the examples. For non-deterministic validations like LLM-as-a-Judge keep in mind this does not 100% guarantee that the LLM outputs are valid. It does however provide developers and QA with a tool to set a baseline to measure quality and identify regression issues. I would say for now this is the best way to automatically test non-deterministic outputs, but be careful about what you will let the LLM-judge evaluate, since the judge itself is an LLM.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=771bbf1c055a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/automated-llm-evaluation-with-promptfoo-771bbf1c055a">Automated LLM Evaluation with Promptfoo</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The role of AI agents in test automation]]></title>
            <link>https://medium.com/detesters/the-role-of-ai-agents-in-test-automation-e59338099178?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/e59338099178</guid>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[playwright-test]]></category>
            <dc:creator><![CDATA[Tim Lolkema]]></dc:creator>
            <pubDate>Mon, 22 Sep 2025 17:39:42 GMT</pubDate>
            <atom:updated>2025-09-22T17:39:23.489Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muhY0TQcx1BinoXshnXWIQ.png" /><figcaption>Autonomous AI agents can do the testing for you</figcaption></figure><h3>What are AI agents?</h3><p>What happens when AI doesn’t just respond to your questions, but takes initiative and acts autonomously? AI agents are changing what AI can do without us.</p><p>But what are AI agents?</p><p>When you interact with ChatGPT or other chatbots you ask it your question and it will respond. AI agents, however, take the next step by autonomously taking actions, making decisions or interact with external systems.</p><blockquote>An AI agent is a system that works <em>autonomously to accomplish a goal.</em></blockquote><p>At the AutomationSTAR conference in Berlin in 2023 I was inspired by <a href="https://www.linkedin.com/in/marcelveselka/">Marcel Veselka</a> on the role that AI agents can hold in the future of QA.</p><p>Since then I have been playing around with agentic AI and there have been some big developments. In this blogpost I’d like to explore what role autonomous AI agents can have in QA.</p><p>To keep the blog focussed on the possibilities I’m deliberately ignoring any ethical, security or other valid concerns for not introducing AI in your project 😃</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qnmX_yd7IqRU3vKfs2B8_A.png" /><figcaption>We’ve progressed from manual tools to automated processes and are now entering the era of intelligent and autonomous agents.</figcaption></figure><h3>My previous agentic workflow demo</h3><p>To show what these autonomous agents might do in the future of quality assurance, I created a demo last year that showcased a fully autonomous pull request review. It basically mimicked a simplified version of what a manual tester would do:</p><ul><li>First, it figured out which test cases to run by reading the context of the pull request or a Jira issue.</li><li>Then, it actually performed these test cases on the preview environment using an autonomous browser agent.</li><li>Finally, it reported the results directly on the pull request.</li></ul><blockquote><a href="https://github.com/tlolkema/autonomous-pr-test"><em>https://github.com/tlolkema/autonomous-pr-test</em></a></blockquote><p>While the demo was very simplified it does show the potential of using autonomous agents in your development pipelines.</p><h3>The current state of agentic AI systems</h3><p>In the months following my demo, the major AI companies all began rolling out their agentic AI systems.</p><p>Some examples are:</p><ul><li>Anthropic came with the release of <a href="https://www.anthropic.com/news/3-5-models-and-computer-use">Computer use</a>, <a href="https://www.anthropic.com/news/model-context-protocol">Model Context Protocol (MCP)</a> and recently <a href="https://www.anthropic.com/news/claude-3-7-sonnet">Claude Code</a></li><li>OpenAI released <a href="https://openai.com/index/introducing-operator/">Operator</a></li><li>Google released <a href="https://deepmind.google/technologies/project-mariner/">Project Mariner</a></li></ul><p>Next to these there have been some exciting open source projects like:</p><ul><li><a href="https://browser-use.com/">Browser Use</a></li></ul><p>OpenAI, Anthropic, Google xAI and other players have all come out with their reasoning models. The idea is that the better the reasoning the better these models will be able to instruct AI agents.</p><p>Right now (<em>March 2025</em>), AI agents controlling computers or browsers aren’t quite at human level yet, but the list of tasks they can handle is growing every single day.</p><blockquote>OpenAI Operator:<br>“In these benchmarks CUA achieved a 58.1% success rate on WebArena and an 87% success rate on WebVoyager for web-based tasks. While CUA achieves a high success rate on WebVoyager, where most tasks are relatively simple, CUA still needs more improvements to close the gap with human performance on more complex benchmarks like WebArena.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mBlPNY_emY-_JBfUS73mnw.png" /><figcaption>There have been major advancements in AI agents.</figcaption></figure><h3>Model Context Protocol (MCP)</h3><p>One of the exciting developments in agentic AI is Anthropic’s <a href="https://www.anthropic.com/news/model-context-protocol">Model Context Protocol (MCP)</a>. MCP aims to standardize how LLMs connect to various tools.</p><p>It’s not tied to just Anthropic’s Claude model or any specific LLM and it can communicate with a wide range of tools already.</p><p>An MCP server is basically a set of instructions for calling a specific tool. They’re very simple to create, which explains why there’s already a substantial list of supported MCP servers from both Anthropic and the community.</p><p>You’ll find MCP servers for automating tools like Puppeteer and Playwright, interacting with Git, connecting to search engines, or working with several popular APIs.</p><blockquote><a href="https://github.com/modelcontextprotocol/servers"><em>https://github.com/modelcontextprotocol/servers</em></a></blockquote><blockquote><em>“MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardised way to connect your devices to various peripherals and accessories, MCP provides a standardised way to connect AI models to different data sources and tools.”</em></blockquote><p>Let’s take a look how the official Puppeteer MCP server looks like 🚀</p><blockquote><a href="https://github.com/modelcontextprotocol/servers/blob/main/src/puppeteer/index.ts"><em>https://github.com/modelcontextprotocol/servers/blob/main/src/puppeteer/index.ts</em></a></blockquote><p>Following the MCP protocol we can define the tools the LLM can call.</p><p>In this example the <strong><em>click</em></strong> method of Puppeteer:</p><pre>  {<br>    name: &quot;puppeteer_click&quot;,<br>    description: &quot;Click an element on the page&quot;,<br>    inputSchema: {<br>      type: &quot;object&quot;,<br>      properties: {<br>        selector: { type: &quot;string&quot;, description: &quot;CSS selector for element to click&quot; },<br>      },<br>      required: [&quot;selector&quot;],<br>    },<br>  },</pre><p>And the implementation of this click method:</p><pre>    case &quot;puppeteer_click&quot;:<br>      try {<br>        await page.click(args.selector);<br>        return {<br>          content: [{<br>            type: &quot;text&quot;,<br>            text: `Clicked: ${args.selector}`,<br>          }],<br>          isError: false,<br>        };<br>      } catch (error) {<br>        return {<br>          content: [{<br>            type: &quot;text&quot;,<br>            text: `Failed to click ${args.selector}: ${(error as Error).message}`,<br>          }],<br>          isError: true,<br>        };<br>      }</pre><p>As you can see, creating an MCP server is actually pretty simple 😃</p><p>And there’s a good chance someone has already built one for your favourite tool!</p><p>Will MCP become the standard going forward? Or will we see more closed-source AI agents take over? That’s still up in the air. Only time will tell which approach wins out in the end.</p><h3>Autonomous testing vs scripted test automation</h3><p>Will autonomous testing fully replace scripted test automation?</p><p>Not in the short term I think.</p><p>Autonomous systems will never be as <strong><em>fast</em></strong>, <strong><em>reproducible</em></strong> and <strong><em>accurate</em></strong> as scripted test automation. This is because AI agents continuously have to evaluate their actions and determine what their next step should be to achieve their goal.</p><p>While scripted automation follows a predetermined path with predictable timing and outcomes, AI agents must constantly process, analyze, and decide — impacting speed and consistency.</p><p>Because of this, predetermined scripts will always reach their goal faster and can be fully repeatable. These are important factors for automated regression testing.</p><p>I expect scripted test automation to remain the primary method for regression testing.</p><p>AI agents can, however, play a valuable role in creating these scripts themselves after a successful test to expand the existing test suite. They could observe their own successful test paths and generate optimized scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2py6o30b49xipXz80t95g.png" /><figcaption>Scripted test automation can go straight to the finish line. While autonomous AI agents will evaluate every step.</figcaption></figure><h3>Where do autonomous AI agents shine?</h3><p>AI agents really shine when testing new features.</p><p>Imagine this: you’ve just built a new feature and immediately several AI agents jump into action. One agent focuses on checking accessibility issues, another tests the core functionality, and a third looks specifically for security problems.</p><p>This happens way faster than manual testing in these areas and gives you quick feedback on what needs fixing.</p><p>AI agents are especially helpful for tricky automation areas like testing screen readers for accessibility. When there is a different page structure, automation around screen readers might fail when following a predefined set of instructions. An AI agent however can still evaluate whether this experience works properly by directly assessing the situation against accessibility guidelines rather than following rigid test steps.</p><h3>Future</h3><p>I think the role of QA engineers will evolve as more testing gets handled by AI agents. QA engineers will have to determine the quality of agent-based testing and design the integration of these agents into CI/CD pipelines.</p><p>In the near future, I expect we’ll see a mixture of types of testing.</p><ul><li>Scripted test automation will continue handling regression testing where speed and reproducibility matter most.</li><li>AI agents will tackle complex exploratory scenarios and edge cases that traditional automation struggles with.</li><li>We’ll likely see a reduction in manual exploratory testing as agents become more capable of discovering unexpected issues.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v8yfLTUS9igUzkZBCYG7dQ.png" /><figcaption>A group of AI agents.. all working together performing their specialised tasks.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e59338099178" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/the-role-of-ai-agents-in-test-automation-e59338099178">The role of AI agents in test automation</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cypress of toch Playwright?]]></title>
            <link>https://medium.com/detesters/cypress-of-toch-playwright-f9e2a080de42?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/f9e2a080de42</guid>
            <category><![CDATA[playwright-test]]></category>
            <category><![CDATA[cypress]]></category>
            <dc:creator><![CDATA[Joep]]></dc:creator>
            <pubDate>Wed, 10 Jan 2024 14:58:54 GMT</pubDate>
            <atom:updated>2024-01-10T14:53:47.383Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*9ACSILmihlUaUcqsO08h9A.jpeg" /></figure><p>Is Cypress of Playwright beter in het automatiseren van E2E testen? Welk van deze twee test frameworks moet ik kiezen? Dit zijn vragen die ik de laatste tijd vaak krijg. Om deze vraag te beantwoorden moet er dieper worden gekeken naar de succesfactoren van Cypress en Playwright.</p><p>Het vergelijken van beide test frameworks kan op meerdere punten. Ik beperk mij tot de basisfunctionaliteit zoals rapportage, documentatie, netwerkinteracties en mocking. Daarnaast zoom ik in op de unieke functionaliteiten van Cypress en Playwright.</p><p>Welk test framework is op welke punt de winnaar en is er überhaupt een duidelijke eindwinnaar aan te wijzen? Dat lees je in deze blog.</p><h3>Goede rapportage voor inzichtelijke feedback</h3><p>Inzicht in de testen en testresultaten is key. Zodra testen falen, moet het direct duidelijk zijn wat er misgaat en waardoor deze fout wordt veroorzaakt. Cypress biedt een betaalde dienst waarmee gebruikers toegang krijgen tot Cypress Dashboard. In dit dashboard worden alle metrieken van de uitgevoerde testruns opgeslagen en weergegeven. Denk hierbij aan doorlooptijd, flakiness percentage, top failures, enzovoort. Naast de video’s en screenshots kunnen de testruns met ‘test-replay’ worden afgespeeld in de browser. Doordat de gefaalde test stap voor stap kan worden doorlopen en de gebruiker gebruik kan maken van devTools, is de oorzaak vaak eenvoudig te achterhalen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bNRcw5367jxgc-i-QOFPIQ.jpeg" /><figcaption>Cypress Dashboard <a href="https://docs.cypress.io/guides/cloud/introduction">https://docs.cypress.io/guides/cloud/introduction</a></figcaption></figure><p>Zonder dashboard heeft de gebruiker de keuze uit Cypress Mocha-, Teamcity- of JUnit-reporters. Deze rapporten kunnen uitgebreid worden met screenshots en video’s. Echter het ontbreken van ‘test-replay’ en de missende inzage van de DOM en het netwerkverkeer maakt het debuggen een stuk lastiger.</p><p>Playwright biedt out-of-the-box een breed scala aan rapportagetools aan. Standaard heeft het al een uitgebreid HTML-rapport inclusief ‘trace-viewer’. Hiermee kan, net als bij Cypress, de testrun in de browser stap voor stap worden afgespeeld. Daarnaast ondersteunt Playwright als ook Cypress ‘test-annotations’ waarmee de rapporten eenvoudig verrijkt kunnen worden met extra informatie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULhmzcve1FbbmRM73numcQ.png" /><figcaption>Playwright trace viewer — <a href="https://playwright.dev/docs/best-practices">https://playwright.dev/docs/best-practices</a></figcaption></figure><p>Cypress Dashboard is echt een unique selling point wat betreft rapportage. Echter hier hangt wel een prijskaartje aan. Het uitgebreide HTML-rapport van Playwright is vergelijkbaar met Cypress test-replay, maar het mist statistieken over meerdere testruns.</p><h3>Netwerk en mocking</h3><p>Hedendaagse moderne websites zijn gebouwd bovenop API’s. Het manipuleren en mocken van REST en GraphQL endpoints is dan ook een belangrijke functionaliteit in Cypress en Playwright.</p><p>Cypress biedt vanaf de eerste versies deze functionaliteit aan en is de afgelopen versies verder geëvolueerd. Het biedt nog meer handvatten in de aansturing van API ‘s zodat workarounds overbodig zijn voor het manipuleren en mocken van API’s.</p><p>In Cypress is deze functionaliteit erg goed uitgewerkt. Zo is in de testrunner zichtbaar wanneer welke API-calls plaatsvinden gedurende de testuitvoer. Tevens is het zichtbaar welke API-routes gemockt zijn en hoe vaak deze zijn aangeroepen. De ‘waits’ op API-calls worden gearceerd in de tesrunner weergegeven. Dit alles maakt het werken en debuggen van API-calls easy peasy in Cypress.</p><p>Uiteraard biedt ook Playwright deze functionaliteit van manipuleren en mocken van API ‘s aan. Hoewel de functionaliteit op hetzelfde niveau is als Cypress, is het debuggen een stuk lastiger. In de testrunner worden de API-calls op een aparte tab weergegeven. De samenhang tussen testacties als een button klik en API-call is hierdoor soms zoek. Zeker als een ‘wait’ op een API-call samen met een button klik in een ‘promise.all’ wordt gewrapt is het lastig om te achterhalen welke van de twee acties niet goed is uitgevoerd.</p><h3>Documentatie en community</h3><p>Niks is zo vervelend om urenlang het Internet af te struinen om een passend voorbeeld of een oplossing te vinden voor hetgeen wat geautomatiseerd dient te worden. Goede documentatie en een grote community helpen enorm om snel testen te schrijven.</p><p>Als Cypress een nieuwe feature toevoegt dan wordt er eerst de documentatie geschreven alvorens deze feature wordt ontwikkeld. Dit is terug te zien in de uitgebreide documentatie op de Cypress website. Hier vind je voorbeelden, uitleg over Cypress commando’s en best practices.</p><p>Playwright is de afgelopen jaren enorm ingelopen op het aanbieden van goede documentatie en voorbeelden. In het begin van Playwright was het erg summier en maakte het starten van een project met Playwright uitdagend.</p><p>Als de documentatie geen antwoord kan geven op de vraag dan is het fijn dat er terug kan worden gevallen op andere gebruikers met een hetzelfde soort probleem. In onderstaande afbeelding is te zien dat Cypress met 5,2 miljoen downloads populairder is dan Playwright met 3,5 miljoen downloads. Erg benieuwd naar komend jaar, ik verwacht dat Playwright verder inloopt op Cypress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iPJrpSjuzOw5VLeDCDCFyg.jpeg" /><figcaption>Bron: <a href="https://npmtrends.com/cypress-vs-playwright">https://npmtrends.com/cypress-vs-playwright</a></figcaption></figure><p>Deze cijfers tonen aan dat in de afgelopen jaren meer gebruikers Cypress gebruiken dan Playwright. Hoe meer gebruikers, hoe groter de kans om binnen de community een snel en passend antwoord op jouw vraag of probleem te vinden.</p><p>Af en toe is het nog wel zoeken naar een goed voorbeeld of een best-practise oplossing in Playwright, maar dit wordt met de dag beter al dan niet met de hulp van chatGPT.</p><h3>De succesfactoren van Cypress</h3><h4>Component testing</h4><p>Middels component testing worden individuele eenheden of componenten geïsoleerd van de rest van de applicatie getest om er zeker van te zijn dat ze functioneren zoals bedoeld. Bij het testen van componenten met Cypress kunnen testscripts worden geschreven waarmee het component onder verschillende ‘states’ en interacties kan worden getest. Zo kan bijvoorbeeld gebruikersinvoer worden gesimuleerd, de weergave en het gedrag van de component onder verschillende voorwaarden worden getest en kan worden geverifieerd dat het component de verwachte uitvoer produceert.</p><p>Playwright biedt ook component testing aan. Echter dit bevindt zich nog in een experimentele fase en is lang nog niet op het niveau van Cypress qua geboden functionaliteit en documentatie.</p><h4>Plug-ins</h4><p>Naast het testen op functionaliteit en betrouwbaarheid moeten applicaties ook worden getest op performance of toegankelijkheid (accessibility). Bestaande testing frameworks voor API- en UI-testen bieden vaak geen functionaliteit voor dit soort tests. Met als gevolg dat de testscripts van het begin af aan opnieuw moeten worden ontwikkeld. Cypress biedt hier een oplossing voor, namelijk plugins. Daarmee wordt de functionaliteit van Cypress uitgebreid. Nieuwe commando’s worden beschikbaar gesteld, ontbrekende functies worden toegevoegd en nog veel meer.</p><h4>Cypress Dashboard</h4><p>Zoals eerder genoemd in de toelichting op rapportage, is Cypress Dashboard een optionele betaalde cloudservice die verschillende functies biedt met betrekking tot projecten en test runs in Cypress. Het is een zeer nuttige en handige tool, vooral wanneer tests worden uitgevoerd in een CI/CD-pijplijn. Op het dashboard kun je aflezen hoelang tests duren, waarom ze al dan niet falen, en krijg je inzicht in trends over verschillende test runs.</p><h3>De succesfactoren van Playwright</h3><h4>Browsercontext</h4><p>Een browsercontext in een testframework verwijst naar de instantie waarin een browser opereert. Het bevat cache, cookies, local- en sessionstorage van de pagina welke wordt getest. Playwright creëert voor elke test een nieuwe browsercontext. Dit zorgt voor volledige testisolatie, omdat de testen niet beïnvloed kunnen worden door de browsercontext van een voorgaande test. Cypress reset alleen de browsercontext voor iedere test. Dit houdt in het herstellen van de browser ‘state‘ door het openen van een ‘about:blank’ pagina. Daarnaast worden cookies, local- en sessionstorage verwijderd. Hierdoor draaien de tests weliswaar in isolatie, maar niet zo volledig als in Playwright.</p><p>Het hebben van een ‘eigen’ browsercontext per test levert nog een groot voordeel op. Cypress kan de tests alleen uitvoeren in een enkel tabblad. Vaak is dit afdoende, maar niet als er interacties tussen verschillende rollen of delen van websites getest moeten worden, zoals bijvoorbeeld een chat-bot. Dit kan eenvoudig in Playwright door op het eerste tabblad de gebruiker vragen te laten stellen, terwijl op het tweede tabblad de virtuele klantenservice de antwoorden geeft. De interacties kunnen zelfs tussen verschillende websites plaatsvinden.</p><h4>Cross browser</h4><p>Het uitvoeren van testen in verschillende browsers is een cruciale stap om een consistente gebruikerservaring in verschillende browsers te garanderen. Conversie op een website daalt significant als de gebruikerservaring in een browser slecht is. Daarom is het van cruciaal belang om ervoor te zorgen dat applicaties soepel werken in verschillende browsers. Playwright kan testen uitvoeren op Chromium-, Webkit- en Firefox-browsers, maar ook op Google Chrome en Microsoft Edge. Daarnaast kan het ook worden uitgevoerd op geëmuleerde tablets en mobiele apparaten.</p><p>Cypress biedt ook cross-browser testing aan echter dit bevindt zich in een experimentele fase. Dit is bij lange na nog niet het niveau van Playwright, Cypress maakt zelfs gebruik van de Playwright Webkit browser.</p><h4>Parallel en sharding</h4><p>Standaard voert Playwright de testen parallel uit. Om dat te bereiken worden meerdere processen tegelijkertijd uitgevoerd. Deze processen zijn OS-processen die onafhankelijk worden uitgevoerd en worden aangestuurd door de testrunner. Playwright biedt verschillende configuratie opties waarmee delen van de testsuite parallel of serieel kunnen worden uitgevoerd.</p><p>Een andere functie, als je je testsuite wilt versnellen, is de shard-configuratie. Deze configuratie maakt het mogelijk de uitvoering van een spec file met daarin meerdere testen over verschillende machines te verdelen. Deze configuratie helpt bij de uitvoering van testen in de CI/CD pipeline. De doorlooptijd is nog korter, waardoor feedback sneller beschikbaar is.</p><p>Cypress kan de testen alleen parallel draaien als de betaalde cloudservice wordt afgenomen. Dit heeft als voordeel dat Cypress een algoritme toepast, zodat de testen op de snelst mogelijk manier worden uitgevoerd. Dit algortime stelt Cypress in staat om testen te prioriteren die foutgevoelig of flaky zijn gebleken in eerdere testruns.</p><h3>De winnaar is….</h3><p>De keuze voor een test framework moet niet worden genomen op basis van een inspirerende presentatie op een conferentie, een blog, eerdere ervaringen of ChatGPT.</p><p>Cypress en Playwright bieden verschillende benaderingen om geautomatiseerde testen op te zetten, elk met hun eigen voor- en nadelen. Elk voor- en nadeel van het test framework moet zorgvuldig worden afgewogen. Het is essentieel om voortdurend af te vragen welke (unieke) eigenschappen van het test framework het beste aansluiten bij de behoeften van het team en hetgeen wat geautomatiseerd moet worden.</p><p>Door elke optie zorgvuldig af te wegen, kan er een weloverwogen beslissing worden genomen over welke tool het beste aansluit bij de behoeften van het team. Dat test framework wordt dan de winnaar voor jullie team!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9e2a080de42" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/cypress-of-toch-playwright-f9e2a080de42">Cypress of toch Playwright?</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlock the power of AI directly in your terminal!]]></title>
            <link>https://medium.com/detesters/unlock-the-power-of-ai-directly-in-your-terminal-5378c553c235?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/5378c553c235</guid>
            <category><![CDATA[terminal]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[zsh]]></category>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Tim Lolkema]]></dc:creator>
            <pubDate>Mon, 27 Feb 2023 17:34:32 GMT</pubDate>
            <atom:updated>2023-03-14T14:01:26.463Z</atom:updated>
            <content:encoded><![CDATA[<p>With the rise of ChatGPT everyone is finding benefits in the power of AI.<br>But the web interface isn’t ideal for everyone, and doesn’t give you the “street cred” of doing everything in the terminal like a true gangster.</p><p>What if you could take it one step further and use AI directly in your terminal via a simple shell script?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3rrJPxwUnLq-Fupp2_ErpA.gif" /><figcaption>It’s that easy!</figcaption></figure><h3>Prerequisites</h3><p>I attempted to make this script as straightforward as possible without the need for NodeJS or Python. However, there is one small shell dependency needed to parse the JSON response.</p><blockquote>You can get jq from <a href="https://github.com/stedolan/jq">https://github.com/stedolan/jq</a></blockquote><p>Or install it with homebrew:</p><pre>brew install jq</pre><h3>Creating a OpenAI account and API-key</h3><p>To interact with OpenAI we need an OpenAI account and API key.</p><blockquote><a href="https://beta.openai.com/signup">https://beta.openai.com/signup</a></blockquote><blockquote><a href="https://platform.openai.com/account/api-keys">https://platform.openai.com/account/api-keys</a></blockquote><h3>The script</h3><pre>#!/bin/bash<br><br>if [  ! -n &quot;$OPENAI_API_KEY&quot; ]<br>  then<br>    echo &quot;ERROR: Please set an environment variable for OPENAI_API_KEY&quot;<br>    exit 1<br>fi<br><br>clean_input=$(printf &#39;%s&#39; &quot;$1&quot; | tr -d &#39;&quot;&#39; | tr -d &#39;\n&#39;)<br><br>response=$(curl -s https://api.openai.com/v1/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Authorization: Bearer $OPENAI_API_KEY&quot; \<br>  -d &#39;{<br>  &quot;model&quot;: &quot;text-davinci-003&quot;,<br>  &quot;prompt&quot;: &quot;&#39;&quot;$clean_input&quot;&#39;&quot;,<br>  &quot;temperature&quot;: 0.1,<br>  &quot;max_tokens&quot;: 600<br>}&#39;)<br><br>generated_text=$(printf &#39;%s&#39; &quot;$response&quot; | jq &#39;.choices[].text&#39; | tr -d &#39;&quot;&#39; | cut -c 3-)<br><br>printf &quot;$generated_text\n&quot;<br>printf &quot;$generated_text\n&quot; | pbcopy</pre><ul><li>The script is pretty simple: remove any newlines and double quotes from the input, make a cURL request to OpenAI, and then print the parsed and cleaned response.</li><li>This version will auto-copy the response to your clipboard (Mac only)</li><li>First set the environment variable OPENAI_API_KEY on your system.</li><li>The script uses the text-davinci-003 model, which is currently the most advanced OpenAI model.</li><li>You can play around with the temperature setting, a higher value will give more randomness / creativity.</li><li>With the max_tokens setting you can specify the maximum amount of tokens spend on one request.</li></ul><p>In order to easily call this script from every directory i’ve added a simple function which you can put in your .zshrc or .bashrc</p><pre>ai() {<br>    if [ -n &quot;$1&quot; ]<br>    then<br>        sh &quot;~/scripts/openai.sh&quot; &quot;$1&quot;<br>    fi<br>}</pre><h3>So how to use this script?</h3><p>Simply call the ai function, and give within quotes your prompt to OpenAI.</p><p>e.g.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UyYi-BdGN1l933i2kUBcWQ.gif" /></figure><p>Wait a few seconds….</p><p>SHEBANG! 🍺 There it is!</p><pre>&lt; ~ &gt; ai &quot;how to perform a cURL with JWT token?&quot;<br><br>1. Generate a JWT token.<br><br>2. Create a cURL request with the JWT token in the Authorization header.<br><br>curl -H \Authorization: Bearer &lt;JWT_TOKEN&gt;\ &lt;URL&gt;<br><br>3. Execute the cURL request.</pre><blockquote>The title of this blog was generated by AI 😎</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5378c553c235" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/unlock-the-power-of-ai-directly-in-your-terminal-5378c553c235">Unlock the power of AI directly in your terminal!</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create a business driven dashboard with Grafana]]></title>
            <link>https://medium.com/detesters/create-a-business-driven-dashboard-with-grafana-3e8edcb76266?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/3e8edcb76266</guid>
            <category><![CDATA[testautomatisering]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[influxdb]]></category>
            <category><![CDATA[grafana]]></category>
            <dc:creator><![CDATA[Anaïs van Asselt]]></dc:creator>
            <pubDate>Thu, 17 Mar 2022 06:52:13 GMT</pubDate>
            <atom:updated>2022-03-17T06:52:13.246Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BTBG0YWjLMJi6UuftYKgSg.png" /></figure><h4>Greenhouse of test automation: Part V</h4><blockquote>🌱 <em>Welcome to the blog series about </em><a href="https://medium.com/detesters/the-benefits-of-visualizing-test-automation-5a0837414076"><em>visualizing test automation</em></a><em>. Dutch greenhouses excel in sustainability and serve as a main theme through this journey. Earlier I showed how results </em><a href="https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482"><em>from test automation gardens are extracted and stored in an InfluxDB data storage</em></a><em>. Let’s visualize the test results on a Grafana dashboard and fine tune them with some climate control!</em></blockquote><p>Marshall Eriksen would have much fun with a Grafana dashboard. I also had lots of fun with Grafana dashboards, but not just with trying to display my favorite pies or bars. My goal was to visualize existing automated tests for the business, so that they gain confidence in the coverage and encouraged to do only the necessary manual testing. This last part of my blog series describes how to visualize data from an InfluxDB in a Grafana dashboard, tips for test results, how to bring in a business perspective and lessons learned. First some requirements for the starting point of my story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g8KoLel1pHNNxZEibCvQWQ.png" /><figcaption>When Marshall Eriksen (HIMYM) shares his important data</figcaption></figure><p><strong>Requirements</strong></p><ul><li>An InfluxDB instance started containing data/test results</li><li>Grafana installed and started</li><li>The InfluxDB instance is configured as datasource in Grafana</li></ul><blockquote>🙏🏻 The blogs of <a href="https://dzone.com/users/3614615/gowthamraj198.html">Gowthamraj Palani</a> really helped me to <a href="https://dzone.com/articles/view-test-results-in-grafana-part-1">get started with InfluxDB and Grafana</a> on Mac. He also demonstrates <a href="https://dzone.com/articles/view-test-results-in-grafana-part-2">how to store TestNG test results in InfluxDB</a>. In previous blogs I shared :<br><a href="https://medium.com/detesters/how-to-store-testcaf%C3%A9-tests-in-an-influxdb-95f4e59abda8?p=95f4e59abda8">How to store TestCafé tests in InfluxDB</a><br><a href="https://medium.com/detesters/how-to-store-junit-tests-in-an-influxdb-7faf63dd562a">How to store JUnit tests in InfluxDB</a></blockquote><p>Let’s start with the creation of a Grafana dashboard. Multiple panels fit in a dashboard, and every panel can be connected to a different datasource. It’s also possible to repeat one panel for different variables (more about that later).</p><p><strong>Configure a panel</strong></p><p>That the possibilities with Grafana are endless, I experienced when creating my first panel. It felt like a large playground, and it was quite overwhelming! So many different visualizations and options to choose from, and almost everything can be customized. Compose the data to visualize in the ‘Query’ tab. Once the connected InfluxDB datasource is chosen, data can be selected and filtered via Influx Query Language, or <a href="https://docs.influxdata.com/influxdb/v1.8/query_language/">InfluxQL</a> . In a nutshell, an Influx query is structured like this:</p><ul><li><strong>FROM</strong>: select the right measurement/table</li><li><strong>WHERE</strong>: data to filter on, only tags* can be selected</li><li><strong>SELECT</strong>: data to show, only fields* can be selected</li><li><strong>GROUP BY</strong>: data to group on, only tags* can be selected</li></ul><p>*<em>NOTE</em>: <em>This limitation of use is based on the attributes of fields/tags. Check </em><a href="https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482"><em>this</em></a><em> out for more information and tips for the application regarding test data.</em></p><p>After some time playing, I managed to realize my first dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*XS4BsJd_UCUeQWYAmvfNSA.png" /><figcaption>My first Grafana dashboard; I was so proud!</figcaption></figure><p><strong>My First Dashboard</strong></p><p>Proudly I presented this dashboard to the business consultant of my team. I expected nothing but praise of course. However, his response was: ‘<em>What am I looking at? Services, applications? I don’t understand, I only know user flows and screens</em>’. You can imagine, this was quite a disappointment for me. Yet, a great learning point at the same time. Valuable input to fine tune the dashboard for the perspective of the business, so it can actually fulfill its goal. A form of climate control, just like in a greenhouse!</p><p><strong>Climate control</strong></p><p>The response of the business consultant reminded me that people in business and development come from different worlds. Developers talk about applications, services, pipelines and layers of test automation. All technical terms that don’t necessarily mean anything to the business. The business care about their users and business processes. Application transcending user flows with specific business goals. Consequently, displaying data from a business perspective is necessary to unburden them during acceptance testing.</p><p><strong>A business driven dashboard</strong></p><p>Instead of a dashboard displaying the tests of one application that covers multiple features, let’s visualize the feature itself! This change of focus can be accomplished with test metadata. For TestCafé, JUnit, and many other test automation tools, it’s possible to add metadata or tags to tests. That can be a value stream, feature or any desired category to help achieving the right business perspective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBL46f3VS67piwl7AkC3SQ.png" /><figcaption>Test metadata ‘Process’ and ‘Section’ are saved as tags to filter on in the Grafana dashboard</figcaption></figure><p>In both the TestCafé reporter and JUnit listener I created enums with categories ‘<em>Process</em>’ and ‘<em>Section</em>’, and made them available for tests. Those categories are stored as tags with the tests in the InfluxDB. This enables the possibility to filter on data from those categories, and subsequently display them in a dashboard. Several feedback rounds resulted in the Grafana dashboard below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*YDW976x9s6HbtmgeXvicoA.png" /><figcaption>An overview in Grafana of the process ‘Planning &amp; Verkeer’ with a panel per section like ‘PA_VERKEER’</figcaption></figure><p>It visualizes all tests of one ‘<em>Process</em>’ tag. Every panel in this dashboard represents a ‘<em>Section</em>’ (part of a UI) recognizable for the users. The overview only displays the test runs, number of tests and the results* within the selected range of time. Lucky for me, these are not 9 different panels I had to configure, but just 1 panel in repeat.</p><blockquote>*LEGEND of the Grafana dashboard:<br>- FE = the frontend TestCafe tests<br>- BE = the backend JUnit tests <br>- ‘Geslaagde tests’ = successful tests<br>- ‘Gefaalde tests’ = failed tests (&gt;1, panel is red)<br>- ‘Overgeslagen tests’ = skipped tests (&gt;1, panel is orange)</blockquote><p><strong>The Sweet Repeat</strong></p><p>Variables enable the possibility to repeat panels. In Grafana a new variable can be created via ‘Dashboard settings’ &gt; ‘Variables’. I wanted a panel for every Section. So I created the variable ‘section’, and retrieved the values of the tags with the following query:</p><pre>SHOW TAG VALUES  WITH KEY = &quot;section&quot;</pre><p>The value of the variable is unlocked by mentioning the key prefixed with a $ sign, in my case $section. In the example below I use the variable value in different ways:</p><ol><li>As title of the panel</li><li>To query on</li><li>As part of a link for a detailed dashboard</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vpYLF0m_KIwwL_1LofCXAw.png" /><figcaption>Different ways to create a variable panel</figcaption></figure><p>Clicking on ‘<em>Geslaagde FE tests</em>’ results in a redirect to the following dashboard, with detailed information about the tests of the specific ‘Section’. Lots of these details are saved as fields in the InfluxDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wo0TjmlNT3244GZURrDjUw.png" /><figcaption>Detailed information about the test in business language builds trust in QA</figcaption></figure><p>This detailed dashboard led to another improvement point. The test titles were expressed in English with technical terms, and it didn’t mean anything to the business. Formulating phrases in business language focused on the goal, helps the business to grasp what exactly is covered by automated tests. Another step towards the goal to build trust in quality assurance through test automation.</p><p><strong>Lessons learned</strong></p><p>Unfortunately I can’t conclude this blog series with the goal achieved and a happily ever after. However, it started a continuous learning process. The dashboard ironically showed how big the distance actually was between business and development, and brought the worlds together. Step by step we are discovering doubles &amp; gaps in test coverage and ways to align our work in the development process. This Greenhouse of Test Automation was also a great personal learning experience<strong>:</strong></p><ol><li><strong>Thrifty With Tooling<br></strong>I’m glad I started my search for visualization tooling in my own context. InfluxDB and Grafana was already used for production monitoring. Reusing existing tooling has a lot of benefits; it’s cheap, it’s integrated in the environment, and people already have knowledge of it. Looking for visualization tooling? Go shop at your operations or platform team, you might just find a vintage treasure!</li><li><strong>Why &amp; Who</strong><br>When I started my quest to visualize automated tests on a Grafana dashboard, it was easy to get lost in a technical bubble. Especially when having fun with the graphs and charts, like Marshall. <em>What do I want to achieve with a dashboard, and for which target group?</em> I would definitely answer these questions first next time, before creating a dashboard. It would help to focus on the right perspective and visualization.</li><li><strong>Fast Feedback</strong><br>A technical bubble feels very safe; just doing your own thing and no one criticizing. The common pitfall is to stay too long in the bubble, which I did. Asking for feedback can be confrontational, but it often yields the best points for improvement, especially if it is someone from the target audience who should start using the dashboard.</li></ol><p>Automated tests represent the crops in the Greenhouse of Test Automation. It is good to make the crops visible to the stakeholders. To determine what the crops need and don’t need. And maybe more importantly how they add value for the stakeholders. Let’s continue with climate control to optimize the added value and health of automated tests!</p><p>🖇️ You are welcome to connect with me on LinkedIn, or follow me on Medium, to keep updated with my latest blogs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e8edcb76266" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/create-a-business-driven-dashboard-with-grafana-3e8edcb76266">Create a business driven dashboard with Grafana</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to store JUnit tests in an InfluxDB]]></title>
            <link>https://medium.com/detesters/how-to-store-junit-tests-in-an-influxdb-7faf63dd562a?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/7faf63dd562a</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[influxdb]]></category>
            <category><![CDATA[junit]]></category>
            <category><![CDATA[test-automation]]></category>
            <dc:creator><![CDATA[Anaïs van Asselt]]></dc:creator>
            <pubDate>Tue, 18 Jan 2022 07:33:04 GMT</pubDate>
            <atom:updated>2022-01-18T07:37:05.730Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DDA_iluxzpdHdivVw3g7bA.png" /><figcaption>Greenhouse of JUnit automated tests</figcaption></figure><h4>Greenhouse of Test Automation: Part IV</h4><p>Instead of looking at bad memes, wouldn’t you rather extract data of JUnit tests and store them in an InfluxDB? Visualizing those tests on a Grafana dashboard to build trust in test automation and analyze the results ? Or are you looking for inspiration to create your own solution? In all cases you are in the right place!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*eCbiWuN5jnVu4jDxg0R1OA.jpeg" /><figcaption>Created my own bad meme at img-flip.com</figcaption></figure><blockquote><em>🌱 </em>Welcome to the blog series about <a href="https://medium.com/detesters/the-benefits-of-visualizing-test-automation-5a0837414076">visualizing test automation</a>. Dutch greenhouses excel in sustainability and serve as a main thread running through this journey. I wrote about the concept of <a href="https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482">extracting test results, storing them in an </a><a href="https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482">InfluxDB</a>, and <a href="https://medium.com/detesters/how-to-store-testcaf%C3%A9-tests-in-an-influxdb-95f4e59abda8?p=95f4e59abda8">how this works for </a><a href="https://medium.com/detesters/how-to-store-testcaf%C3%A9-tests-in-an-influxdb-95f4e59abda8?p=95f4e59abda8">TestCafé tests</a>. Now it’s time to deep dive in the implementation for JUnit tests, to enable visualization on a <em>Grafana</em> dashboard!</blockquote><p><strong>How to extract and store JUnit tests in an InfluxDB?</strong></p><p>I didn’t find an out the box solution to help me answer this question, so I made my own; <a href="https://search.maven.org/artifact/io.github.vanbadasselt/junit-listener-influxdb">junit-listener-influxdb</a>. Use it in a project with JUnit tests to extract and store them in an InfluxDB. It’s ALMOST plug &amp; play:</p><ul><li>🔌 <strong>Plug<br>- </strong>Import the junit-reporter-influxdb as a dependency</li><li>🔧<strong> Configure<br></strong>- Set the right environment variables</li><li>🎮 <strong>Play<br></strong>- Run your Influx database on the specified host and port<br>- Run your JUnit tests</li></ul><p>That’s it! Note that the junit-listener-influxdb library is generically set up where the tests run remotely via Gitlab pipelines. You might want to save data of your tests specific to your context. My advice? Check out the details and possibilities in this <a href="https://github.com/vanBadAsselt/testcafe-reporter-influxdb#readme">README</a>. Then fork the library and specify it to save the most relevant data for your project, like test types, categories, or application names. I will give an example why and how to customize test categories.</p><p><strong>Customize JUnit test extraction</strong></p><p>When visualizing tests on a dashboard, it is very useful to group them based on categories, risk or feature for example. This is enabled by storing tags with the tests. Create tags with fixed values in the junit-listener-influxdb library like the example below shows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ouwwSX7pGkFZlBam6zgR7Q.png" /><figcaption>Specify the tags you want to filter your JUnit tests on</figcaption></figure><p>Add the relevant categories to yourJUnit test (note the difference between Java &amp; Kotlin). These tags will be saved along the tests, to enable filtering during the setup of a Grafana dashboard.</p><p>Java:</p><pre>@Tag(TestMetadata.Risk.HIGH)<br>@Tag(TestMetadata.Feature.LOGIN)</pre><p>Kotlin:</p><pre>@Tags(<br>        Tag(TestMetadata.Risk.HIGH),<br>        Tag(TestMetadata.Feature.LOGIN)<br>)</pre><p><strong>How does the </strong><strong>junit-reporter-influxdb library work?</strong></p><p>The picture below shows the <a href="https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482">concept of extracting and storing test results in an InfluxDB</a>. With every step I will explain the implementation for the junit-reporter-influxdb library below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-KbHrzeud1acvKZh2CUncw.png" /><figcaption>Visualization of the JUnit test automation sensor: junit-reporter-influxdb</figcaption></figure><p><strong>Extract JUnit test data</strong></p><p>My environment is enriched with different flavors of backend tests. The tests are written both inJava and Kotlin, and there is a distinction of three different test layers:</p><ol><li><strong>Unit</strong> : test in isolation using the Mockito, JPA or WebMVC extension.</li><li><strong>Component</strong> : test within Spring application context using the SpringBootTest extension, external services are mocked using WireMock.</li><li><strong>Integration</strong> : in theSpring application context the integration between the system under test and external services is tested.</li></ol><p>Luckily they have one thing in common; the tests run using the JUnit testrunner. The JUnit TestExecutionListener provides an interface to tap into the tests during a test run, and supports all above mentioned extension flavors. Only condition is that the tests must run using JUnit5. Upgrading from JUnit4 to JUnit5 can be a challenge for old services running with ancient Spring versions, I can tell 😵.</p><p>The methods of <a href="https://junit.org/junit5/docs/5.0.3/api/org/junit/platform/launcher/TestExecutionListener.html">TestExecutionListener</a> are called during the test run; at the beginning and end of a test, class or complete test run. junit-listener-influxdb contains the implementation of those methods, and a service loader to enable the use of the library by other services.</p><p><strong>Store JUnit test data in an InfluxDB</strong></p><p>Let’s zoom in the executionFinished method of the TestExecutionListener, which is called when a JUnit test is completed. I use the influxdb-java client to interact with InfluxDB in a Java environment. This client helps to formulate, address and send the data to an InfluxDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*maxDw9_N4XN31CuA6ltGKQ.png" /><figcaption>Structure JUnit test data in an envelope which fits perfectly in an InfluxDB</figcaption></figure><ul><li>✉️ <strong>Formulate; set the data in the correct structure<br></strong>The completed test is accessible via the input parameters of executionFinished. Relevant data like duration of the test are processed and structured in the plain old Java object (POJO) TestResult . Annotations provided by influxdb-java ensure that the test result fits perfectly in the Influx table testResults (@Measurement). The POJO is saved as a Point, which is a single data record in Influx terminology. <br>All test results are collected in a batch of Points; we only want to connect with the database and send all results at once, when the test run is done. Finally, I reset the test result to extract the next one with a clean slate.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-TBvfOcliX1I7_FZdp2Mw.png" /><figcaption>Put the right address of the Influx database on the envelope</figcaption></figure><ul><li>📧 <strong>Address; set the properties for the correct database <br></strong>When the junit-listener-influxdb starts, an InfluxDB object is created with the correct address of the Influx database we want to send the data to. These are originated from the environment variables. You can set them locally when running the application, or as remote variables in your pipeline.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r0-7OODRNZMtfpon8QPNVw.png" /><figcaption>Send the envelope with JUnit test data to the Influx database!</figcaption></figure><ul><li>📩 <strong>Send; post the data to the database<br></strong>When the test run is completed, a connection with the addressed Influx database is setup and checked. All test results collected in the batch of Points are validated and send with a REST POST call, provided by the InfluxDB client. Afterwards, the connection with the database is closed.</li></ul><p>That’s it! To me it was quite a puzzle to find out how the data of JUnit tests can be extracted and processed. Several moments I wanted to throw my laptop through the window, but with help from my colleagues of deTesters and some perseverance I struggled through. I learned a lot and can monitor JUnit tests on a Grafana dashboard now!</p><p>What’s next? How to display this testdata in an Grafana dashboard, and the key question of this project; how is it used in practice and what is the result? Stay tuned!</p><p>🖇️ You are welcome to connect with me on LinkedIn, or follow me on Medium, to keep updated with my latest blogs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7faf63dd562a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/how-to-store-junit-tests-in-an-influxdb-7faf63dd562a">How to store JUnit tests in an InfluxDB</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to store TestCafé tests in an InfluxDB]]></title>
            <link>https://medium.com/detesters/how-to-store-testcaf%C3%A9-tests-in-an-influxdb-95f4e59abda8?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/95f4e59abda8</guid>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[testcafe]]></category>
            <category><![CDATA[influxdb]]></category>
            <category><![CDATA[software-test-automation]]></category>
            <dc:creator><![CDATA[Anaïs van Asselt]]></dc:creator>
            <pubDate>Wed, 15 Sep 2021 12:21:27 GMT</pubDate>
            <atom:updated>2021-09-15T12:52:03.031Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PieW1wze5PQ3diSP_9xg8w.png" /><figcaption>Greenhouse of TestCafe automated tests</figcaption></figure><h4>Greenhouse of test automation: part III</h4><blockquote>🍹 A tester walks into a TestCafé and orders a Mockito.</blockquote><p>Instead of reading bad jokes, wouldn’t you rather extract the data of TestCafé tests and store them in an InfluxDB? Do you want to visualize those tests on a Grafana dashboard? Or are you looking for inspiration to create your own solution? Then you’ve come to the right place!</p><blockquote><em>🌱 </em>Welcome to the blog series <em>Greenhouse of test automation</em>; a journey of visualizing test automation. To build confidence in quality assurance through test automation, for the benefit of sustainable development. Dutch greenhouses excel in sustainability and serve as a main thread running through this journey.</blockquote><p>This journey starts with the <a href="https://medium.com/detesters/the-benefits-of-visualizing-test-automation-5a0837414076">benefits of visualizing test automation</a>. Followed by <a href="https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482">the concept of extracting test results, and storing them in an </a><a href="https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482">InfluxDB</a>, in order to make them visible on a Grafana dashboard. This third article covers the implementation of this concept for TestCafé tests.</p><p><strong>How to extract and store TestCafé tests in an InfluxDB?</strong></p><p>This was the big question of my quest to visualize TestCafé tests on a Grafana dashboard. It led to a lot of obstacles, learnings and ultimately, the <a href="https://www.npmjs.com/package/testcafe-reporter-influxdb">testcafe-reporter-influxdb</a> library. You can use it in the project where your TestCafé tests are located, and it’s ALMOST plug &amp; play:</p><ol><li>🔌 <strong>Plug</strong> <br>- Fork the testcafe-reporter-influxdb library* <br>- You also need the dotenv library to set environment variables</li><li>🔧<strong> Configure</strong> <br>- Specify the TestCafé reporter <br>- Set the address of your Influx database (host, port etc) via environment variables</li><li>🎮 <strong>Play</strong><br>- Run your Influx database on the specified host and port<br>- Run your TestCafé tests</li></ol><p>ALMOST…</p><p>Note that the testcafe-reporter-influxdb library is generically set up for a project with TestCafé tests run remotely via Gitlab pipelines. There will be data specific to your context, so my advice:</p><p><em>*Fork the library and specify it to save the most relevant data for your project, like test types, categories, or application names.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*W0kLnLXXW9PjcrS06UBNLw.png" /><figcaption>Specify the TestMetadata you want to use in your TestCafe test</figcaption></figure><p>Standardize the test metadata in the testcafe-reporter-influxdb library like the example above shows. Then add it to your TestCafé tests, like below. These tags will be saved, so you can filter on this data and setup Grafana dashboards based on risk categories for example.</p><pre>fixture`The user sees the home screen after a successful login`<br>  .meta({ feature: Feature.LOGIN, risk: Risk.SMOKE })</pre><p>Hmm.. on second thought not just plug &amp; play🤔 No worries! I explain all usage details and possibilities in this <a href="https://github.com/vanBadAsselt/testcafe-reporter-influxdb#readme">README</a>.</p><p><strong>How does the </strong><strong>testcafe-reporter-influxdb library work?</strong></p><p>This two-step <a href="https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482">concept of extracting test results and storing test results in an InfluxDB</a> acts as a tool to explain the implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s3xqFpbmxk7iImAi37tCqw.png" /><figcaption>Visualization of the TestCafé test automation sensor: testcafe-reporter-influxdb</figcaption></figure><p><strong>Extract TestCafé test data</strong></p><p>With the <a href="https://testcafe.io/documentation/402810/guides/extend-testcafe/reporter-plugin">TestCafé Reporter Plugin</a>, it’s possible to tap into TestCafé tests at different times during a test run. One of the built-in reporters collects data from these tests to generate a test report in XML format. Unlucky for me, there is no reporter that out of the box stores test results in an InfluxDB. However, the generator-testcafe-reporter provides a template for a custom reporter.</p><p>In just a few steps the project is generated, and ready to implement the reporter methods. Reporter methods are called byTestCafé during the test run; at the beginning and end of a test, fixture or complete test run. The available data can be extracted via the arguments. Within the testcafe-reporter-influxdb the reporter methods are implemented to process and store this test data.</p><p><strong>Store TestCafé test data in an InfluxDB</strong></p><p>The example below shows a fragment of the implementation for reportTestDone; the reporter method called when a TestCafé test is done. I use the node-influx client to interact with InfluxDB in a Node.js environment. This client helps to formulate, address and send the data to an InfluxDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3DhWNL8UHgWz_p5fVPUw_A.png" /><figcaption>Formulate data from TestCafé tests in a point to fit in an InfluxDB</figcaption></figure><ul><li>✉️ <strong>Formulate; set the data in the correct structure </strong><br>reportTestDone provides access to the data of one test; name, testRunInfo and testMeta. After extraction, the test data is saved in an IPoint object, made available by node-influx. A point stands for a single data record. It is linked to a table - or so-called measurement - and contains a timestamp, fields and tags. Imagine sending an IPoint object after every test to the database… that wouldn’t be efficient. Therefore the point is added to a collection of points, then reset for the next test.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cu7b7cNZksScjTtpfmK1sg.png" /><figcaption>Address the Influx database so that the TestCafé test data arrive at the right location</figcaption></figure><ul><li>📧 <strong>Address; set the properties for the correct database<br></strong>node-influx provides an envelope in the form of the InfluxDB object to address the right database instance. Using schema, a validation is performed on the data that you send in that envelope. Are the fields filled with the correct field types? Like a lot of code, the schema is not forgiving. If one field is missing in the data, or does not contain the correct data type, this object will immediately give an error and the process will be aborted. I learned this the hard way…</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*go_ERprxcK5mejiwxWlkkQ.png" /><figcaption>At last! Time to send the collected TestCafé test data to the Influx database</figcaption></figure><ul><li>📩 <strong>Send<br></strong>At the end of the test run the method reportTaskDone is called. The moment where allpoints are saved, collected and sent. Using the InfluxDB object a connection is made to the correct database. Then all TestCafé test data is sent via a REST POST call.</li></ul><p>To be honest I wasn’t quite sure to start this adventure implementing the testcafe-reporter-influxdb, but I would definitely recommend it. Especially if you aspire to learn more about automation. If you have a similar challenge to tackle, don’t be afraid, just go for it! The journey is as rewarding as the result!</p><p>What’s next? The implementation of extraction and storage of JUnit tests in an InfluxDB, and how to display this testdata in an Grafana dashboard. Stay tuned!</p><p>🖇️ You are welcome to connect with me on LinkedIn, or follow me on Medium, to keep updated with my latest blogs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95f4e59abda8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/how-to-store-testcaf%C3%A9-tests-in-an-influxdb-95f4e59abda8">How to store TestCafé tests in an InfluxDB</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extract and store automated test results in an InfluxDB]]></title>
            <link>https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482?source=rss----57c73ce6102---4</link>
            <guid isPermaLink="false">https://medium.com/p/63053abb6482</guid>
            <category><![CDATA[sustainable-development]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[influxdb]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[test-automation-framework]]></category>
            <dc:creator><![CDATA[Anaïs van Asselt]]></dc:creator>
            <pubDate>Fri, 28 May 2021 06:24:21 GMT</pubDate>
            <atom:updated>2021-05-28T06:36:52.082Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrCGt0FAzEUoAKYbJ4LkMg.png" /></figure><h4>Greenhouse of test automation: part II</h4><h3>Extract and store automated tests in an InfluxDB</h3><blockquote><em>🌱 </em>Welcome to the blog series Greenhouse of test automation! A context-driven journey of visualizing test automation using InfluxDB and Grafana. Dutch greenhouses excel in sustainability and serve as a main thread running through this journey. Abstract concepts, implementation examples and learning points are provided to start your own journey.</blockquote><p><a href="https://medium.com/detesters/the-benefits-of-visualizing-test-automation-5a0837414076">Part I</a> explains the concept greenhouse of test automation and two customer cases where this is applicable. The decision of visualizing test automation to build more confidence in quality assurance and to identify possible doubles/gaps in the test coverage. And the available tools in that context; InfluxDB and Grafana.</p><p>What do you need to extract results from automated tests? How can you store them in an Influx database ? Especially from different test automation tools? <strong>This second part </strong>answers all these questions and more. You will learn more about InfluxDB to store and visualize test results on a Grafana dashboard as efficiently as possible.</p><p><strong>Extract test results</strong></p><p>Extracting results from automated tests depends on the tooling you are using. I researched the test automation tooling used in the context of my client: Jest &amp; TestCafé to test in the frontend, JUnit and Spring for the backend. For all this tooling it is possible to tap into the tests at different times during a test run. At the beginning and end of a test, test file (class/fixture) or complete test run. The available data depends on the tooling. Usually the obvious like the result of a test, its title, and any possible error messages are at your disposal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*biVVN2HrtBz9hPNiJ5TfMQ.png" /><figcaption>A ‘reporter’ or ‘listener’ can extract several data of an automated test</figcaption></figure><p>So the concept is the same, only the name might differ.Reporters is a popular term for frontend test automation tooling, JUnit and Spring like the sound of listeners. If you use another tool, just try a Google search with your ‘tool name’ + reporter, listener or just output, you probably find one that does the trick for you as well!</p><p>Often standard reporters/listeners exist that export results in several formats, like xml. Unfortunately, there are none that out of the box store results in an Influx database. This means I have to create a custom reporter in JavaScript and listener in Java, what a challenge!</p><p><strong>Store test results in an InfluxDB</strong></p><p>Another challenge is to store the test results in an Influx database. Fortunately, there are several clients that allow interaction with Influx in different contexts. Below you will find a table with the available clients for JavaScript and Java. With InfluxDB version 1.8 it is recommended to use the new libraries supported by InfluxDB. Unfortunately, 1.7.9 is still used in my context, so the implementation examples in my future blogs for TestCafé and JUnit will refer to the older libraries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2KT-KrdCMm4X-aPLO1FAdg.png" /><figcaption>check out <a href="https://docs.influxdata.com/influxdb/v2.0/tools/client-libraries/">https://docs.influxdata.com/influxdb/v2.0/tools/client-libraries/</a> for all supported contexts</figcaption></figure><p>Despite the different contexts and syntax, the concept of storing test results using the InfluxDB client boils down to the same three steps:</p><ul><li>✉️ <strong>Formulate</strong> <br>Set the data in the correct structure for InfluxDB</li><li>📧 <strong>Address</strong> <br>Set the properties for the correct Influx database</li><li>📩 <strong>Send</strong> <br>Send the data toInfluxDB via a REST POST call</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PVGMXqkDvIjnBCZ7OYwOXQ.png" /><figcaption>The concept of storing automated test results in an InfluxDB</figcaption></figure><p>In order to efficiently store your test results in an Influx database and to visualize them effectively, it is useful to know more about Influx.</p><p><strong>About InfluxDB</strong></p><p>InfluxDB is an open source database and I can tell they like cool new terms. A database can consist of one or more measurements; a container of values comparable to a table. All measurements contain a specific column for time, because InfluxDB is a <a href="https://www.influxdata.com/time-series-database/">time-series database</a>. This means that every database record is linked to a timestamp.</p><p>Other columns of a measurement can be filled with field or tag values. The main difference? Tags are indexed and fields are not, which has quite an impact on the query capabilities. Therefore it is important to consider how you want to visualize your test data.</p><p>If you want to display actual test data, like test titles, fields are great. However, if you want to filter or group on test results or other categories, tags are the best option for you. In the picture below you will find an overview of the differences between a field and tag, and some examples of appropriate test data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uccsa73_FYtm0hIi5RjOAg.png" /><figcaption>The difference between fields and tags, and for which kind of test data they are most suitable</figcaption></figure><p>One single data record, or row of a measurement is called a point. A point is unique by the combination of a measurement name, tag set and timestamp.</p><blockquote><em>🌱 NOTE</em>: if a point is not unique, it will be overwritten! Quite a pitfall I fell deeply into…</blockquote><p>Now you know more about the abstract concepts of extracting test results, and storing them in an InfluxDB. Also about how InfluxDB works, to retrieve and visualize results as effectively as possible. What’s next? A technical deep dive starting at the test automation tooling TestCafé and JUnit, to the Influx database, and to resurface again at the Grafana dashboard. Stay tuned for <strong>part three </strong>and find out!</p><p>🖇️ You are welcome to connect with me on LinkedIn, or follow me on Medium, to keep updated with my latest blogs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63053abb6482" width="1" height="1" alt=""><hr><p><a href="https://medium.com/detesters/extract-and-store-automated-test-results-in-an-influxdb-63053abb6482">Extract and store automated test results in an InfluxDB</a> was originally published in <a href="https://medium.com/detesters">deTesters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>