<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Oleksandr Podoliako on Medium]]></title>
        <description><![CDATA[Stories by Oleksandr Podoliako on Medium]]></description>
        <link>https://medium.com/@oleksandr-podoliako?source=rss-3df9d47575ee------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-JG8J3T2MwZPuXz2tvLDSQ.jpeg</url>
            <title>Stories by Oleksandr Podoliako on Medium</title>
            <link>https://medium.com/@oleksandr-podoliako?source=rss-3df9d47575ee------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:54:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@oleksandr-podoliako/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Do We Still Need to Write API Tests? An Experiment with LLM Agents Testing a REST API]]></title>
            <link>https://oleksandr-podoliako.medium.com/do-we-still-need-to-write-api-tests-an-experiment-with-llm-agents-testing-a-rest-api-67dfede65986?source=rss-3df9d47575ee------2</link>
            <guid isPermaLink="false">https://medium.com/p/67dfede65986</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Oleksandr Podoliako]]></dc:creator>
            <pubDate>Mon, 20 Jul 2026 17:44:40 GMT</pubDate>
            <atom:updated>2026-10-15T21:35:54.583Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u1csY6PFuhO_KmnUVCWCGg.jpeg" /></figure><p>There’s a version of this question that’s already been answered: yes, LLMs are useful for writing test code. Point one at an OpenAPI spec, ask for pytest cases, get a reasonable first draft. That’s codegen, and it’s been done to death.</p><p>The question I actually wanted an answer to was different: skip codegen entirely, give an LLM agent a single tool — make an HTTP request, see the response — and let it test a live API directly, the way a human tester would poke at a staging environment. And within that: is it better to let the agent explore freely, or hand it a fixed checklist and have it execute that? “Better” meaning what, exactly — cheaper, more bugs found, more <em>useful</em> bugs found? Those are different axes, and conflating them is where a lot of the AI-testing discourse goes wrong.</p><p>This is a report on 50 runs across three models and two agent modes (10 runs per mode for Sonnet and Haiku, 5 per mode for Opus), against a REST API with bugs I planted myself and know the ground truth for. Some of the results confirm what you’d guess. At least one doesn’t, and it’s arguably the most interesting finding in the whole dataset.</p><h3>Setup</h3><p>The target is a small Library Management API built in FastAPI, covering books, members, and loans, with 7 bugs deliberately planted into otherwise-working CRUD logic: an off-by-one in pagination, a DELETE endpoint that returns 200/empty-body for both hits and misses on a missing resource, a DELETE with no auth check at all, a loan-creation path that succeeds even when available_copies is zero, a date-range filter that excludes an exact boundary match, an invalid-email input that produces a 500 instead of a 400, and a race condition that lets concurrent loan requests push available_copies negative.</p><p>A custom API was necessary rather than a well-known public sandbox (something like restful-booker) because of contamination risk: if a model has seen a public API’s known bugs, quirks, or existing test suites during training, you’re no longer measuring exploration — you’re measuring recall. A from-scratch API with an undisclosed bug list is the only way to get an honest read.</p><p>Two agent modes were tested. In “explore,” the agent gets the API’s base URL and a general instruction to test it, with no predefined plan, capped at 45 turns. In “test-cases,” the agent is given a 22-item checklist of test case descriptions (written without explicit bug markers) and works through them one by one. Three models were run: Claude Sonnet 4.6, Claude Haiku 4.5, and Claude Opus 4.8, with 10 runs per mode for Sonnet and Haiku and 5 per mode for Opus.</p><p>The target API and the CLI test harness used to drive these runs are open-sourced:</p><ul><li><a href="https://github.com/OleksandrPodoliako/buggy-library-api">Target API</a></li><li><a href="https://github.com/OleksandrPodoliako/agentic-tester">Test harness</a></li></ul><h3>Results: cost and performance</h3><p>Of the 50 planned runs, a handful had to be excluded for reasons unrelated to the experiment itself — one Sonnet-explore run hit the 45-turn cap mid-thought with no final report, one Haiku-explore run failed with a 529 overloaded_error (an Anthropic API availability issue, not a model property), and one Haiku-test-cases run is a genuine oddity: the harness recorded it as &quot;completed,&quot; but the agent&#39;s entire output was a single turn asking for the base URL, no tool calls, 5.7 seconds elapsed. That run is excluded below and should be footnoted anywhere completion rate is quoted as a headline stat, since including it would silently inflate Haiku-test-cases&#39;s completion rate. That leaves 47 usable runs, which is the N behind every figure in this section: 9 Sonnet-explore, 10 Sonnet-test-cases, 9 Haiku-explore, 9 Haiku-test-cases, 5 Opus-explore, 5 Opus-test-cases.</p><p>With those exclusions, per-run cost (mean ± standard deviation, computed from actual token usage at Anthropic’s published per-token rates for these models — Sonnet $3/$15 per MTok in/out, Haiku $1/$5, Opus $5/$25) breaks age across all 47 usable runs gives approximately 15.6M tokens and an estimated total cost of $36.83. Separately, the actual Anthropic API billing export for the experiment period shows $40.58 in charges across the three models ($20.47 Sonnet, $9.57 Haiku, $10.54 Opus), all falling on two billing dates. The billing export is aggregated by date and model only, not by individual run or mode, so it can’t verify the per-run figures directly — but the token-derived total and the billed total sit within about 10% of each other, a reasonable sanity check rather than an independent confirmation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3JyMNdfB6U6K6h5UZ0KoNQ.png" /></figure><p>Test-cases mode is cheaper than explore mode for every model, but the ratio isn’t consistent: 5.9x for Sonnet, 2.3x for Haiku, 2.0x for Opus. The Sonnet ratio is the cleanest expression of what you’d expect — explore runs go deep (30+ turns) while test-cases runs are short and predictable (~7–8 turns). The Haiku ratio is smaller mostly because Haiku’s explore runs frequently burn through the full 45-turn cap without ever converging, and its test-cases runs are themselves comparatively bloated next to Sonnet’s. Opus’s test-cases mode has a notable outlier — one run took 24 turns and cost $1.4273, nearly 3x its four sibling runs, which pulls up both the mean and the standard deviation for that combination.</p><p>Summing token usage across all 47 usable runs gives approximately 15.6M tokens and an estimated total cost of $36.83. Separately, the actual Anthropic API billing export for the experiment period shows $40.58 in charges across the three models ($20.47 Sonnet, $9.57 Haiku, $10.54 Opus), all falling on two billing dates. The billing export is aggregated by date and model only, not by individual run or mode, so it can’t verify the per-run figures directly — but the token-derived total and the billed total sit within about 10% of each other, a reasonable sanity check rather than an independent confirmation.</p><h3>Results: bug detection against ground truth</h3><p>Test-cases mode has near-total recall on all 7 planted bugs, across all three models — unsurprising, since the checklist was built to trigger them. The one exception worth naming: a single Haiku-test-cases run missed both the date-range bug and the pagination bug despite both being deterministic backend defects against static seed data, which looks like the model failing to fully verify the response rather than genuine non-reproduction. Explore mode’s recall is bifurcated in a way that’s more interesting than a single aggregate number would suggest.</p><p>Four of the seven bugs — the 404-returns-200 bug, the missing DELETE auth, the zero-copy loan creation, and the invalid-email 500 — were found in essentially every completed explore run for Sonnet and Opus. Haiku is the exception: its usable explore sample is only 2 completed runs, and within those two it found the 404 bug in both, the DELETE-auth bug in one, the zero-copy loan bug in neither, and the invalid-email bug in one — a small enough sample that “essentially every run” simply doesn’t describe Haiku’s explore behavior, even though the same four bugs are close to universal for the other two models. For Sonnet and Opus, these four require no specific setup: an agent poking at the API in almost any reasonable order will trip over them (one caveat: 3 of Sonnet’s 9 explore runs described the invalid-email bug in a partial or inconsistent way rather than a clean repro, even though all 9 did surface it in some form).</p><p>The other three split into two very different failure patterns. The pagination off-by-one and the date-range boundary bug were found <em>inconsistently</em> — the date-range case, for instance, was found in 3 of 9 Sonnet-explore runs, 0 of 5 Opus-explore runs, and 0 of 2 completed Haiku-explore runs, with real variance across models and even run-to-run within Sonnet under identical prompting. Explore-mode agents often found <em>a</em> problem with the search endpoint (a 500 error, an inverted-range issue) without landing on the specific documented off-by-one — a partial-match situation that’s more of a judgment call than a clean hit or miss, and one where the underlying single defect may simply present different symptoms depending on which dates an agent happens to try.</p><p>The race condition is the sharp case. It was found in 0 of 23 usable explore runs across all three models — Sonnet, Haiku, and Opus alike — and in essentially every test-cases run, because only the fixed checklist scripts an explicit concurrent-request scenario. No explore-mode agent, on any model, ever spontaneously chose to fire two simultaneous requests at the same endpoint. One caveat on that “23”: only 2 of Haiku’s 9 usable explore runs actually produced a coherent testing pass far enough to have a real opportunity to test concurrency, so the true number of independent “chances” the race condition had to be found is somewhat lower than 23 suggests. Still, across every run that did get far enough to test meaningfully, none tried it. One thing worth being explicit about, since a skeptical reader will ask it: could the agent even issue two requests at once within this harness, or is “no agent chose concurrency” really a fact about the test rig rather than about model behavior? The data shows agents routinely make multiple tool calls within a single turn (Sonnet-explore averages ~2.6 tool calls per turn, for instance), so the harness isn’t limited to one HTTP call per turn — an agent emitting two POST /loans requests in the same turn was mechanically possible. What I can&#39;t confirm from the run data is whether the harness executes same-turn tool calls sequentially or genuinely in parallel, which matters for whether such an attempt would actually race against itself the way the bug requires. So the more defensible claim is narrower than &quot;structural&quot;: no agent, on any model, ever <em>tried</em> to issue simultaneous requests at the same endpoint, even though nothing in the harness obviously prevented the attempt — whether the harness would have executed that attempt concurrently enough to trigger the bug is a separate, unverified question.</p><p>One scoring nuance worth being explicit about: two of the checklist items covering the missing-auth bug are written as observational checks rather than strict pass/fail assertions, and several runs — notably some from Opus — marked the test as “passed” while describing the exact underlying defect in the narrative. Where that happened, the bug-detection numbers above use the behavioral finding rather than the agent’s own pass/fail label, because a raw pass/fail count would understate how consistently the defect was actually observed.</p><h3>Results: bonus findings</h3><p>This is where explore mode earns its cost. A set of issues outside the seven planted bugs recurred across nearly every explore run, regardless of model: duplicate ISBNs and duplicate member emails both accepted without a uniqueness check, negative available_copies values accepted on write, empty or whitespace-only required string fields accepted, and a 403 returned instead of 401 for missing (versus invalid) credentials. More strikingly, an authorization bypass on loan creation — a valid token belonging to one member could create a loan on behalf of a different member&#39;s ID — showed up repeatedly in Opus&#39;s explore runs specifically, alongside a referential-integrity gap where deleting a book orphans any active loans against it. A related but distinct bug showed up in some Sonnet-explore runs: any valid token could read any other member&#39;s profile, a read-path IDOR rather than the write-path one Opus kept surfacing. One further finding is worth calling out as more confidently real than a one-off: PUT /loans/{id}/return accepts no authentication at all, a second, distinct missing-auth issue beyond the documented DELETE-books bug. None of these were on the 22-item checklist, and none of them would have been found by test-cases mode by construction.</p><p>The flip side: a few less-certain findings need to be flagged rather than treated as confirmed. An XSS-style unescaped script tag in a book title showed up in exactly one Haiku-explore run — almost certainly leftover state carried over from a <em>prior</em> run’s own injection test rather than a newly discovered vulnerability, since the API happily stores whatever string it’s given, so this looks like a cross-run reset failure specifically. A pattern where GET /books returned 500 for limit values above roughly 15-20 appeared only in the later, higher-numbered Sonnet-explore runs; this one is a different kind of uncertainty — it may simply require a large accumulated record count to trigger, meaning it&#39;s a real bug with an under-tested precondition rather than evidence of a reset failure between runs. A third case, PUT /books/{id} with an all-null-fields payload corrupting the record or triggering a 500, appeared in three of nine Sonnet-explore runs and not the rest; plausibly a genuine bug that simply wasn&#39;t tried by every run, since the trigger is a specific edge case, but with only three occurrences and no confirmation elsewhere it&#39;s carried here as a probable-but-unconfirmed finding rather than a settled one. All three are noted in the underlying data with these caveats, and I&#39;m carrying them forward rather than presenting any of them as confirmed defects.</p><h3>Model comparison</h3><p>Test-cases mode narrows the gap between models dramatically. Haiku, run against the fixed checklist, catches the same 8–10 known-bug hits as Sonnet and Opus in 8 of 9 usable runs. It isn’t the cheapest option in this mode, though: at $0.3224 per run it’s actually about 21% <em>more</em> expensive than Sonnet’s $0.2660, and only cheaper than Opus’s $0.7118. The real story in test-cases mode isn’t Haiku being a bargain — it’s that all three models converge on similar recall and similar (low) cost once the checklist is doing the work, which erodes most of the reason to pay for a bigger model in this mode.</p><p>Explore mode is where the models diverge, and the standout number here isn’t a bug-detection rate at all: Haiku completed only 2 of its 10 explore runs. The other 8 either hit the 45-turn cap without producing a scorable final report or otherwise failed to converge — one to an infrastructure error unrelated to the model, the rest to the model simply not wrapping up. And the two runs that did complete show narrower coverage than either Sonnet or Opus: both got stuck re-verifying a GET /books 500 error and pagination edge cases, and neither ever reached the loans endpoint (0/2 on the zero-copy loan bug) — missing one of the &quot;easy&quot; bugs that Sonnet and Opus found in virtually every explore run. Invalid-email handling fared slightly better: one of the two runs did test it and found the bug, the other never tested email validation at all, so it&#39;s a 1/2 rather than a clean miss. Haiku&#39;s explore-mode failure mode looks less like &quot;found fewer bugs&quot; and more like &quot;frequently failed to complete a coherent testing pass within the turn budget,&quot; and it did this while burning a higher mean token count than Sonnet (686K vs. 481K) without converging. That 2/10 completion rate is a headline finding in its own right, not a footnote to the bug tables — any cost-per-finding comparison involving Haiku-explore has to lead with the fact that 8 of 10 runs produced no scorable output.</p><p>Opus versus Sonnet in explore mode is a more even trade. Opus completes in fewer turns and fewer total tokens (18 turns / 240K tokens vs. Sonnet’s 30.8 turns / 481K tokens) but at roughly double the per-token price, so net per-run cost ends up close ($1.40 vs. $1.57). Bug coverage on the “easy” four bugs is comparable between the two; Opus is slightly weaker on the pagination bug (2/5 vs. 4/9) and matches Sonnet’s zero detection on the race condition. Opus and Haiku both show zero detection on the date-range boundary bug in explore mode (0/5 and 0/2 respectively, versus Sonnet’s 3/9) — though Haiku’s n=2 makes that comparison hard to weigh, so the more defensible statement is that Opus, on a real 5-run sample, never found it, while Sonnet did on a minority of runs. Set against that, Opus does surface the authorization-bypass and orphaned-loan bonus findings more consistently than either other model, which is worth noting as a qualitative strength even without a clean quantitative comparison to back it.</p><h3>Limitations</h3><p>The sample sizes here are uneven and, for two of the six model/mode combinations, genuinely small. Opus has 5 runs per mode against 10 for Sonnet and Haiku, and after exclusions, Haiku-explore’s <em>usable-for-bug-detection</em> sample is just 2 completed runs — everything said above about Haiku-explore’s bug coverage rests on those two runs. Any variance claim involving Opus or Haiku-explore should be read as illustrative, not statistically robust; a defensible N of 10–15 per combination, which this dataset doesn’t yet reach everywhere, is a known gap rather than an oversight.</p><p>A few scoring decisions are judgment calls rather than clean facts. The observational-versus-assertive test case wording noted above means a raw “test passed” count for the missing-auth bug understates true detection consistency in some runs. Whether an explore run “found” the date-range boundary bug is itself a judgment call when the run describes a different search-related symptom instead of the specific documented behavior — the numbers above treat those as not-found unless the description specifically matches the exact-boundary-exclusion case, which is a conservative choice that could reasonably be argued the other way. And two of the three bonus findings flagged as less certain carry different kinds of risk: the XSS instance looks like a cross-run reset failure (state leaking from a prior run), while the limit-threshold 500 and the all-null-PUT bug look more like real defects with under-tested preconditions than reset artifacts — worth distinguishing rather than folding into one &quot;state leakage&quot; bucket, since state isolation between runs was a deliberate design goal of the harness and only one of the three findings actually implicates it.</p><p>Finally, the 45-turn cap on explore mode is a hard methodological constraint baked into every explore result here. A higher cap might improve Haiku’s completion rate and shift pagination or date-range recall for all three models — though it wouldn’t change the fact that no explore run, at any point, tried testing concurrency. As noted above, that’s a fact about agent behavior within a harness that technically allowed multiple tool calls per turn, not a claim that the harness made concurrency impossible to attempt.</p><h3>Conclusion</h3><p>The question worth answering isn’t “do we still need automated tests,” stated that broadly — nothing here suggests deterministic regression suites become optional. The narrower, answerable question is where LLM-agent exploration sits relative to structured test execution and to deterministic CI regression, and the data gives a fairly clean answer on that: explore mode is not a substitute for either. No explore run, on any model, ever tried issuing simultaneous requests at the same endpoint — and since the harness allowed multiple tool calls within a single turn, that looks like a real gap in agent behavior rather than a hard limit of the test rig, though whether same-turn tool calls would have executed concurrently enough to trigger the bug is a separate question this dataset can’t answer. Explore mode is also inconsistent on bugs that require a specific parameter combination rather than being tripped over by chance. What it’s good for is different — it’s a periodic audit that surfaces classes of defects a fixed checklist will never look for, because nobody thought to write that item down: authorization bypasses, referential-integrity gaps, input-validation holes that only a wandering tester stumbles into. Test-cases mode, meanwhile, is close to deterministic, several times cheaper, and has near-total recall on whatever bugs it was written to catch — which is exactly the trade-off a hand-written regression suite already makes, just executed by an agent instead of a CI pipeline.</p><p>The one result that doesn’t fit a clean “explore for breadth, script for reliability” narrative is Haiku’s explore-mode completion rate — though it’s worth flagging immediately that this reading rests on just 2 completed Haiku-explore runs out of 10 attempted, a sample size this dataset can only treat as a hypothesis to re-test at larger N, not a settled result. With that caveat up front: model capability turns out to matter more for whether explore mode <em>finishes at all</em> than for what it finds when it does. That’s a genuinely open question rather than a settled one, and it’s one of the more suggestive data points in this dataset for treating “let an agent explore your API” as something you’d currently want to run on a stronger, more expensive model, not because cheaper models find worse bugs, but because they may not reliably finish the job.</p><p>A next iteration of this would need: a larger N across all six model/mode combinations, particularly Opus and Haiku-explore, to move variance claims from illustrative to statistically supportable; a wider set of models to check whether Haiku’s completion problem is Haiku-specific or a smaller-model pattern generally; and an adversarial variant of the target API — one built after this article is public — to control for the possibility that future models have simply seen this specific bug set during training.</p><p><strong>Update (Oct 2026):</strong> After a reader comment prompted me to recheck the harness and the target API, I found that the race-condition result doesn’t hold. The harness executes tool calls sequentially, so it could never send overlapping requests. And the loan endpoint has no availability check at all, so the “race” was really the zero-copy loan bug showing up through the concurrency checklist item. The race-condition findings in this article should be disregarded; the remaining results are unaffected.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67dfede65986" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[UI integration tests for React applications with java]]></title>
            <link>https://oleksandr-podoliako.medium.com/ui-integration-tests-for-react-application-with-java-bb873af13841?source=rss-3df9d47575ee------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb873af13841</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[selenide]]></category>
            <category><![CDATA[automation-testing]]></category>
            <dc:creator><![CDATA[Oleksandr Podoliako]]></dc:creator>
            <pubDate>Tue, 11 Apr 2023 22:27:00 GMT</pubDate>
            <atom:updated>2023-04-29T08:07:42.585Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89on_4FE_XWhMICkMDjpyQ.jpeg" /></figure><p>In the past, I wrote about<a href="https://oleksandr-podoliako.medium.com/automated-testing-principles-6cffc32f7305?source=your_stories_page-------------------------------------"> how to tackle a major issue with UI end-to-end (E2E) tests</a>, which is the problem of tests being unreliable or “flaky.” Another approach to address this problem is to shift towards using UI integration tests, which are lower in the testing pyramid. The benefits of using integration tests instead of E2E tests are:</p><ul><li>Faster execution time.</li><li>Higher reliability.</li></ul><p>However, there are also some downsides to using integration tests compared to E2E tests, including:</p><ul><li>Difficulty in maintaining mock data.</li><li>Limited ability to detect certain defects.</li></ul><p>I will use a simple React application as the test subject, which displays cat facts obtained from <a href="https://catfact.ninja/fact">an external service</a>.</p><pre>import { useEffect, useState } from &quot;react&quot;;<br> <br>function App() {<br>  const [fact, setFact] = useState(&quot;&quot;);<br> <br>  useEffect(() =&gt; {<br>    fetch(process.env.REACT_APP_CAT_FACT_URL)<br>      .then((response) =&gt; response.json())<br>      .then((data) =&gt; setFact(data.fact))<br>      .catch((error) =&gt; console.log(error));<br>  }, []);<br> <br>  return (<br>    &lt;div className=&quot;App&quot;&gt;<br>      &lt;h1&gt;Cat Fact&lt;/h1&gt;<br>      &lt;p data-test-id=&quot;fact-text&quot;&gt;{fact}&lt;/p&gt;<br>    &lt;/div&gt;<br>  );<br>}<br> <br>export default App;</pre><p>React application has two modes to fetch from real service(https://catfact.ninja/fact) and to fetch from mock(http://localhost:8080/fact)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iMDMBkiF3WyuRLjT6PU-Mg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hiTkuiqL6LiSFPZanVWNfw.png" /></figure><p>In the test, I want to verify that the user is able to view a fact about a cat. However, we face two challenges:</p><ul><li>Our application depends on an external service that we don’t own and control, and we don’t want to actually test it.</li><li>Since the fact is random, our test can either be too abstract (not checking the actual fact) or flaky (varying results).</li></ul><p>Both these challenges can be addressed by using mock data to simulate the behavior of the external service in our tests. The framework is built using Java and utilizes Maven, Lombok, TestNG, Wirework, Cucumber, and Selenide. The architecture follows a classic layered approach. The framework is stored in the same repo as the application to maintain versioning.</p><pre>@integration<br>Feature: Data presentation<br> <br>  Scenario Outline: User is able to see cat fact<br>    When [Sys] Cat fact service is mocked<br>    When [UI] User opens main page<br>    Then [UI] The cat fact &lt;catFact&gt; should be shown<br> <br>    Examples: Examples:<br>      | catFact                                                              |<br>      | Approximately 40,000 people are bitten by cats in the U.S. annually. |</pre><p>The key difference from E2E tests is the addition of an extra step, where the external service is stubbed.</p><pre>@When(&quot;[Sys] Cat fact service is mocked&quot;)<br>public void catFactServiceIsMocked() throws IOException {<br>    String json = new String(Files.readAllBytes(Paths.get(mockJsonLocation() + &quot;/catFact.json&quot;)));<br> <br>    stubFor(get(urlEqualTo(&quot;/fact&quot;))<br>            .willReturn(aResponse()<br>                    .withStatus(200)<br>                    .withHeader(&quot;Access-Control-Allow-Origin&quot;, &quot;*&quot;)<br>                    .withBody(json)));<br> <br>}</pre><p>To deploy the application locally, you can use the command</p><pre>npm run start:development // external service is mocked<br>npm run start:production // external service is NOT mocked</pre><p>For manual checks and test development, you can utilize <a href="https://mockoon.com/">Mockoon</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QUASVc0r_F9Djcwqd1QgmA.png" /></figure><p>For automated test runs, <a href="https://wiremock.org/docs/">Wiremock</a> is used as a mock server.</p><pre>public class MockHooks {<br> <br>    private final static WireMockServer wireMockServer = new WireMockServer();<br> <br>    @BeforeAll<br>    public static void startWiremockServer() {<br>        wireMockServer.start();<br>    }<br> <br>    @AfterAll<br>    public static void stopWiremockServer() {<br>        wireMockServer.stop();<br>    }<br>}</pre><p>Jenkins pipeline consists of four steps:</p><ul><li>clone repo</li><li>deploy react application in develop mode(pointed to mock server)</li><li>run integration tests</li><li>generate report</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*orVvSd58BNidsrdSJYIalg.png" /></figure><pre>pipeline {<br>    agent any<br> <br>    stages {<br>        stage(&#39;Clone repo&#39;) {<br>            steps {<br>                git credentialsId: &#39;git&#39;, branch: &#39;main&#39;, url: &#39;https://gitlab.com/OleksandrPodoliako/cat-fact.git&#39;<br>            }<br>        }<br> <br>        stage(&#39;Build and start app&#39;) {<br>            steps {<br>                dir(&#39;app&#39;) {<br>                    bat &#39;npm install&#39;<br>                    bat &#39;start /B cmd /C &quot;npm run start:development | findstr /C:\&quot;Compiled successfully!\&quot;&quot;&#39;<br>                }<br>            }<br>        }<br> <br>        stage(&#39;Run integration tests&#39;) {<br>            steps {<br>                dir(&#39;integration-tests&#39;) {<br>                    bat &#39;mvn clean test -Pui-integration&#39;<br>                }<br>            }<br>        }<br> <br>        stage(&#39;Generate report&#39;) {<br>            steps {<br>                dir(&#39;integration-tests&#39;) {<br>                    publishHTML([allowMissing: false, alwaysLinkToLastBuild: true, keepAll: true, reportDir: &#39;target/cucumber-reports&#39;, reportFiles: &#39;cucumber-report.html&#39;, reportName: &#39;Cucumber Report&#39;])<br>                }<br>            }<br>        }<br>    }<br>}</pre><p>The full code can be found by the <a href="https://gitlab.com/OleksandrPodoliako/cat-fact">link</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb873af13841" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NeaRest — test automation library for API interaction]]></title>
            <link>https://oleksandr-podoliako.medium.com/nearest-test-automation-library-for-api-interaction-4013b3df60c9?source=rss-3df9d47575ee------2</link>
            <guid isPermaLink="false">https://medium.com/p/4013b3df60c9</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[api-testing]]></category>
            <category><![CDATA[restassured]]></category>
            <dc:creator><![CDATA[Oleksandr Podoliako]]></dc:creator>
            <pubDate>Fri, 16 Dec 2022 00:46:23 GMT</pubDate>
            <atom:updated>2023-01-08T20:10:46.001Z</atom:updated>
            <content:encoded><![CDATA[<h3>Nearest — test automation library for API interaction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89on_4FE_XWhMICkMDjpyQ.jpeg" /></figure><p>I have thought how classic architecture approach for building test automation framework for API testing with java can be improved for last two months. The first iteration was described in the <a href="https://oleksandr-podoliako.medium.com/test-automation-framework-for-api-testing-with-java-ffd80259fe87">article</a>. Nearest is result of second iteration, where I have decided to incapsulate logic with generics by moving it to dedicated library.</p><p>Nearest is applicable and efficient for two from three common tasks in automation testing, namely, to test business logic of application and to create pre- / post-conditions for UI tests. Nearest reduces amount of code, which needed to be written for interacting with REST services, by using generics. Nearest provides global logging configuration. Also, Nearest unifies manual and automation verifications by export/import functionality, which allows to use the same requests for both types of testing. Nearest supports import from Postman. Nearest does not limit Rest Assured functionality and provides easy access to it.</p><p>Nearest inherent HTTP two level structure. On first level Nearest has RequestWrapper and ResponseWrapper. On second level RequestWrapper has headers and body. ResponseWrapper has body and raw response. Also, Nearest has RestAssuredWrapper, which allow to do more specific actions with raw Rest Assured. Also, Nearest has IRestClient, which contains already implemented CRUD operations.</p><p>There are just 4 steps to set up Nearest</p><p>Add Nearest to the project</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;io.github.oleksandrpodoliako&lt;/groupId&gt;<br>    &lt;artifactId&gt;nearest&lt;/artifactId&gt;<br>    &lt;version&gt;${LAST_VERSION}&lt;/version&gt;<br>&lt;/dependency&gt;</pre><p>Create POJO model</p><pre>@Getter<br>@Setter<br>@Builder<br>@AllArgsConstructor<br>@JsonIgnoreProperties(ignoreUnknown = true)<br>public class Post {<br>    public int userId;<br>    public int id;<br>    public String title;<br>    public String body;<br><br>    public Post() {<br>    }<br>}</pre><p>Create Client class and add interface IRestClient. Specify request’s and response’s body types in interface</p><pre>public class PostClient implements IRestClient&lt;Post, Post&gt; {<br>}</pre><p>Now client class can be instantiated. Client object has all CRUD operations already implemented</p><pre>public class Main {<br>    private static final String URL = &quot;https://jsonplaceholder.typicode.com/posts/&quot;;<br><br>    public static void main(String[] args) {<br><br>        RequestWrapper&lt;Post&gt; requestWrapper = RequestWrapper.&lt;Post&gt;builder()<br>                .url(https://medium.com/URL)<br>                .apiMethod(APIMethod.GET_ARRAY)<br>                .build();<br><br>        ResponseWrapper&lt;Post&gt; responseWrapper = new PostClient().send(requestWrapper);<br>        <br>    }<br>}</pre><p>By default the global request and response logging levels are NONE. It can be changed easily</p><pre>NearestConfig.setRequestLogging(RequestLoggingLevel.ALL);<br>NearestConfig.setResponseLogging(ResponseLoggingLevel.ALL);</pre><p>Nearest allows to switch off mapping to object functionality</p><pre>NearestConfig.setMappingStrategy(MappingStrategy.NOT_TO_MAP);</pre><p>Nearest allows to use plain Rest Assured functionality</p><pre>RestAssuredWrapper.given().when().then();</pre><p>Nearest allows to convert all used requests to curl and store it to file. All you need to do is to set export strategy to ExportStrategy.TO_FILE(By default NOT_TO_EXPORT).</p><p>All requests will be converted to curl and stored to targets/export.txt. File can be changed with NearestConfig.setExportFileName() method</p><pre>NearestConfig.setExportStrategy(ExportStrategy.TO_FILE);</pre><p>Nearest allows to convert specific request to curl</p><pre>Converter.toCurl(https://medium.com/requestWrapper);</pre><p>Nearest allows to convert curl to RequestWrapper</p><pre>RequestWrapper&lt;Post&gt; requestWrapper = Converter.toRequestWrapper(curl, Post.class);</pre><p>Export/import functionality has limitations. Not all curl syntax is supported. Example of supported curl syntax provided below</p><pre>curl --request GET &#39;http://localhost:44444/posts/1&#39; \<br>--header &#39;Content-type: application/json; charset=UTF-8&#39; \<br>--data-raw {&quot;userId&quot;:72810,&quot;id&quot;:78256,&quot;title&quot;:&quot;Some Buried Caesar&quot;,&quot;body&quot;:&quot;Chick Publications&quot;}</pre><p>Nearest allows to import requests from Postman export json. All you need is to export collection from Postman, instantiate PostmanIntegration class with path to exported Postman collection json and generate requestWrapper with getRequestWrapper() method, which has three parameters: request name, body class, folders names</p><pre>PostmanIntegration postmanIntegration = new PostmanIntegration(exportedPostmanCollaction.json);<br>RequestWrapper&lt;Post&gt; requestWrapper = postmanIntegration.getRequestWrapper(&quot;request1&quot;, Post.class, &quot;folder1level&quot;, &quot;folder2level&quot;);</pre><p>Nearest source can be got by <a href="https://github.com/OleksandrPodoliako/nearest">link</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4013b3df60c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Test automation framework for UI testing with java]]></title>
            <link>https://oleksandr-podoliako.medium.com/test-automation-framework-for-ui-testing-with-java-fddd1e3fd75b?source=rss-3df9d47575ee------2</link>
            <guid isPermaLink="false">https://medium.com/p/fddd1e3fd75b</guid>
            <category><![CDATA[testng]]></category>
            <category><![CDATA[allure]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[cucumber]]></category>
            <dc:creator><![CDATA[Oleksandr Podoliako]]></dc:creator>
            <pubDate>Tue, 06 Dec 2022 15:13:43 GMT</pubDate>
            <atom:updated>2022-12-13T18:29:07.629Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89on_4FE_XWhMICkMDjpyQ.jpeg" /></figure><p>Previously I have written an article about <a href="https://oleksandr-podoliako.medium.com/test-automation-framework-for-api-testing-with-java-ffd80259fe87">testing API services with Java</a>. I think that it would be a nice idea to write an article about more general test framework, which will contain parts for testing Web UI, be capable to interact with DB and has logging and advanced reporting.</p><p>I have a few requirements to test object:<br>- it should have Web UI and API<br>- it should have DB<br>- it should be open source and free<br>- deploy should be local and easy</p><p>I have chosen <a href="https://kanboard.org/">KanBoard</a> because it fulfills all the requirements. KanBoard is open source, free software, which allows to create boards with tasks.</p><p>Deploy can be done with just one command, <strong>docker compose up</strong>, from the folder, which contains docker-compose.yml. For the test framework the file is put in a root folder. I will not describe how to install docker, this information can be gotten <a href="https://docs.docker.com/get-docker/">here</a>. By default the KanBoard will be available by the link <a href="http://127.0.0.1/login">http://127.0.0.1/login</a> , user has creds admin/admin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*44bIJYPsbV_-V-LoldKoYg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KHohTSO8YRtmwd7nDrjktQ.png" /></figure><p>The framework was written with java. Maven, Lombok, TestNG, Rest Assured, Cucumber, JDBI, Allure and Selenide are used. The framework consists of page objects; definitions, assertions and hooks; and features layers. Additionally the test framework has Config, Data generation, Shared data, API and DB components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MN8WRJNq093BOco1FciSJA.png" /></figure><p>As an example, the user can create task test was automated. A user, a project is created by API as part of preconditions. All these entities are removed by API as part of postconditions. Additionally, the task, which is created during the test by UI interactions, is removed by API in postconditions. Creating and removing testing entities increases tests isolation and improves test’s reliability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*fFW8XCr4Jv10OJ5rh4i5hA.png" /></figure><p>Config component has two classes EnvProperties and PropertiesReader. Component takes data from maven cli parameters. If a parameter is not provided, the component will take the parameter from the env.properties file. Better to get sensitive information from cli because storing sensitive information in repo is a bad practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*RfQzD3XpEjvah4OaiUSjPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*hlL-SoWgS7SBoVj4zn0iNQ.png" /></figure><p>Data generation component has DataProvider class and uses Faker library. Component generates objects with fake data. Id fields are exception.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*fvC_YAmObMcGP5zBBGwJxQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*dlE3sZVj0klwEuou05sR9g.png" /></figure><p>Shared data has two classes SharedDataSingelton and SharedData. Sharing data between steps is a problem for cucumber. Thread Singleton can be used to solve the problem. Thread Singleton gives assurance that every thread will have just one container object, which will be used to share data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*2n67KHkeiVFdxWXlvkfG8g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*4jBK_2Qf4aD7vAlIEfzixw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*i2a42DIqokC701t3My0yEg.png" /></figure><p>KanBoard has untypical API, which based on HTTP. Post method, with specific application method (for example getAllProjects) inside body, is used. More information about API can be gotten by the <a href="https://docs.kanboard.org/v1/api/">link</a>. API component uses modified generics approach from the previous <a href="https://oleksandr-podoliako.medium.com/test-automation-framework-for-api-testing-with-java-ffd80259fe87">article</a>. The goal of the approach is to use one method, which will be typed in tests. Object representation of request and response has complicated two level structure. The first level consists of RequestWrapper and ResponseWrapper, which contains HTTP metadata (Like headers) and application data. The second level consists of APIRequestModel, APIResponseArrayModel and APIResponseEntityModel, which contains application metadata (Like jsonrpc) and data(like tasks).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8lbEB5Zom4kSJrw0tp_1OQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EeBm4IJs1gWr0T7IqXzy0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fFD6BtVQ9AGQFFRYLwaV0w.png" /></figure><p>DB component is used to interact with DB and get object represented results. DB component has TaskDBController, DBController and DBConnection classes. DBConnection is built with using Thread Singleton. It was done to minimize resources spends. I do not recommend to use DB component to create test data because of data consistency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y6VBvX7jMDAplWBp0ErFaw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1PPkmA0qYiMbXwB3YHKpA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y2AFZRsvOVtLN2binKHD0g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kWr0GGAiD--fLa6whYyxSg.png" /></figure><p>UI component is used to interact with UI. UI component consists of page objects. Wrapper methods, which searching elements by text is used because some elements do not have good locators. In real projects it is recommended to add test ids([<a href="http://twitter.com/testId">@testId</a> = ‘taskLabel’]) to pages instead of using searching by text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h8ksW5JkwiMFIo05WRsrWg.png" /></figure><p>All components are used in definitions, assertions, and hooks. It allows to separate logic and reduce duplication of code.</p><p>TestCasesRunnerForTasksFeatures is created to run tests. The class should contain path to feature files and path to steps. Additionally, scenarios method is added, which allows to run tests in parallel. Count of threads can be set up by changing threadCount option in maven surefire plugin section in pom.xml file. Local run with intellij idea requires to install cucumber plugin and modify Run configuration with two options Main class: io.cucumber.core.cli.Main, Glue: steps.hooks steps.definitions steps.assertions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fJ0oKIEO1u-lSJcl_ELVjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*-e4n3YVabxlxy7infDV1jA.png" /></figure><p>Regression.xml file is created for cli test run. Regression.xml has link to TestCasesRunnerForTasksFeatures class. Regression.xml should be added as a value to suiteXmlFile option in maven surefire plugin section in pom.xml file. After that the tests can be run with a command <strong>mvn clean test</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oRlYmHXbTcUpUOcgGgWrg.png" /></figure><p>Allure is used as a reporting tool. Allure setting up requires to modify options argLine and allure.results.directory in maven surefire plugin section in pom.xml file. Also, <a href="https://docs.qameta.io/allure/#_installing_a_commandline">allure cli should be installed</a>. The folder target/allure-results with tests results will be created after tests run. Allure report can be deployed locally with command <strong>allure serve target/allure-results</strong>. ReportHooks is used to add screenshots.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-d-NhbnOA-lFDPfhj3Um5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTUYeiJyGlRtN-bf5dCjXg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dsRSsK5sDlNZLlWloBzEOg.jpeg" /></figure><p>Slf4j-simple and Rest Assured in-built functionality is used to do logging on definitions, assertions, and hooks level.</p><p>The full code can be found by the <a href="https://gitlab.com/OleksandrPodoliako/ui-tests">link</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fddd1e3fd75b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Test automation framework for API testing with java]]></title>
            <link>https://oleksandr-podoliako.medium.com/test-automation-framework-for-api-testing-with-java-ffd80259fe87?source=rss-3df9d47575ee------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffd80259fe87</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Oleksandr Podoliako]]></dc:creator>
            <pubDate>Fri, 07 Oct 2022 17:28:52 GMT</pubDate>
            <atom:updated>2022-11-17T17:50:38.484Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89on_4FE_XWhMICkMDjpyQ.jpeg" /></figure><p>Recently I have received a test task to write test automation framework for API testing with java. I think it could be interesting for others and I would like to share the results.</p><p>The framework was written with java. Maven, Lombok, testNG and Rest Assured were used. The framework consists of clients and test layers. If tests are planned to be E2E or business logic is complicated than business layer, which contains a few API clients, could be added. Also, cucumber layer could be added.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1-mLZaUyF-LYy7owL0hsDQ.jpeg" /></figure><p>The main idea is to generalize base API client, which contains main basic API methods (GET, POST, PUT and DELETE). This gives us an opportunity to use one client to interaction almost with all endpoints. Also, it allows to configure Rest Assured log in one place through baseAPIClient</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*2ymyl7aVK3-auBGqoibPuQ.jpeg" /></figure><p>Two wrappers, which were generalized, were created to have unified approach. RequestWrapper and ResponseWrapper get their types in tests. Also, they contain raw data from the Rest Assured.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*S_xXBqnrHTDh6vbOBzd9sQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c-YC0fdfhzJ-ftjVOYq8lw.jpeg" /></figure><p>BaseAPIClient can be used to interact with different endpoints, however I recommend to create separate client for every endpoint, which inherits from BaseAPIClient main functionality. Separate API clients could be expanded with specific API methods like search and filter. Base API method from BaseAPIClient can be overridden.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*W8HqfuYHwaq9IyDdFjl5uQ.jpeg" /></figure><p>I support ideas of isolation and managing state of application. You can read more about it in my previous article(<a href="https://oleksandr-podoliako.medium.com/automated-testing-principles-6cffc32f7305">Automated Testing Principles</a>). I encourage you to create and remove test data in before and after methods. It will increase tests reliability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*dLJiEYjHpoUzTBs1IXZIgg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*tc_dsmYXZPH3BC99Dbx13A.jpeg" /></figure><p>I recommend writing test, which cover one REST method, in one class because the preconditions are complicated and can be different for every REST method. Also, I recommend grouping a few asserts in one test method because of optimization. The more atomic approach is valid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*DcTFhf6iX17ekxQ_fRE1Zg.jpeg" /></figure><p>The test has presentation goal and is being failed because the testing API application does not actually create and remove entities.</p><p>The full code can be found by the link —<a href="https://gitlab.com/OleksandrPodoliako/api-tests"> gitlab</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffd80259fe87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Testing Principles]]></title>
            <link>https://oleksandr-podoliako.medium.com/automated-testing-principles-6cffc32f7305?source=rss-3df9d47575ee------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cffc32f7305</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[automation-testing]]></category>
            <dc:creator><![CDATA[Oleksandr Podoliako]]></dc:creator>
            <pubDate>Tue, 19 Jan 2021 17:21:50 GMT</pubDate>
            <atom:updated>2021-01-19T17:25:26.850Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GMjXdB2Vuu2KDmaIL9cb8g.jpeg" /></figure><p>The biggest challenge of automated testing is results certainty. The failed test does not give us assurance that we found a defect. This happens because there are a lot of possible reasons for failures. For instance, issues with environment, issues with tools, eligible changes due to application development or even purely designed tests. Due to a survey, which was conducted in the end of 2020, roughly 50% of 264 recipients had more than 5% flaky tests. Test failing requires human attention, which makes feedback time bigger. The worse thing is that automated tests cease to be automatic. I think there are 4 principles, which could make automated tests more reliable.</p><p>The first principle is isolation. The test should be isolated as much as possible from the outer environment and other tests. I often see that one test indirectly or even directly affects another or when some global environment config affects a group of tests. Obviously, isolation capabilities depend on an application. One approach is to use unique data for every test. For example, use different users, products, etc. The good idea is to analyze where and how different tests can interact with each other and reduce these interactions. The ultimate solution is to use different environments.</p><p>In this example the tests are isolated by testing data. The TEST 1 could be affected by TEST 2 without isolation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*6lhxKvBelAA6FJSDXBVjfQ.png" /></figure><p>The next principle is managing application state. An engineer should be able to manage application state through API or directly with database interaction. All outer services should be mocked. This gives us predictable application behavior. An engineer is sure that the application has the correct state when the test starts.</p><p>Also, the principal helps to create small, atomic tests, which are more readable and faster.</p><p>In this example the special product is created through API and we are sure that product is available. The product is removed after TEST 3 execution because we need to keep the environment clean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*6tJDBCg1kwFcOZfkWvKFQg.png" /></figure><p>The next principle is API over UI. The principle is heavily based on the previous one. The aim is to use the user interface just on a page where we want to check something. Proxy is a tool, which helps to discover what happens under the hood and then simulate it with API and Database interactions.</p><p>In this example the required authentication token is obtained through API. The product is added through API as well and we open just one page, where we do some checks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*r6Xxs-VbfcqKw9InGo7e4Q.png" /></figure><p>The last principle is functionality driven tests coverage. UI is extremely changeable on the development stage. I think this problem could be avoided by covering just core functionality. This is less possible that the buy button will disappear on a page, which aim is to sell something.</p><p>Following these principles requires special application design, which allows isolation and state managing. This requires additional efforts. However, I think this would make work easier for all team members, from QA to developers and what is more important is increasing the quality of the product.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cffc32f7305" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>