<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anton Belyaev on Medium]]></title>
        <description><![CDATA[Stories by Anton Belyaev on Medium]]></description>
        <link>https://medium.com/@avvero.abernathy?source=rss-6d700debbbbc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vUQWYNqs3tamvBtC0L1fng.png</url>
            <title>Stories by Anton Belyaev on Medium</title>
            <link>https://medium.com/@avvero.abernathy?source=rss-6d700debbbbc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:44:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@avvero.abernathy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Code That Isn’t Afraid of Change]]></title>
            <link>https://medium.com/@avvero.abernathy/code-that-isnt-afraid-of-change-22122f9e3124?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/22122f9e3124</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[refactoring]]></category>
            <category><![CDATA[evolvability]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Tue, 12 Aug 2025 05:38:36 GMT</pubDate>
            <atom:updated>2025-08-12T05:38:36.363Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eDcIrL9CBqlo-8KiPxvolg.png" /></figure><p>In this article, you’ll find some personal observations and tips on how to keep a project alive and healthy over the years. No illusions of omniscience. With a touch of healthy cynicism.</p><p>My personal experience is in building services as part of product development. In this article, I’m talking specifically about that kind of development — not about creating libraries, frameworks, databases, or other wonderful things. Nor am I touching on project-based development, where code is written to order and then handed off to others for maintenance.</p><h3>A Lyrical Prologue</h3><p>Do you remember when it first appeared? So small, beautiful, not overloaded with features, and untouched by bad advice from Uncle. Do you remember how it timidly rolled out to the staging environment, with the gentle hands of a DevOps helping it take its first steps? And its very first logs? Impossible to understand, but oh, how it kept writing them!</p><p>Then you went on vacation. The team promised to look after it, but overfed it with poorly tested features. It no longer fit within its old resource limits, and the containers were bursting at the seams. Years passed. Your sweet little angel turned into a clumsy, gluttonous monster. It stopped cleaning up after itself. It resisted any attempts to teach it new features. Where once a week was enough, now even a month isn’t.</p><p>Some people, at that point, leave for another company — hoping to find a good product owner, start new services, and avoid past mistakes. I suggest another path: write code that isn’t afraid of change, preserving its ability to grow without the need for a complete overhaul.</p><h3>Ability to Change</h3><p>Everything discussed in this article can be summed up in one word — evolvability. Evolvability is the ability of a system to change: to evolve, adapt, expand, or grow over time without requiring a complete rewrite. What matters is not just whether changes are possible, but how hard those changes are to make. The ideal scenario: adding the same feature should take roughly the same amount of effort in the first month of a project’s life as it does two or three years later. When I talk about code flexibility, I also mean its ability to change.</p><p>Evolvability is the opposite of software entropy: as a project ages, its development speed drops, and adding new features or fixing bugs becomes harder and harder. Each step requires more effort and time, which slows development velocity.</p><h3>Don’t Plan Too Far Ahead</h3><p>There’s an old saying: “If you want to make God laugh, tell Him your plans.” In software development, it fits perfectly. The most dangerous architectural mistake is trying to design the future in detail when that future is hazy and unknown. We’ve all been there: at the start of a project, trying to account for every possible feature, extension, and load scenario. You think, “What if we need multi-currency support in a year? Better build it in now.” Or you decide, “Let’s build for scale from day one — a flexible plugin system, just in case.”</p><p>A year later, the project has taken an entirely different turn: multi-currency is forgotten, but suddenly you need horoscope integration instead. All that “just in case” code is not only useless — it’s a hindrance. You spend time maintaining code that might be useful someday instead of delivering features that are actually needed.</p><p>The YAGNI principle (You Aren’t Gonna Need It) is where true product flexibility shows itself. And if the mention of horoscopes triggered you, that’s exactly the point — you can’t predict the future. Even the wisest developers (yes, even the ones who sneeze right after speaking the truth) can’t tell you where the business will go or what the system will look like in a couple of years. So there’s no point in overcomplicating the design for hypothetical requirements. It’s better to build a minimal working solution today and improve it as real requirements come in.</p><p>This isn’t a call to code thoughtlessly — thinking about the future is useful. But there’s a fine line between flexibility and overengineering. You don’t need to create a universal Swiss Army knife right away. You need to make it easy to change later. It’s not about how simple the code is — it’s about how much effort a change will take and what risks it will carry. Code shouldn’t get in the way of future changes; it should support them.</p><p>And this especially applies to tests. Jumping ahead a bit, I’ll say: tests play the key role here — more on that later. (Alright, you caught me — that’s one thing I can predict.)</p><h4>Don’t Plan Ahead — Even You</h4><p>A quick side note. Even if you’ve guessed the product’s direction correctly many times and your instincts have become legendary — maybe it’s time to try horse betting. At least there, you’re risking your own money and not someone else’s business. (This is not investment advice.)</p><h3>Premature Generality</h3><p>Product development teaches humility: even the most elegant abstractions quickly run into awkward edge cases. More often than not, plain duplication is better than premature generalization.</p><p>A classic example: you get a request — “We need another slightly different report format.” You already have a `ReportGenerator` class with a couple of subclasses for different formats. The temptation is strong — create a third subclass, or better yet, design a universal mega-class with plugins for any format. But if these reports truly differ by just a couple of fields, it’s often faster and more reliable to… yes, copy the existing generator code and tweak it for the new case. Sure, it sounds like “embracing copy-paste.” But you’ll get the correct result quickly, without breaking existing logic.</p><p>Anticipating the cries about maintainability: yes, code duplication has a cost. Two similar modules will need to be updated in parallel for shared changes. But in my experience, that’s the lesser evil compared to premature abstraction. An overly clever, generalized system often turns out inflexible. The moment you encounter a case that doesn’t fit your elegant hierarchy, you’ll spend more time reshaping your abstractions than you would have spent maintaining two independent, simple solutions.</p><p>The golden rule here is simple: abstract only after something has happened three times. In other words:<br>- Once — no reason to abstract at all.<br>- Twice — a weak signal; a third time may be coming.<br>- Thrice — it’s time to extract a common mechanism.</p><p>Following this rule helps avoid the bad habit of “writing a framework for a single task.” I’ve seen it described in various sources under names like The Rule of Three Envelopes, Triangulation, 1–2-refactor, or simply The Rule of Three.</p><h3>Accept It: You Won’t Understand Everything Right Away</h3><p>One of the key truths: you won’t know everything from the start. Big features often begin with vague requirements. Half of them are incomplete, and the other half only seem clear. As a result, the first version of the system is built almost blindly — and that’s fine. The goal isn’t to get it right the first time, but to make it possible to redo it later.</p><p>Many developers fear rewriting “what’s already done.” The time spent feels too precious to throw away. But often, discarding and rewriting is faster and safer. Code isn’t carved in stone, even if we strive to write it beautifully and cleanly. If it turns out the current solution doesn’t scale well or fails to cover an important new scenario, it’s better to admit it and rework it than to heroically drag along a flawed implementation.</p><p>In practice, code flexibility means that refactoring and redesign are routine, not disasters. Your code should be ready for it. The architecture should allow components, modules, or algorithms to be swapped out with minimal effort. If a module is designed so that removing it requires rewriting the entire system — that’s a red flag.</p><p>Accepting the limits of your knowledge leads to more adaptable design: you don’t build for specific anticipated extensions, you build for change in general. That means shifting from “we accounted for all possible scenarios in advance” to “we made it so that adding a new scenario is relatively painless.”</p><h3>Iterations Over Revolutions</h3><p>A product’s life is a continuous sequence of iterations. Instead of shipping a massive release every six months, teams tweak screws weekly — or even daily — adding small features, fixing bugs, and making minor improvements. This pace shapes the development approach: decisions are quick and localized. Code is written to “solve the problem now,” with the understanding that another developer may come along later and rewrite it to suit new needs.</p><p>Speaking of developers — over the product’s lifetime, you’ll probably see a whole parade of them. Each will bring their own style and ideas. Somewhere deep in the project are files whose authors barely remember what they wrote back in 2017. In such conditions, trying to “plan everything in advance” is especially pointless: a newcomer with fresh eyes will look at your carefully crafted architecture and think, “Why is this so complicated? Let’s simplify.” Then they’ll rework half the system.</p><p>Iterativeness means being ready to change course in small steps. It’s like adjusting a ship’s heading: small maneuvers instead of a sharp 90-degree turn. A project that evolves iteratively is easier to adapt — each small improvement tests hypotheses and allows for timely course correction. In contrast, trying to lock in a finished architecture from the start often ends in massive refactoring — or even a rewrite — once it becomes clear reality doesn’t match the plan.</p><h3>Feature Grafting</h3><p>In gardening, there’s a technique called “grafting”: attaching a branch from one plant onto the trunk of another. Software development sometimes works the same way. When you want to quickly grow a new feature, it’s often easier to graft it into an existing service rather than immediately building a separate module or microservice. You embed the new functionality “on foreign territory,” taking advantage of the resources and context of existing code. Once that branch grows strong, takes root, and proves its value, you can carefully separate it and transplant it into its own “pot” (a dedicated service or module).</p><p>This approach helps avoid premature complexity. It’s tempting to fear touching an old monolith or large service, and to push every new idea into a separate place “so as not to break anything.” But the truth is that early-stage internal integration is often faster and more reliable. You already have the infrastructure, configurations, and database — why create extra entities if you can quickly plug into what exists?</p><p>Of course, this approach has downsides: the codebase temporarily swells, boundaries of responsibility blur, and old code gets “extra baggage.” But if you keep the end goal in mind (extracting the feature into its own component), that debt can be paid off gradually. In return, you validate the idea with minimal cost, and later refactor cleanly, removing the grafted code from its original host.</p><h3>API Is Not Set in Stone</h3><p>Ever notice how some internal API docs say, “This method is deprecated, use the new one” — and keep saying that for several versions in a row? That’s just evolution at work. No matter how much you try to design interfaces and contracts perfectly, reality proves one thing: APIs change. Maybe less often than internal logic, but they still change.</p><p>This is especially true for internal APIs between services. There’s no such thing as going five years without a single endpoint signature changing. Even if the contract itself stays stable, new versions appear, old ones get phased out, or new parameters get added. A mature product lives in a state of constant restructuring, and APIs are part of that life.</p><p>So how do you live with it? First — versioning. Accept that sooner or later you’ll have to roll out v2, v3… It’s better to design for multiple API versions upfront, so you’re not breaking clients every time. Second — deprecation policy. Have a strategy for retiring old functionality, even if it’s just marking it “Deprecated” and giving it a year of support before removal. That’s far better than trying to keep the first version alive forever out of fear of user backlash. And users adapt — better to give them a clear plan (“The old API will be shut down in N months, please migrate to the new one”) than to silently change behavior and then awkwardly avoid the topic.</p><p>One more thing — flexibility in integrations. If your product interacts with external APIs, be ready to rewrite things when the partner updates theirs. Budget time for it in your planning. Few things stall evolution as effectively as a suddenly broken integration you didn’t allocate resources to fix.</p><h3>Tests: Safety and Freedom</h3><p>I’ll never get tired of saying it: tests are a flexible code’s best friends. When you have hundreds of modules and services, it’s all too easy to accidentally break a contract or a dependency. Tests need maintenance too, and bad tests can hinder progress as much as bad code. But living without at least some tests is far scarier. Without them, every refactoring becomes a game of roulette: “Will it break or won’t it?” And fear of changing code is the number one killer of flexibility.</p><p>My personal choice — proven over years of working on various services — is the Testing Trophy approach. It focuses on integration tests centered around observable behavior. When tests are decoupled from the implementation, you can change that implementation every day and still remain safe.</p><p>I also want to call out TDD (Test-Driven Development). Some debate its overall value as a methodology, but for me there’s one major benefit: it separates the effort of writing code to fulfill a contract from the effort of shaping the code’s design. You’re not trying to nail both the perfect design and the correct behavior in one pass — you split those tasks. This gives your code room to rest and mature while awaiting refactoring. In the context of our topic, that means fewer chances to overbuild or overlook something important.</p><h4>One Pile</h4><p>That bit about “letting the code rest” wasn’t just a throwaway line. There’s an interesting practice called One Pile, described in Kent Beck’s book Tidy First?. The idea is that understanding code is harder than writing it. If the code is split into too many tiny pieces, it can be useful to first combine it into one whole to see the overall structure, and only then break it back down into methods and modules. This is especially helpful in the early stages, when the task and its boundaries are still unclear.</p><h3>Conclusion</h3><p>The ability to change isn’t about a specific language or framework — it’s about an approach to development. It’s about building systems that can shed their skin, drop their tail, grow new limbs — and still keep doing their job. There’s no silver bullet, but there is a mindset: don’t carve everything in granite, be ready for change, stay open to it, and don’t fall so in love with your own solutions that you become blind to better ones.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22122f9e3124" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are seniors actually for?]]></title>
            <link>https://medium.com/@avvero.abernathy/what-are-seniors-actually-for-ad794de5673c?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad794de5673c</guid>
            <category><![CDATA[senior-developer]]></category>
            <category><![CDATA[humor]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Thu, 07 Aug 2025 04:55:29 GMT</pubDate>
            <atom:updated>2025-08-07T04:55:29.294Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Rx59u_EEsAlhsPpduja5Q.png" /></figure><p>There are three things you can watch forever: a product manager pushing the deadline, an architect aligning arrows, and feathers flying when two seniors clash. All thrilling, no doubt — but it still begs the question: what are seniors actually for?</p><p>Let’s recall who’s on the team:<br>- Product Manager — his only product so far is making it into Forbes 30 under 30.<br>- Team Lead — a climate activist: ensures everything in the team is soft, cozy, and free from toxic emissions. No longer a developer, not yet a manager.<br>- Middle — the backbone, carrying it all. Dreams of becoming a senior so they can “think instead of do.” But performance review gets in the way: “Needs more technical initiative… yada yada yada.”<br>- Junior — extinct. Replaced by Copilot.<br>- Senior — joined last. But hey, thanks for showing up.</p><p>Sometimes, to get things moving, you need to bring pain. Say the unpleasant. Push through the hard stuff. Go one-on-one with a senior from another team. Slip changes past the pipeline, hotfix in prod. Who else is going to do it?</p><p>A sharp team lead always keeps a well-fed senior nearby — one who shares the values but isn’t bound by corporate fluff. Someone who can occasionally bend best practices. This senior can dive into fire, ship to prod, or DM the architect across the aisle. When it’s time, they’re unleashed. They strike, tear through, get shit done.</p><p>Of course, the team lead scolds them afterwards — for the record. But quietly throws them a treat. Because hey — results.</p><p>A senior is the perfect tool for dirty work. When you need more progress and less consensus. A blend of boldness, individualism, and controlled chaos.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad794de5673c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Message Testing]]></title>
            <link>https://medium.com/@avvero.abernathy/kafka-message-testing-648884aa2b9c?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/648884aa2b9c</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[spring]]></category>
            <category><![CDATA[spring-boot]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Thu, 05 Sep 2024 05:33:43 GMT</pubDate>
            <atom:updated>2024-09-05T05:33:43.998Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7qXCaJZLKQ3caphqi9I1sQ.jpeg" /></figure><p>This article offers an approach to writing integration tests for Kafka-based applications that focuses on interaction specification, making tests more readable and easier to maintain. The proposed method not only enhances testing efficiency but also contributes to a better understanding of the integration processes within the application.</p><p>The article builds on three ideas presented in relevant articles: <a href="https://medium.com/@avvero.abernathy/ordering-chaos-arranging-http-request-testing-in-spring-c625520d2418">writing tests with a clear separation of Arrange-Act-Assert stages</a>, <a href="https://medium.com/@avvero.abernathy/isolation-in-testing-with-kafka-16e00f5d5d7e">isolation in Kafka tests</a>, and <a href="https://medium.com/@avvero.abernathy/enhancing-the-visibility-of-http-tests-24ce9493aae3">using tools to enhance test visibility</a>. I recommend reviewing these before delving into the material of this article.</p><h3>Demonstration Scenario</h3><p>Let’s take a Telegram bot that forwards requests to the OpenAI API and returns the result to the user as an example. If the request to OpenAI violates the system’s security rules, the client will be notified. Additionally, a message will be sent to Kafka for the behavioral control system so that the manager can contact the user, explain that their request was too sensitive even for our bot, and ask them to review their preferences.</p><p>The interaction contracts with services are described in a simplified manner to emphasize the core logic. Below is a sequence diagram demonstrating the application’s architecture. I understand that the design may raise questions from a system architecture perspective, but please approach it with understanding — the main goal here is to demonstrate the approach to writing tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/0*HYdbrCqb0lkWWBIU.png" /></figure><h4>Message Capture</h4><p>The main testing tool will be the message capture object — <a href="https://github.com/avvero/kafka-test-support/blob/sb3/kafka-support/src/main/java/pw/avvero/test/kafka/RecordCaptor.java">RecordCaptor</a>. Its operation is quite similar to the outgoing request capture object — <a href="https://github.com/avvero/spring-sandbox/blob/main/request-captor/src/main/java/pw/avvero/test/http/RequestCaptor.java">RequestCaptor</a>, which can be read about in the article <a href="https://medium.com/@avvero.abernathy/ordering-chaos-arranging-http-request-testing-in-spring-c625520d2418">Ordering Chaos: Arranging HTTP Request Testing in Spring</a>.</p><p>Message capture will be performed through a standard Kafka consumer. The list of topics must be specified explicitly via a configuration parameter.</p><pre>@KafkaListener(id = &quot;recordCaptor&quot;, topics = &quot;#{&#39;${test.record-captor.topics}&#39;.split(&#39;,&#39;)}&quot;, groupId = &quot;test&quot;)<br>public void eventCaptorListener(ConsumerRecord&lt;Object, Object&gt; record,<br>                                @Headers Map&lt;String, Object&gt; boundedHeaders) {<br>    RecordSnapshot recordSnapshot = mapper.recordToSnapshot(record, boundedHeaders);<br>    recordCaptor.capture(recordSnapshot);<br>}</pre><p>The <strong>RecordCaptor </strong>object accumulates information from captured messages.</p><p>Using this approach requires adhering to <a href="https://medium.com/@avvero.abernathy/isolation-in-testing-with-kafka-16e00f5d5d7e">isolation in Kafka tests</a>. Waiting for offset commit confirmation before verifying test results should be done using the <a href="https://github.com/avvero/kafka-test-support/blob/6595086188252f4ed89dff24e2129d75e26d9ece/kafka-support/src/main/java/pw/avvero/test/kafka/KafkaSupport.java#L109">KafkaSupport#waitForPartitionOffsetCommit</a> method.</p><h4>Test Example</h4><p>Below is the test code for the described scenario.</p><pre>def &quot;User Message Processing with OpenAI&quot;() {<br>    setup:<br>    KafkaSupport.waitForPartitionAssignment(applicationContext)                           // 1<br>    and:                                                                                  // 2<br>    def openaiRequestCaptor = restExpectation.openai.completions(withBadRequest().contentType(APPLICATION_JSON)<br>            .body(&quot;&quot;&quot;{<br>                &quot;error&quot;: {<br>                &quot;code&quot;: &quot;content_policy_violation&quot;,<br>                &quot;message&quot;: &quot;Your request was rejected as a result of our safety system.&quot;<br>                }<br>            }&quot;&quot;&quot;))<br>    def telegramRequestCaptor = restExpectation.telegram.sendMessage(withSuccess(&#39;{}&#39;, APPLICATION_JSON))<br>    when:<br>    mockMvc.perform(post(&quot;/telegram/webhook&quot;)                                             // 3<br>            .contentType(APPLICATION_JSON_VALUE)<br>            .content(&quot;&quot;&quot;{<br>                &quot;message&quot;: {<br>                &quot;from&quot;: {<br>                    &quot;id&quot;: 10000000<br>                },<br>                &quot;chat&quot;: {<br>                    &quot;id&quot;: 20000000<br>                },<br>                &quot;text&quot;: &quot;Hello!&quot;<br>                }<br>            }&quot;&quot;&quot;.toString())<br>            .accept(APPLICATION_JSON_VALUE))<br>            .andExpect(status().isOk())<br>    KafkaSupport.waitForPartitionOffsetCommit(applicationContext)                         // 4<br>    then:<br>    openaiRequestCaptor.times == 1                                                        // 5<br>    JSONAssert.assertEquals(&quot;&quot;&quot;{<br>        &quot;content&quot;: &quot;Hello!&quot;<br>    }&quot;&quot;&quot;, openaiRequestCaptor.bodyString, false)<br>    and:<br>    telegramRequestCaptor.times == 1<br>    JSONAssert.assertEquals(&quot;&quot;&quot;{<br>        &quot;chatId&quot;: &quot;20000000&quot;,<br>        &quot;text&quot;: &quot;Your request was rejected as a result of our safety system.&quot;<br>    }&quot;&quot;&quot;, telegramRequestCaptor.bodyString, false)<br>    when:                                                                                 // 6<br>    def message = recordCaptor.getRecords(&quot;topicC&quot;, &quot;20000000&quot;).last<br>    then:<br>    message != null<br>    JSONAssert.assertEquals(&quot;&quot;&quot;{<br>        &quot;webhookMessage&quot;: {<br>        &quot;message&quot;: {<br>            &quot;chat&quot;: {<br>            &quot;id&quot;: &quot;20000000&quot;<br>            },<br>            &quot;text&quot;: &quot;Hello!&quot;<br>        }<br>        },<br>        &quot;error&quot;: {<br>            &quot;code&quot;: &quot;content_policy_violation&quot;,<br>            &quot;message&quot;: &quot;Your request was rejected as a result of our safety system.&quot;<br>        }<br>    }&quot;&quot;&quot;, message.value as String, false)<br>}</pre><p>Key steps:</p><ol><li>Wait for partition assignment before starting the test scenario.</li><li>Mock requests to OpenAI and Telegram.</li><li>Execute the test scenario.</li><li>Wait for offset confirmation.</li><li>Verify requests to OpenAI and Telegram.</li><li>Check the message in Kafka.</li></ol><p>Using <strong>JSONAssert.assertEquals</strong> ensures consistency in data representation across Kafka messages, logs, and tests. This simplifies testing by providing flexibility in comparison and accuracy in error diagnosis.</p><p>The article provides an example with JSON message format; other formats are not covered, but the described approach does not impose format restrictions.</p><h4>How to Find Your Message in RecordCaptor</h4><p>Messages in <strong>RecordCaptor </strong>are organized by topic name and key. In the provided test, the key used is the Kafka message key. When sending, we explicitly specify it:</p><pre>sendMessage(&quot;topicC&quot;, chatId, ...);<br>...<br>private void sendMessage(String topic, String key, Object payload) {<br>    Message message = MessageBuilder<br>            .withPayload(objectMapper.writeValueAsString(payload))<br>            .setHeader(KafkaHeaders.TOPIC, topic)<br>            .setHeader(KafkaHeaders.KEY, key)  &lt;-- set key<br>            .build();<br>    kafkaTemplate.send(message).get();<br>}</pre><p>To search by message key within a topic:</p><pre>when:                                                                                <br>def message = recordCaptor.getRecords(&quot;topicC&quot;, &quot;20000000&quot;).last &lt;-- use key</pre><p>If this option is not suitable, you need to describe your own indexes based on message parameters for constructing the search. An example can be seen in the tests <a href="https://github.com/avvero/kafka-test-support/blob/sb3/example-testcontainers/src/test/groovy/pw/avvero/example/feature1/PolicyViolationTestsCustomIndex.groovy">PolicyViolationTestsCustomIndex.groovy</a>.</p><h4>Connecting RecordCaptor</h4><p>The code for connecting <strong>RecordCaptor </strong>looks as follows:</p><pre>@TestConfiguration(proxyBeanMethods = false)<br>public class RecordCaptorConfiguration {<br>    @Bean<br>    RecordCaptor recordCaptor() {<br>        return new RecordCaptor();<br>    }<br><br>    @Bean<br>    RecordCaptorConsumer recordCaptorConsumer(RecordCaptor recordCaptor) {<br>        return new RecordCaptorConsumer(recordCaptor, new RecordSnapshotMapper());<br>    }<br>}</pre><h3>OffsetSnapshotFrame</h3><p>Experience has shown that working with Kafka-based applications requires tools to facilitate understanding the state of consumers and message consumption status. For this task, you can compare topic offsets and consumer groups in the offset confirmation waiting operation and log discrepancies, as illustrated in the image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xVP84Z09XkAtU684rB7xMw.png" /></figure><p>The code for <a href="https://github.com/avvero/kafka-test-support/blob/sb3/kafka-support/src/main/java/pw/avvero/test/kafka/OffsetSnapshotFrame.java">OffsetComparisonFrame</a> is available for review.</p><h3>Conclusion</h3><p>Testing messages in Kafka using the proposed approach not only simplifies test writing but also makes it more structured and understandable. Utilizing tools like <strong>RecordCaptor</strong>, as well as adhering to isolation principles and clear separation of test stages, ensures high accuracy and efficiency.</p><p>Link to the project repository with test demonstrations — <a href="https://github.com/avvero/kafka-test-support/tree/sb3/example-testcontainers">kafka-test-support</a>.</p><p>Thank you for reading the article, and good luck in your efforts to write effective and clear tests!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=648884aa2b9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Garden]]></title>
            <link>https://medium.com/@avvero.abernathy/digital-garden-a197323784ae?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a197323784ae</guid>
            <category><![CDATA[notes]]></category>
            <category><![CDATA[obsidian]]></category>
            <category><![CDATA[notetaking]]></category>
            <category><![CDATA[digital-garden]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Fri, 30 Aug 2024 12:16:21 GMT</pubDate>
            <atom:updated>2024-08-30T12:16:51.416Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrYVLMVdnjrHQmIXvkhuHA.jpeg" /></figure><p>The topic of note-taking remains relevant today. We know the benefits it provides to the author. We are familiar with various approaches to note-taking, tools that can be used, and we have choices. Imagine you have found your approach, your tool, and your note base is growing and pleasing to the eye. What next? I want to discuss one path of development in this area.</p><p>This article is dedicated to the concept of a digital garden — the philosophy of publicly maintaining personal notes.</p><h3>My Path</h3><p>For the past 20 years, I have kept notes using methods that lacked systematization, reliability, or usefulness: paper notebooks, text files, Evernote, and other applications whose names have faded from memory. Three years ago, I started using the Zettelkasten style (as I understood and adapted it for myself). I learned about this approach from <a href="https://github.com/rwxrob">Rob Muhlestein</a> whom I encountered on Twitch. Experiments led me to my current note-taking method: in the form of Markdown files stored in a Git repository. On my computer, I work with them in VSCode, and on my phone, I use the default “Notes” app. Why not Obsidian? I tried it and decided to stick with VSCode for the following reasons:</p><ol><li>VSCode is a basic editor that is always open and used for work files.</li><li>I could not set up Obsidian synchronization via Git, and iCloud synchronization caused app crashes on my phone.</li><li>I need a quickly accessible app on my phone to jot down fleeting thoughts before, for example, reaching for a roll of toilet paper and experiencing discomfort from waiting. At the same time, I need the note base on my phone much less frequently than the need to urgently write something down.</li></ol><p>Separating apps for sudden thoughts and the main base has an advantage: it creates a ritual of transferring notes from the phone app to the main base. I review fresh notes, tag them, and add details. This allows me to revisit the recorded thought and increase the chances of not forgetting it.</p><h3>What is a Digital Garden</h3><p>The phrase “digital garden” is a metaphor describing an approach to note-taking. It is not just a set of tools like WordPress plugins or Jekyll templates. The idea of a garden is familiar to all of us — it is a place where something grows. Gardens can be very personal and filled with gnome figurines, or they can be sources of food and vitality. And who knows what a sudden visitor to your garden might see? You in a lovely pajama with a glass of fresh juice under an apple tree? Or perhaps standing upside down, trying to bring a bit of order and pull out weeds?</p><p>The digital gardening metaphor emphasizes the slow growth of ideas through writing, rewriting, editing, and revisiting thoughts in a public space. Instead of fixed opinions that never change, this approach allows ideas to develop over time. The goal of digital gardening is to use the collective intelligence of your network to create constructive feedback loops. If done right, you will have an accessible representation of your thoughts that can be “sent out” into the world, and people will be able to respond to it. Even for the most raw ideas, it helps to create a feedback loop to strengthen and fully develop the idea.</p><p>Core principles of gardening:</p><ol><li>Connections over timelines. Gardens are organized around contextual and associative connections; concepts and themes within each note define how it relates to others. The publication date is not the most important aspect of the text.</li><li>Continuous growth. Gardens never end; they are constantly growing, evolving, and changing, like a real garden.</li><li>Imperfection. Gardens are inherently imperfect. They do not hide their rough edges and do not claim to be a permanent source of truth.</li><li>Learning in public. To create constructive feedback loops.</li><li>Personal and experimental. Gardens are inherently heterogeneous. You may plant the same seeds as your neighbor but get a different arrangement of plants. You organize the garden around ideas and means that fit your way of thinking rather than a standard template.</li><li>Independent ownership. Gardening is about creating your own little corner of the internet that you fully control.</li></ol><h4>Learning in Public</h4><p>This point may raise questions, as it suggests sharing something for free. The approach implies that you publicly document your steps, thoughts, mistakes, and successes in mastering a new topic or skill. This allows you not only to share the result but also to demonstrate the thinking and learning process, which can be useful to others.</p><p>Besides altruistic motives, personal interest is worth noting. Sometimes I am lazy to clearly formulate thoughts in notes, which then backfires — I can’t understand what I meant. It’s amusing how I always rely on my future superpower to decipher my own nonsensical notes — a belief that remains unshakable, though entirely unfounded. But when it comes to others, I don’t allow such naivety. Knowing that my note might not only attract attention but also genuinely help someone motivates me to make an effort and articulate the thought properly.</p><h3>How to Share</h3><p>Today, convenient tools have made creating a fully customizable website much easier. Services like Netlify and Vercel have removed deployment complexities. Static site generators like Jekyll, Gatsby, 11ty, and Next simplify creating complex sites that automatically generate pages and handle load time, image optimization, and SEO.</p><p>Obsidian offers the ability to publish notes through its subscription platform. Using this service does not feel like “independent ownership.”</p><p>I chose <a href="https://quartz.jzhao.xyz/">Quartz</a> for myself. It is a free static generator based on Markdown content. Quartz is designed primarily as a tool for publishing digital gardens on the internet. It is simple enough for people without technical experience but powerful enough for customization by experienced developers.</p><h3>My Digital Garden</h3><p>As mentioned, I store my notes in a public Git repository. Most of it is in Russian, with some in English. Changes are automatically published on GitHub Pages and available on <a href="https://devirium.avvero.pw/">my digital garden page</a>. The design, solution scheme, and GitHub Actions scripts are available for review in <a href="https://devirium.avvero.pw/2024/2024-07/How-I-Built-Devirium">this note</a> if you want to create something similar. Some use RSS for updates, while I use a Telegram channel for this purpose. It is specifically an update channel, messages are not posted separately.</p><h3>Materials for Further Study</h3><p>The concept has a philosophy and history; I won’t retell them but will provide links where you can read more: <a href="https://maggieappleton.com/">https://maggieappleton.com</a>. Examples of digital gardens can be found at <a href="https://github.com/MaggieAppleton/digital-gardeners">https://github.com/MaggieAppleton/digital-gardeners </a>and <a href="https://github.com/jackyzha0/quartz/blob/v4/docs/showcase.md">https://github.com/jackyzha0/quartz/blob/v4/docs/showcase.md</a>, as well as through GitHub search if the repository is tagged with the relevant topic — <a href="https://github.com/topics/digital-garden">https://github.com/topics/digital-garden</a>.</p><h3>Conclusion</h3><p>Establishing a digital garden has been a logical continuation of my journey, started with the Zettelkasten method. How has it affected me? After the initial effort to set up and deploy the system, I hardly maintain it except for occasional issues. And now I continue to push my notes to Git. The only thing is, I have started to make them more understandable.</p><p>Thank you for reading the article, and good luck in your quest for organizing thoughts and creating an effective space for ideas!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a197323784ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Methodcentipede]]></title>
            <link>https://medium.com/@avvero.abernathy/methodcentipede-94096e748120?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/94096e748120</guid>
            <category><![CDATA[antipattern]]></category>
            <category><![CDATA[patterns]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Fri, 23 Aug 2024 05:52:53 GMT</pubDate>
            <atom:updated>2024-08-23T15:25:48.876Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eWTDH_3VHTqGVa9DrAdlBg.png" /></figure><p>When I was a child, I used to lie on the bed and gaze for a long time at the patterns on an old Soviet rug, seeing animals and fantastical figures within them. Now, I more often look at code, but similar images still emerge in my mind. Like on the rug, these images form repetitive patterns. They can be either pleasing or repulsive. Today, I want to tell you about one such unpleasant pattern that can be found in programming.</p><h3>Scenario</h3><p>Imagine a service that processes a client registration request and sends an event about it to the Kafka. In this article, I will show an implementation example that I consider an antipattern and suggest an improved version.</p><h4>Option 1: Methodcentipede</h4><p>The Java code below shows the code of the <strong>RegistrationService</strong> class, which processes the request and sends the event.</p><pre>public class RegistrationService {<br><br>    private final ClientRepository clientRepository;<br>    private final KafkaTemplate&lt;Object, Object&gt; kafkaTemplate;<br>    private final ObjectMapper objectMapper;<br><br>    public void registerClient(RegistrationRequest request) {<br>        var client = clientRepository.save(Client.builder()<br>                .email(request.email())<br>                .firstName(request.firstName())<br>                .lastName(request.lastName())<br>                .build());<br>        sendEvent(client);<br>    }<br><br>    @SneakyThrows<br>    private void sendEvent(Client client) {<br>        var event = RegistrationEvent.builder()<br>                .clientId(client.getId())<br>                .email(client.getEmail())<br>                .firstName(client.getFirstName())<br>                .lastName(client.getLastName())<br>                .build();<br>        Message message = MessageBuilder<br>                .withPayload(objectMapper.writeValueAsString(event))<br>                .setHeader(KafkaHeaders.TOPIC, &quot;topic-registration&quot;)<br>                .setHeader(KafkaHeaders.KEY, client.getEmail())<br>                .build();<br>        kafkaTemplate.send(message).get();<br>    }<br><br>    @Builder<br>    public record RegistrationEvent(int clientId, String email, String firstName, String lastName) {}<br>}</pre><p>The structure of the code can be simplified as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/0*r_nQorX4XMIbQ5n2.png" /></figure><p>Here, you can see that the methods form an unbroken chain through which the data flow, like through a long, narrow intestine. The methods in the middle of this chain are responsible not only for the logic directly described in their body but also for the logic of the methods they call and their contracts (e.g., the need to handle specific errors). All methods preceding the invoked one inherit its entire complexity. For example, if <strong>kafkaTemplate.send</strong> has a side effect of sending an event, then the calling <strong>sendEvent</strong> method also acquires the same side effect. The <strong>sendEvent</strong> method also becomes responsible for serialization, including handling its errors. Testing individual parts of the code becomes more challenging because there is no way to test each part in isolation without using mocks.</p><h4>Option 2: Improved Version</h4><p>Code:</p><pre>public class RegistrationService {<br><br>    private final ClientRepository clientRepository;<br>    private final KafkaTemplate&lt;Object, Object&gt; kafkaTemplate;<br>    private final ObjectMapper objectMapper;<br><br>    @SneakyThrows<br>    public void registerClient(RegistrationController.RegistrationRequest request) {<br>        var client = clientRepository.save(Client.builder()<br>                .email(request.email())<br>                .firstName(request.firstName())<br>                .lastName(request.lastName())<br>                .build());<br>        Message&lt;String&gt; message = mapToEventMessage(client);<br>        kafkaTemplate.send(message).get();<br>    }<br><br>    private Message&lt;String&gt; mapToEventMessage(Client client) throws JsonProcessingException {<br>        var event = RegistrationEvent.builder()<br>                .clientId(client.getId())<br>                .email(client.getEmail())<br>                .firstName(client.getFirstName())<br>                .lastName(client.getLastName())<br>                .build();<br>        return MessageBuilder<br>                .withPayload(objectMapper.writeValueAsString(event))<br>                .setHeader(KafkaHeaders.TOPIC, &quot;topic-registration&quot;)<br>                .setHeader(KafkaHeaders.KEY, event.email)<br>                .build();<br>    }<br><br>    @Builder<br>    public record RegistrationEvent(int clientId, String email, String firstName, String lastName) {}<br>}</pre><p>The diagram is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/0*FQtenqjQb7w2a23F.png" /></figure><p>Here, you can see that the <strong>sendEvent</strong> method is completely absent, and <strong>kafkaTemplate.send</strong> is responsible for sending the message. The entire process of constructing the message for Kafka has been moved to a separate <strong>mapToEventMessage</strong> method. The <strong>mapToEventMessage</strong> method has no side effects, and its responsibility boundaries are clearly defined. Exceptions related to serialization and message sending are part of the individual methods’ contracts and can be handled separately.</p><p>The <strong>mapToEventMessage</strong> method is a pure function. When a function is deterministic and has no side effects, we call it a “pure” function. Pure functions are:</p><ul><li>easier to read,</li><li>easier to debug,</li><li>easier to test,</li><li>independent of the order in which they are called,</li><li>simple to run in parallel.</li></ul><h3>Recommendations</h3><p>I would suggest the following techniques that can help avoid such antipatterns in the code:</p><ul><li>Testing Trophy Approach</li><li>One Pile Technique</li><li>Test-Driven Development (TDD)</li></ul><p>All these techniques are closely related and complement each other.</p><h4>Testing Trophy</h4><p>This is an approach to test coverage that emphasizes integration tests, which verify the service’s contract as a whole. Unit tests are used for individual functions that are difficult or costly to test through integration tests. I have described tests with this approach in my articles: <a href="https://medium.com/@avvero.abernathy/ordering-chaos-arranging-http-request-testing-in-spring-c625520d2418">Ordering Chaos: Arranging HTTP Request Testing in Spring</a>, <a href="https://medium.com/@avvero.abernathy/enhancing-the-visibility-of-http-tests-24ce9493aae3">Enhancing the Visibility of Integration Tests</a>, <a href="https://medium.com/@avvero.abernathy/isolation-in-testing-with-kafka-16e00f5d5d7e">Isolation in Testing with Kafka</a>.</p><h4>One Pile</h4><p>This technique is described in Kent Beck’s book “Tidy First?” The main idea is that reading and understanding code is harder than writing it. If the code is broken into too many small parts, it may be helpful to first combine it into a whole to see the overall structure and logic, and then break it down again into more understandable pieces.</p><p>In the context of this article, it is suggested not to break the code into methods until it ensures the fulfillment of the required contract.</p><h4>Test-Driven Development</h4><p>This approach allows separating the efforts of writing code to implement the contract and designing the code. We don’t try to create a good design and write code that meets the requirements simultaneously, but instead, we separate these tasks. The development process looks like this:</p><p>1. Write tests for the service contract using the Testing Trophy approach.</p><p>2. Write code in the One Pile style, ensuring that it fulfills the required contract. Don’t worry about code design quality.</p><p>3. Refactor the code. All the code is written, and we have a complete understanding of the implementation and potential bottlenecks.</p><h3>Conclusion</h3><p>The article discusses an example of an antipattern that can lead to difficulties in maintaining and testing code. Approaches like Testing Trophy, One Pile, and Test-Driven Development allow you to structure your work in a way that prevents code from turning into an impenetrable labyrinth. By investing time in the proper organization of code, we lay the foundation for the long-term sustainability and ease of maintenance of our software products.</p><p>Thank you for your attention to the article, and good luck in your quest for writing simple code!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94096e748120" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testcontainers-based Load Testing Bench]]></title>
            <link>https://medium.com/@avvero.abernathy/testcontainers-based-load-testing-bench-112a275f549d?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/112a275f549d</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[testcontainer]]></category>
            <category><![CDATA[spock-framework]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Thu, 06 Jun 2024 05:53:43 GMT</pubDate>
            <atom:updated>2024-06-06T05:53:43.918Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nCQwiErXyfXchBmk.jpeg" /></figure><p>Using Testcontainers has radically improved the process of working with test scenarios. Thanks to this tool, creating environments for integration tests has become simpler (see the article <a href="https://medium.com/@avvero.abernathy/isolation-in-testing-with-kafka-16e00f5d5d7e">Isolation in Testing with Kafka</a>). Now we can easily launch containers with different versions of databases, message brokers, and other services. For integration tests, Testcontainers has proven indispensable. Although load testing is less common than functional testing, it can be much more enjoyable. Studying graphs and analyzing the performance of a particular service can bring real pleasure. Such tasks are rare, but they are especially exciting for me.</p><p>The purpose of this article is to demonstrate an approach to creating a setup for load testing in the same way that regular integration tests are written: in the form of Spock tests using Testcontainers in a Gradle project environment. Load testing utilities such as Gatling, WRK, and Yandex.Tank are used.</p><h3>Creating a Load Testing Environment</h3><p>Toolset: Gradle + Spock Framework + Testcontainers. The implementation variant is a separate Gradle module. Load testing utilities used are Gatling, WRK, and Yandex.Tank.</p><p>There are two approaches to working with the test object:</p><ul><li>Testing published images;</li><li>Building images from the project’s source code and testing.</li></ul><p>In the first case, we have a set of load tests that are independent of the project’s version and changes. This approach is easier to maintain in the future, but it is limited to testing only published images. We can, of course, manually build these images locally, but this is less automated and reduces reproducibility. When running in CI/CD without the necessary images, the tests will fail.</p><p>In the second case, the tests are run on the latest version of the service. This allows for integrating load tests into CI and obtaining performance data changes between service versions. However, load tests usually take longer than unit tests. The decision to include such tests in CI as part of the quality gate is up to you.</p><p>This article considers the first approach. Thanks to Spock, we can run tests on multiple versions of the service for comparative analysis:</p><pre>where:<br>image | _<br>&#39;avvero/sandbox:1.0.0&#39; | _<br>&#39;avvero/sandbox:1.1.0&#39; | _</pre><p>It is important to note that the goal of this article is to demonstrate the organization of the testing space, not full-scale load testing.</p><h4>Target Service</h4><p>For the testing object, let’s take a simple HTTP service named Sandbox, which publishes an endpoint and uses data from an external source to handle requests. The service has a database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*3qJ2lyBsuLf6v15A8jRyYQ.png" /></figure><p>The source code of the service, including the Dockerfile, is available in the project repository <a href="https://github.com/avvero/spring-sandbox">spring-sandbox</a>.</p><h4>Module Structure Overview</h4><p>As we delve into the details later in the article, I want to start with a brief overview of the structure of the `load-tests` Gradle module to provide an understanding of its composition:</p><pre>load-tests/<br>|-- src/<br>|   |-- gatling/<br>|   |   |-- scala/<br>|   |   |   |-- MainSimulation.scala         # Main Gatling simulation file<br>|   |   |-- resources/<br>|   |   |   |-- gatling.conf                 # Gatling configuration file<br>|   |   |   |-- logback-test.xml             # Logback configuration for testing<br>|   |-- test/<br>|   |   |-- groovy/<br>|   |   |   |-- pw.avvero.spring.sandbox/<br>|   |   |   |   |-- GatlingTests.groovy      # Gatling load test file<br>|   |   |   |   |-- WrkTests.groovy          # Wrk load test file<br>|   |   |   |   |-- YandexTankTests.groovy   # Yandex.Tank load test file<br>|   |   |-- java/<br>|   |   |   |-- pw.avvero.spring.sandbox/<br>|   |   |   |   |-- FileHeadLogConsumer.java # Helper class for logging to a file<br>|   |   |-- resources/<br>|   |   |   |-- wiremock/<br>|   |   |   |   |-- mappings/                # WireMock setup for mocking external services<br>|   |   |   |   |   |-- health.json          <br>|   |   |   |   |   |-- forecast.json<br>|   |   |   |-- yandex-tank/                 # Yandex.Tank load testing configuration<br>|   |   |   |   |-- ammo.txt<br>|   |   |   |   |-- load.yaml<br>|   |   |   |   |-- make_ammo.py<br>|   |   |   |-- wrk/                         # LuaJIT scripts for Wrk<br>|   |   |   |   |-- scripts/                 <br>|   |   |   |   |   |-- getForecast.lua<br>|-- build.gradle</pre><p>Project repository — <a href="https://github.com/avvero/testing-bench.">https://github.com/avvero/testing-bench.</a></p><h3>Environment</h3><p>From the description above, we see that the service has two dependencies: the service <em>https://external-weather-api.com</em> and a database. Their description will be provided below, but let’s start by enabling all components of the scheme to communicate in a Docker environment — we’ll describe the network:</p><pre>def network = Network.newNetwork()</pre><p>and provide network aliases for each component. This is extremely convenient and allows us to statically describe the integration parameters.</p><p>Dependencies such as WireMock and the load testing utilities themselves require configuration to work. These can be parameters that can be passed to the container or entire files and directories that need to be mounted to the containers. In addition, we need to retrieve the results of their work from the containers. To solve these tasks, we need to provide two sets of directories:</p><ul><li><em>workingDirectory</em><strong> </strong>— the module’s resource directory, directly at `load-tests/`.</li><li><em>reportDirectory</em> — the directory for the results of the work, including metrics and logs. More on this will be in the section on reports.</li></ul><h4>Database</h4><p>The Sandbox service uses Postgres as its database. Let’s describe this dependency as follows:</p><pre>def postgres = new PostgreSQLContainer&lt;&gt;(&quot;postgres:15-alpine&quot;)<br>        .withNetwork(network)<br>        .withNetworkAliases(&quot;postgres&quot;)<br>        .withUsername(&quot;sandbox&quot;)<br>        .withPassword(&quot;sandbox&quot;)<br>        .withDatabaseName(&quot;sandbox&quot;)</pre><p>The declaration specifies the network alias <em>postgres</em>, which the Sandbox service will use to connect to the database. To complete the integration description with the database, the service needs to be provided with the following parameters:</p><pre>&#39;spring.datasource.url&#39; : &#39;jdbc:postgresql://postgres:5432/sandbox&#39;,<br>&#39;spring.datasource.username&#39; : &#39;sandbox&#39;,<br>&#39;spring.datasource.password&#39; : &#39;sandbox&#39;,<br>&#39;spring.jpa.properties.hibernate.default_schema&#39;: &#39;sandbox&#39;</pre><p>The database structure is managed by the application itself using Flyway, so no additional database manipulations are needed in the test.</p><h4>Mocking Requests to https://external-weather-api.com</h4><p>If we don’t have the possibility, necessity, or desire to run the actual component in a container, we can provide a mock for its API. For the service <em>https://external-weather-api.com</em>, WireMock is used.</p><p>The declaration of the WireMock container will look like this:</p><pre>def wiremock = new GenericContainer&lt;&gt;(&quot;wiremock/wiremock:3.5.4&quot;)<br>        .withNetwork(network)<br>        .withNetworkAliases(&quot;wiremock&quot;)<br>        .withFileSystemBind(&quot;${workingDirectory}/src/test/resources/wiremock/mappings&quot;, &quot;/home/wiremock/mappings&quot;, READ_WRITE)<br>        .withCommand(&quot;--no-request-journal&quot;)<br>        .waitingFor(new LogMessageWaitStrategy().withRegEx(&quot;.*https://wiremock.io/cloud.*&quot;))<br>wiremock.start()</pre><p>WireMock requires mock configuration. The <em>withFileSystemBind</em> instruction describes the file system binding between the local file path and the path inside the Docker container. In this case, the directory <em>”${workingDirectory}/src/test/resources/wiremock/mappings”</em> on the local machine will be mounted to <em>/home/wiremock/mappings</em> inside the WireMock container. Below is an additional part of the project structure to understand the file composition in the directory:</p><pre>load-tests/<br>|-- src/<br>|   |-- test/<br>|   |   |-- resources/<br>|   |   |   |-- wiremock/<br>|   |   |   |   |-- mappings/               <br>|   |   |   |   |   |-- health.json<br>|   |   |   |   |   |-- forecast.json</pre><p>To ensure that the mock configuration files are correctly loaded and accepted by WireMock, you can use a helper container:</p><pre>helper.execInContainer(&quot;wget&quot;, &quot;-O&quot;, &quot;-&quot;, &quot;http://wiremock:8080/health&quot;).getStdout() == &quot;Ok&quot;</pre><p>The helper container is described as follows:</p><pre>def helper = new GenericContainer&lt;&gt;(&quot;alpine:3.17&quot;)<br>        .withNetwork(network)<br>        .withCommand(&quot;top&quot;)</pre><p>By the way, IntelliJ IDEA version 2024.1 <a href="https://blog.jetbrains.com/idea/2024/02/intellij-idea-2024-1-eap-7">introduced support for WireMock</a>, and the IDE provides suggestions when forming mock configuration files.</p><h4>Target Service Launch Configuration</h4><p>The declaration of the Sandbox service container looks as follows:</p><pre>def javaOpts = &#39; -Xloggc:/tmp/gc/gc.log -XX:+PrintGCDetails&#39; +<br>        &#39; -XX:+UnlockDiagnosticVMOptions&#39; +<br>        &#39; -XX:+FlightRecorder&#39; +<br>        &#39; -XX:StartFlightRecording:settings=default,dumponexit=true,disk=true,duration=60s,filename=/tmp/jfr/flight.jfr&#39;<br>def sandbox = new GenericContainer&lt;&gt;(image)<br>        .withNetwork(network)<br>        .withNetworkAliases(&quot;sandbox&quot;)<br>        .withFileSystemBind(&quot;${reportDirectory}/logs&quot;, &quot;/tmp/gc&quot;, READ_WRITE)<br>        .withFileSystemBind(&quot;${reportDirectory}/jfr&quot;, &quot;/tmp/jfr&quot;, READ_WRITE)<br>        .withEnv([<br>                &#39;JAVA_OPTS&#39;                                     : javaOpts,<br>                &#39;app.weather.url&#39;                               : &#39;http://wiremock:8080&#39;,<br>                &#39;spring.datasource.url&#39;                         : &#39;jdbc:postgresql://postgres:5432/sandbox&#39;,<br>                &#39;spring.datasource.username&#39;                    : &#39;sandbox&#39;,<br>                &#39;spring.datasource.password&#39;                    : &#39;sandbox&#39;,<br>                &#39;spring.jpa.properties.hibernate.default_schema&#39;: &#39;sandbox&#39;<br>        ])<br>        .waitingFor(new LogMessageWaitStrategy().withRegEx(&quot;.*Started SandboxApplication.*&quot;))<br>        .withStartupTimeout(Duration.ofSeconds(10))<br>sandbox.start()</pre><p>Notable parameters and JVM settings include:</p><ul><li>Collection of garbage collection event information.</li><li>Use of Java Flight Recorder (JFR) to record JVM performance data.</li></ul><p>Additionally, directories are configured for saving diagnostic results of the service.</p><h4>Logging</h4><p>If you need to see the logs of any container to a file, which is likely necessary during the test scenario writing and configuration stage, you can use the following instruction when describing the container:</p><pre>.withLogConsumer(new FileHeadLogConsumer(&quot;${reportDirectory}/logs/${alias}.log&quot;))</pre><p>In this case, the <em>FileHeadLogConsumer</em> class is used, which allows writing a limited amount of logs to a file. This is done because the entire log is likely not needed in load testing scenarios, and a partial log will be sufficient to assess whether the service is functioning correctly.</p><h3>Implementation of Load Tests</h3><p>There are many tools for load testing. In this article, I propose to consider using three of them: Gatling, Wrk, and Yandex.Tank. All three tools can be used independently of each other.</p><h4>Gatling</h4><p>Gatling is an open-source load testing tool written in Scala. It allows creating complex testing scenarios and provides detailed reports. The main simulation file of Gatling is connected as a Scala resource to the module, making it convenient to work with using the full range of support from IntelliJ IDEA, including syntax highlighting and navigation through methods for documentation reference.</p><p>The container configuration for Gatling is as follows:</p><pre>def gatling = new GenericContainer&lt;&gt;(&quot;denvazh/gatling:3.2.1&quot;)<br>        .withNetwork(network)<br>        .withFileSystemBind(&quot;${reportDirectory}/gatling-results&quot;, &quot;/opt/gatling/results&quot;, READ_WRITE)<br>        .withFileSystemBind(&quot;${workingDirectory}/src/gatling/scala&quot;, &quot;/opt/gatling/user-files/simulations&quot;, READ_WRITE)<br>        .withFileSystemBind(&quot;${workingDirectory}/src/gatling/resources&quot;, &quot;/opt/gatling/conf&quot;, READ_WRITE)<br>        .withEnv(&quot;SERVICE_URL&quot;, &quot;http://sandbox:8080&quot;)<br>        .withCommand(&quot;-s&quot;, &quot;MainSimulation&quot;)<br>        .waitingFor(new LogMessageWaitStrategy()<br>                .withRegEx(&quot;.*Please open the following file: /opt/gatling/results.*&quot;)<br>                .withStartupTimeout(Duration.ofSeconds(60L * 2))<br>        );<br>gatling.start()</pre><p>The setup is almost identical to other containers:</p><ul><li>Mount the directory for reports from <em>reportDirectory</em>.</li><li>Mount the directory for configuration files from <em>workingDirectory</em>.</li><li>Mount the directory for simulation files from <em>workingDirectory</em>.</li></ul><p>Additionally, parameters are passed to the container:</p><ul><li>The <em>SERVICE_URL</em> environment variable with the URL value for the Sandbox service. Although, as mentioned earlier, using network aliases allows hardcoding the URL directly in the scenario code.</li><li>The <em>-s MainSimulation</em> command to run a specific simulation.</li></ul><p>Here is a reminder of the project source file structure to understand what is being passed and where:</p><pre>load-tests/<br>|-- src/<br>|   |-- gatling/<br>|   |   |-- scala/<br>|   |   |   |-- MainSimulation.scala         # Main Gatling simulation file<br>|   |   |-- resources/<br>|   |   |   |-- gatling.conf                 # Gatling configuration file<br>|   |   |   |-- logback-test.xml             # Logback configuration for testing</pre><p>Since this is the final container, and we expect to get results upon its completion, we set the expectation <em>.withRegEx(“.*Please open the following file: /opt/gatling/results.*”)</em>. The test will end when this message appears in the container logs or after <em>60 * 2</em> seconds.</p><p>I will not delve into the DSL of this tool’s scenarios. You can check out the code of the used scenario in the <a href="https://github.com/avvero/testing-bench/blob/main/load-tests/src/gatling/scala/MainSimulation.scala">project repository</a>.</p><h4>Wrk</h4><p><a href="https://github.com/wg/wrk">Wrk</a> is a simple and fast load testing tool. It can generate significant load with minimal resources. Key features include:</p><ul><li>Support for Lua scripts to configure requests.</li><li>High performance due to multithreading.</li><li>Ease of use with minimal dependencies.</li></ul><p>The container configuration for Wrk is as follows:</p><pre>def wrk = new GenericContainer&lt;&gt;(&quot;ruslanys/wrk&quot;)<br>        .withNetwork(network)<br>        .withFileSystemBind(&quot;${workingDirectory}/src/test/resources/wrk/scripts&quot;, &quot;/tmp/scripts&quot;, READ_WRITE)<br>        .withCommand(&quot;-t10&quot;, &quot;-c10&quot;, &quot;-d60s&quot;, &quot;--latency&quot;, &quot;-s&quot;, &quot;/tmp/scripts/getForecast.lua&quot;, &quot;http://sandbox:8080/weather/getForecast&quot;)<br>        .waitingFor(new LogMessageWaitStrategy()<br>                .withRegEx(&quot;.*Transfer/sec.*&quot;)<br>                .withStartupTimeout(Duration.ofSeconds(60L * 2))<br>        )<br>wrk.start()</pre><p>To make Wrk work with requests to the Sandbox service, the request description via a Lua script is required, so we mount the script directory from <em>workingDirectory</em>. Using the command, we run Wrk, specifying the script and the URL of the target service method. Wrk writes a report to the log based on its results, which can be used to set expectations.</p><h4>Yandex.Tank</h4><p>Yandex.Tank is a load testing tool developed by Yandex. It supports various load testing engines, such as JMeter and Phantom. For storing and displaying load test results, you can use the free service <a href="https://overload.yandex.net/">Overload</a>.</p><p>Here is the container configuration:</p><pre>copyFiles(&quot;${workingDirectory}/src/test/resources/yandex-tank&quot;, &quot;${reportDirectory}/yandex-tank&quot;)<br>def tank = new GenericContainer&lt;&gt;(&quot;yandex/yandex-tank&quot;)<br>        .withNetwork(network)<br>        .withFileSystemBind(&quot;${reportDirectory}/yandex-tank&quot;, &quot;/var/loadtest&quot;, READ_WRITE)<br>        .waitingFor(new LogMessageWaitStrategy()<br>                .withRegEx(&quot;.*Phantom done its work.*&quot;)<br>                .withStartupTimeout(Duration.ofSeconds(60L * 2))<br>        )<br>tank.start()</pre><p>The load testing configuration for Sandbox is represented by two files: <em>load.yaml</em> and <em>ammo.txt</em>. As part of the container description, configuration files are copied to the <em>reportDirectory</em>, which will be mounted as the working directory. Here is the structure of the project source files to understand what is being passed and where:</p><pre>load-tests/<br>|-- src/<br>|   |-- test/<br>|   |   |-- resources/<br>|   |   |   |-- yandex-tank/                <br>|   |   |   |   |-- ammo.txt<br>|   |   |   |   |-- load.yaml<br>|   |   |   |   |-- make_ammo.py</pre><h3>Reports</h3><p>Test results, including JVM performance recordings and logs, are saved in the directory `build/${timestamp}`, where `${timestamp}` represents the timestamp of each test run.</p><p>The following reports will be available for review:</p><ul><li>Garbage Collector logs.</li><li>WireMock logs.</li><li>Target service logs.</li><li>Wrk logs.</li><li>JFR (Java Flight Recording).</li></ul><p>If Gatling was used:</p><ul><li>Gatling report.</li><li>Gatling logs.</li></ul><p>If Wrk was used:</p><ul><li>Wrk logs.</li></ul><p>If Yandex.Tank was used:</p><ul><li>Yandex.Tank result files, with an additional upload to [Overload](https://overload.yandex.net/).</li><li>Yandex.Tank logs.</li></ul><p>The directory structure for the reports is as follows:</p><pre>load-tests/<br>|-- build/<br>|   |-- ${timestamp}/<br>|   |   |-- gatling-results/<br>|   |   |-- jfr/<br>|   |   |-- yandex-tank/<br>|   |   |-- logs/<br>|   |   |   |-- sandbox.log<br>|   |   |   |-- gatling.log<br>|   |   |   |-- gc.log<br>|   |   |   |-- wiremock.log<br>|   |   |   |-- wrk.log<br>|   |   |   |-- yandex-tank.log<br>|   |-- ${timestamp}/<br>|   |-- ...</pre><h3>Conclusion</h3><p>Load testing is a crucial phase in the software development lifecycle. It helps assess the performance and stability of an application under various load conditions. This article presented an approach to creating a load testing environment using Testcontainers, which allows for easy and efficient setup of the testing environment.</p><p>Testcontainers significantly simplifies the creation of environments for integration tests, providing flexibility and isolation. For load testing, this tool enables the deployment of necessary containers with different versions of services and databases, making it easier to conduct tests and improve result reproducibility.</p><p>The provided configuration examples for Gatling, Wrk, and Yandex.Tank, along with container setup, demonstrate how to effectively integrate various tools and manage testing parameters. Additionally, the process of logging and saving test results was described, which is essential for analyzing and improving application performance. This approach can be expanded in the future to support more complex scenarios and integration with other monitoring and analysis tools.</p><p>Thank you for your attention to this article, and good luck in your endeavor to write useful tests!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=112a275f549d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing the Visibility of Integration Tests]]></title>
            <link>https://medium.com/@avvero.abernathy/enhancing-the-visibility-of-http-tests-24ce9493aae3?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/24ce9493aae3</guid>
            <category><![CDATA[spock]]></category>
            <category><![CDATA[spring-framework]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[spring]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Tue, 02 Apr 2024 08:44:44 GMT</pubDate>
            <atom:updated>2024-06-11T09:23:52.432Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TmKs_MpR_TRQeXZ0-zdPQg.jpeg" /></figure><p>In modern software development, effective testing plays a key role in ensuring the reliability and stability of applications.</p><p>This article offers practical recommendations for writing integration tests, demonstrating how to focus on the specifications of interactions with external services, making the tests more readable and easier to maintain. The approach not only enhances the efficiency of testing but also promotes a better understanding of the integration processes within the application. Through the lens of specific examples, various strategies and tools — such as DSL wrappers, JsonAssert, and Pact — will be explored, offering the reader a comprehensive guide to improving the quality and visibility of integration tests.</p><p>The article presents examples of integration tests performed using the Spock Framework in Groovy for testing HTTP interactions in Spring applications. At the same time, the main techniques and approaches suggested in it can be effectively applied to various types of interactions beyond HTTP.</p><h4>Problem Description</h4><p>The article <a href="https://medium.com/@avvero.abernathy/ordering-chaos-arranging-http-request-testing-in-spring-c625520d2418">Ordering Chaos: Arranging HTTP Request Testing in Spring</a> describes an approach to writing tests with a clear separation into distinct stages, each performing its specific role. Let’s describe a test example according to these recommendations, but with mocking not one, but two requests. The Act stage (Execution) will be omitted for brevity (a full test example can be found in the <a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/groovy/pw/avvero/spring/sandbox/bot/mock/FeatureGTests.groovy">project repository</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZEGgrw8ubwph71Po.jpg" /></figure><p>The presented code is conditionally divided into parts: “Supporting Code” (colored in gray) and “Specification of External Interactions” (colored in blue). The Supporting Code includes mechanisms and utilities for testing, including intercepting requests and emulating responses. The Specification of External Interactions describes specific data about external services that the system should interact with during the test, including expected requests and responses. The Supporting Code lays the foundation for testing, while the Specification directly relates to the business logic and main functions of the system that we are trying to test.</p><p>The Specification occupies a minor part of the code but represents significant value for understanding the test, whereas the Supporting Code, occupying a larger part, presents less value and is repetitive for each mock declaration. The code is intended for use with MockRestServiceServer. Referring to the <a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/groovy/pw/avvero/spring/sandbox/bot/wiremock/FeatureWiremockGTests.groovy">example on WireMock</a>, one can see the same pattern: the specification is almost identical, and the Supporting Code varies.</p><p>The aim of this article is to offer practical recommendations for writing tests in such a way that the focus is on the specification, and the Supporting Code takes a back seat.</p><h4>Demonstration Scenario</h4><p>For our test scenario, i propose a hypothetical Telegram bot that forwards requests to the OpenAI API and sends responses back to users.</p><p>The contracts for interacting with services are described in a simplified manner to highlight the main logic of the operation. Below is a sequence diagram demonstrating the application architecture. I understand that the design might raise questions from a systems architecture perspective, but please approach this with understanding — the main goal here is to demonstrate an approach to enhancing visibility in tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/0*EHq6WCyY0b-RQGOn.png" /></figure><h4>Proposal</h4><p>This article discusses the following practical recommendations for writing tests:</p><p>- Use of DSL wrappers for working with mocks.</p><p>- Use of JsonAssert for result verification.</p><p>- Storing the specifications of external interactions in JSON files.</p><p>- Use of Pact files.</p><h4>Using DSL Wrappers for Mocking</h4><p>Using a DSL wrapper allows hiding the boilerplate mock code and provides a simple interface for working with the specification. It’s important to emphasize that what’s proposed is not a specific DSL, but a general approach it implements. A corrected test example using DSL is presented below (<a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/groovy/pw/avvero/spring/sandbox/bot/mock/FeatureGTestsStep1.groovy">full test text</a>).</p><pre>setup:<br>def openaiRequestCaptor = restExpectation.openai.completions(withSuccess(&quot;{...}&quot;))<br>def telegramRequestCaptor = restExpectation.telegram.sendMessage(withSuccess(&quot;{}&quot;))<br>when:<br>...<br>then:<br>openaiRequestCaptor.times == 1<br>telegramRequestCaptor.times == 1</pre><p>Where the method <em>restExpectation.openai.completions</em>, for example, is described as follows:</p><pre>public interface OpenaiMock {<br><br>    /**<br>     * This method configures the mock request to the following URL: {@code https://api.openai.com/v1/chat/completions}<br>     */<br>    RequestCaptor completions(DefaultResponseCreator responseCreator);<br>}</pre><p>Having a comment on the method allows, when hovering over the method name in the code editor, to get help, including seeing the URL that will be mocked.</p><p>In the proposed implementation, the declaration of the response from the mock is made using <em>ResponseCreator</em> instances, allowing for custom ones, such as:</p><pre>public static ResponseCreator withResourceAccessException() {<br>    return (request) -&gt; {<br>        throw new ResourceAccessException(&quot;Error&quot;);<br>    };<br>}</pre><p>An example test for unsuccessful scenarios specifying a set of responses is shown below:</p><pre>import static org.springframework.http.HttpStatus.FORBIDDEN<br><br>setup:<br>def openaiRequestCaptor = restExpectation.openai.completions(openaiResponse)<br>def telegramRequestCaptor = restExpectation.telegram.sendMessage(withSuccess(&quot;{}&quot;))<br>when:<br>...<br>then:<br>openaiRequestCaptor.times == 1<br>telegramRequestCaptor.times == 0<br>where:<br>openaiResponse                | _<br>withResourceAccessException() | _<br>withStatus(FORBIDDEN)         | _</pre><p>For WireMock, everything is the same, except the response formation is slightly different (<a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/groovy/pw/avvero/spring/sandbox/bot/wiremock/FeatureWiremockGTestsStep1.groovy">test code</a>, <a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/java/pw/avvero/spring/sandbox/bot/wiremock/CustomMockRestResponseCreators.java">response factory class code</a>).</p><h4>Using the @Language(“JSON”) Annotation for Better IDE Integration</h4><p>When implementing a DSL, it’s possible to annotate method parameters with <em>@Language(“JSON”)</em> to enable language feature support for specific code snippets in IntelliJ IDEA. With JSON, for example, the editor will treat the string parameter as JSON code, enabling features such as syntax highlighting, auto-completion, error checking, navigation, and structure search. Here’s an example of the annotation’s usage:</p><pre>public static DefaultResponseCreator withSuccess(@Language(&quot;JSON&quot;) String body) {<br>    return MockRestResponseCreators.withSuccess(body, APPLICATION_JSON);<br>}</pre><p>Here’s how it looks in the editor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gjCFXici-2cHGCEl.png" /></figure><h4>Using JsonAssert for Result Verification</h4><p>The JSONAssert library is designed to simplify testing of JSON structures. It enables developers to easily compare expected and actual JSON strings with a high degree of flexibility, supporting various comparison modes.</p><p>This allows moving from a verification description like this</p><pre>openaiRequestCaptor.body.model == &quot;gpt-3.5-turbo&quot;<br>openaiRequestCaptor.body.messages.size() == 1<br>openaiRequestCaptor.body.messages[0].role == &quot;user&quot;<br>openaiRequestCaptor.body.messages[0].content == &quot;Hello!&quot;</pre><p>to something like this</p><pre>assertEquals(&quot;&quot;&quot;{<br>    &quot;model&quot;: &quot;gpt-3.5-turbo&quot;,<br>    &quot;messages&quot;: [{<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Hello!&quot;<br>    }]<br>}&quot;&quot;&quot;, openaiRequestCaptor.bodyString, false)</pre><p>In my opinion, the main advantage of the second approach is that it ensures data representation consistency across various contexts — in documentation, logs, tests. This significantly simplifies the testing process, providing flexibility in comparison and accuracy in error diagnosis. Thus, we not only save time on writing and maintaining tests but also improve their readability and informativeness.</p><p>When working within Spring Boot, starting from at least version 2, no additional dependencies are needed to work with the library, as <em>org.springframework.boot:spring-boot-starter-test</em> already includes a dependency on <em>org.skyscreamer:jsonassert</em>.</p><h4>Storing the Specification of External Interactions in JSON Files</h4><p>One observation we can make is that JSON strings take up a significant portion of the test. Should they be hidden? Yes and no. It’s important to understand what brings more benefits. Hiding them makes tests more compact and simplifies grasping the essence of the test at first glance. On the other hand, for thorough analysis, part of the crucial information about the specification of external interaction will be hidden, requiring extra jumps across files. The decision depends on convenience: do what’s more comfortable for you.</p><p>If you choose to store JSON strings in files, one simple option is to keep responses and requests separately in JSON files. Below is a test code (<a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/groovy/pw/avvero/spring/sandbox/bot/mock/FeatureGTestsStep2.groovy">full version</a>) demonstrating an implementation option:</p><pre>setup:<br>def openaiRequestCaptor = restExpectation.openai.completions(withSuccess(fromFile(&quot;json/openai/response.json&quot;)))<br>def telegramRequestCaptor = restExpectation.telegram.sendMessage(withSuccess(&quot;{}&quot;))<br>when:<br>...<br>then:<br>openaiRequestCaptor.times == 1<br>telegramRequestCaptor.times == 1</pre><p>The <em>fromFile</em> method simply reads a string from a file in the <em>src/test/resources</em> directory and doesn’t carry any revolutionary idea but is still available in the project repository for reference.</p><p>For the variable part of the string, it’s suggested to use substitution with <a href="https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/StringSubstitutor.html">org.apache.commons.text.StringSubstitutor</a> and pass a set of values when describing the mock, for example:</p><pre>setup:<br>def openaiRequestCaptor = restExpectation.openai.completions(withSuccess(fromFile(&quot;json/openai/response.json&quot;,<br>        [content: &quot;Hello! How can I assist you today?&quot;])))</pre><p>where the part with substitution in the JSON file looks like this:</p><pre>...<br>&quot;message&quot;: {<br>    &quot;role&quot;: &quot;assistant&quot;,<br>    &quot;content&quot;: &quot;${content:-Hello there, how may I assist you today?}&quot;<br>},<br>...</pre><p>The single challenge for developers when adopting the file storage approach is to develop a proper file placement scheme in test resources and a naming scheme. It’s easy to make a mistake that can worsen the experience of working with these files. One solution to this problem could be using specifications, such as those from Pact, which will be discussed later.</p><p>When using the described approach in tests written in Groovy, you might encounter inconvenience: there’s no support in IntelliJ IDEA for navigating to the file from the code, but <a href="https://youtrack.jetbrains.com/issue/IDEA-349059">support for this functionality is expected to be added in the future</a>. In tests written in Java, this works great.</p><h4>Using Pact Contract Files</h4><p>Let’s start with the terminology.</p><p>Contract testing is a method of testing integration points where each application is tested in isolation to confirm that the messages it sends or receives conform to a mutual understanding documented in a “contract”. This approach ensures that interactions between different parts of the system meet expectations.</p><p>A contract in the context of contract testing is a document or specification that records an agreement on the format and structure of messages (requests and responses) exchanged between applications. It serves as a basis for verifying that each application can correctly process data sent and received by others in the integration.</p><p>The contract is established between a consumer (for example, a client wanting to retrieve some data) and a provider (for example, an API on a server providing the data needed by the client).</p><p>Consumer-driven testing is an approach to contract testing where consumers generate contracts during their automated test runs. These contracts are passed to the provider, who then runs their set of automated tests. Each request contained in the contract file is sent to the provider, and the response received is compared with the expected response specified in the contract file. If both responses match, it means the consumer and service provider are compatible.</p><p>Finally, Pact. Pact is a tool that implements the ideas of consumer-driven contract testing. It supports testing both HTTP integrations and message-based integrations, focusing on code-first test development.</p><p>As I mentioned earlier, we can use Pact’s contract specification and tools for our task. The implementation might look like this (<a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/groovy/pw/avvero/spring/sandbox/bot/mock/FeatureGTestsStep3.groovy">full test code</a>):</p><pre>setup:<br>def openaiRequestCaptor = restExpectation.openai.completions(fromContract(&quot;openai/SuccessfulCompletion-Hello.json&quot;))<br>def telegramRequestCaptor = restExpectation.telegram.sendMessage(withSuccess(&quot;{}&quot;))<br>when:<br>...<br>then:<br>openaiRequestCaptor.times == 1<br>telegramRequestCaptor.times == 1</pre><p>The contract file is <a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/resources/contracts/openai/SuccessfulCompletion-Hello.json">available for review</a>.</p><p>The advantage of using contract files is that they contain not only the request and response body but also other elements of the external interactions specification — request path, headers, HTTP response status, allowing a mock to be fully described based on such a contract.</p><p>It’s important to note that in this case, we limit ourselves to contract testing and do not extend into consumer-driven testing. However, someone might want to explore Pact further.</p><h4>Conclusion</h4><p>This article reviewed practical recommendations for enhancing the visibility and efficiency of integration tests in the context of development with the Spring Framework. My goal was to focus on the importance of clearly defining the specifications of external interactions and minimizing boilerplate code. To achieve this goal, I suggested using DSL wrappers, JsonAssert, storing specifications in JSON files, and working with contracts through Pact. The approaches described in the article aim to simplify the process of writing and maintaining tests, improve their readability, and most importantly, enhance the quality of testing itself by accurately reflecting interactions between system components.</p><p>Link to the project repository demonstrating the tests — <a href="https://github.com/avvero/spring-sandbox/tree/main/src/test/groovy/pw/avvero/spring/sandbox/bot">sandbox/bot</a>.</p><p>Thank you for your attention to the article, and good luck in your pursuit of writing effective and visible tests!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24ce9493aae3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Isolation in Testing with Kafka]]></title>
            <link>https://medium.com/@avvero.abernathy/isolation-in-testing-with-kafka-16e00f5d5d7e?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/16e00f5d5d7e</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[spock]]></category>
            <category><![CDATA[spring-testing]]></category>
            <category><![CDATA[spring]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Thu, 29 Feb 2024 06:05:42 GMT</pubDate>
            <atom:updated>2024-06-11T07:33:18.452Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lZZP7POSmHa3o_kPPBULEQ.jpeg" /></figure><p>The experience of running Kafka in test scenarios has reached a high level of convenience thanks to the use of Testcontainers and enhanced support in Spring Boot 3.1 with the <em>@ServiceConnection</em> annotation. However, writing and maintaining integration tests with Kafka remains a challenge. This article describes an approach that significantly simplifies the testing process by ensuring test isolation and providing a set of tools to achieve this goal. With successful implementation of isolation, Kafka tests can be organized in such a way that at the stage of result verification, there is full access to all messages that have arisen during the test, thereby avoiding the need for forced waiting methods such as <em>Thread.sleep()</em>.</p><p>This method is suitable for use with Testcontainers, Embedded Kafka, or other methods of running the Kafka service (e.g., a local instance).</p><h4>Test Isolation</h4><p>As detailed in the article <a href="https://martinfowler.com/articles/nonDeterminism.html">Eradicating Non-Determinism in Tests</a>, clear control over the testing environment is crucial for the reliable execution of tests. This ensures that each test begins from a known state. An example could be a situation where one test creates data in a database and fails to clean it up afterwards, negatively impacting the execution of subsequent tests that expect a different database state.</p><p>Various methods can be applied to achieve test isolation, including:</p><ul><li>Restoring the system’s original state before each test run.</li><li>Mandatory data cleanup after the execution of each test to prevent influence on subsequent tests.</li></ul><h4>Isolation in Kafka Testing</h4><p>Restoring Kafka’s initial environment state from scratch for each test scenario can be achieved by restarting Kafka. This option is straightforward in terms of implementation but costly in terms of startup time. There are methods to speed up this process (for more details, you can read about <a href="https://medium.com/@avvero.abernathy/how-to-reduce-memory-consumption-in-integration-tests-with-kafka-using-graalvm-de2393f7fe8a">running Kafka on GraalVM</a>), but this article suggests considering an option where we work with a common Kafka instance across all test scenarios.</p><p>This approach poses certain challenges: if one test sends messages to a topic and neglects the fact of their receipt and processing, it could harm the execution of another test that may rely on a different topic state.</p><p>To ensure isolation, it is necessary that in the case of sending a message to a topic being listened to by the same application, all processes initiated by these messages are completed before the end of the test scenario.</p><h4>Implementation</h4><p>Consider the example of a hypothetical Telegram bot that redirects requests to the OpenAI API and sends responses to users.</p><p>For simplicity, the interaction contracts with the services are described in a simplified form to highlight the main logic of operation. Below is a sequence diagram showing the architecture of the application. I understand that the design may raise questions from a system architecture perspective, but please bear with me — the main goal here is to demonstrate the approach to isolation in testing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cUhIsosTDxAu1XR1cBM6lQ.png" /></figure><p>Below is a diagram illustrating the testing approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CXq-vTiG6ZSB_583.png" /></figure><p>The key feature of the proposed solution is the strict division of test code into phases corresponding to the Arrange-Act-Assert pattern. You can read more about this approach in the article <a href="https://medium.com/@avvero.abernathy/ordering-chaos-arranging-http-request-testing-in-spring-c625520d2418">Ordering Chaos: Arranging HTTP Request Testing in Spring</a>.</p><p>To achieve isolation, it is critically important to maintain the following relationships between the key elements of the scheme (numbers correspond to those indicated in the diagram):</p><p>- (1) Scenario setup (Arrange) occurs before Scenario execution (Act).</p><p>- (2) The message is sent and (3) confirmed by the partition leader before the ServiceA’s request processing is considered complete.</p><p>- (4) Manual offset management with commit occurs only after all processing by ServiceB or ServiceA is fully completed.</p><p>- (5) Scenario execution (Act) occurs before Result verification (Assert).</p><p>This approach ensures that by the time of result verification, all processes within the test scenario are completed, and all messages are sent and received, ensuring the test environment is in a known and final state.</p><h4>Scenario Setup (Arrange) Occurs Before Scenario Execution (Act)</h4><p>The goal of this stage is to prepare everything necessary for the test scenario. In the context of our example, the main components of the test environment include the application context, HTTP request mocks, Kafka, and Record Captor.</p><p>Regarding integration with Kafka, it is critical to ensure that all consumers are ready to receive messages. This verification is implemented in the <a href="https://github.com/avvero/kafka-test-support/blob/8e9bdbfebe865f3de48972ab9a7f33cb34326348/kafka-support/src/main/java/pw/avvero/test/kafka/KafkaSupport.java#L39">KafkaSupport#waitForPartitionAssignment</a> method. The solution is based on the original <em>ContainerTestUtils</em> from the <em>org.springframework.kafka:spring-kafka-test</em> library, modified according to the described usage scenario. This method guarantees that at least one partition will be assigned to each Kafka consumer. This implies a limitation: there must be only one partition per topic in the test environment, although this limitation is a result of the current implementation of the method and can be changed.</p><p>Using a common Kafka instance requires setting the <em>auto.offset.reset = latest</em> parameter for consumers. For Spring applications, this is done as follows:</p><pre>spring.kafka.consumer.auto-offset-reset=latest</pre><p>Record Captor is a key element of this solution. Its task is to “catch” messages from the specified topics in the configuration and provide access to them for the result verification step of the test scenario. Technically, it is a simple consumer for a Kafka topic with a message storage mechanism and an access interface. The Record Captor code is available in the <a href="https://github.com/avvero/kafka-test-support/blob/sb3/kafka-support/src/main/java/pw/avvero/test/kafka/RecordCaptor.java">project repository</a>.</p><p>The current implementation of Record Captor offers the use of the message key for identifying messages related to a specific test case. This is useful in systems where unique identifiers are present, such as a client ID or a process identifier in the domain model. Using such identifiers as the key allows for effectively grouping and tracking all messages related to the same test scenario, even if they are distributed across different topics or processed by different system components.</p><h4>Synchronous Message Sending with Acknowledgment</h4><p>The goal is to implement synchronous message sending to Kafka with acknowledgment from the partition leader. To achieve this, the <em>acks = 1 </em>parameter must be set for the producer. In the context of a Spring application, this setting is specified as follows:</p><pre>spring.kafka.producer.acks=1</pre><p>When using <em>KafkaTemplate</em> for sending messages, it is important to ensure the synchronicity of sending, as this component provides only an asynchronous interface <em>org.springframework.kafka.core.KafkaTemplate#send(org.springframework.messaging.Message&lt;?&gt;) return CompletableFuture&lt;SendResult&lt;K, V&gt;&gt;</em>. The following approach can be used for synchronous sending:</p><pre>kafkaTemplate.send(message).get()</pre><p>This ensures that the message sending will be completed synchronously, with waiting for Kafka’s acknowledgment before proceeding with the program execution.</p><h4>Manual Offset Management</h4><p>Manual offset management with commit means that the message consumer will commit the message processing only after their complete processing. In this context, the offset for <em>topicA</em> will only be committed after the message has been successfully sent to <em>topicB</em> and the corresponding acknowledgment has been received.</p><p>To implement this logic, it is necessary to disable the automatic offset commit for consumers by setting the <em>enable.auto.commit = false</em> parameter. In the context of a Spring application, this is configured as follows:</p><pre>spring.kafka.consumer.enable-auto-commit=false</pre><p>Furthermore, the commit mechanism should be configured so that the offset is committed after processing each individual message, which is achieved by setting the parameter:</p><pre>spring.kafka.listener.ack-mode=record</pre><h4>Scenario Execution (Act) Occurs Before Result Verification (Assert)</h4><p>Before starting the result verification stage, it is necessary to ensure that all processes related to the scenario are completed, all messages are sent and received, ensuring the test environment transitions to a “known” final state. Thanks to the preceding stages, we have ensured adherence to the happens-before principle between the actions of producers and consumers, as well as between all processing within the application. At this stage, it is required to perform a check for offset commits for each partition for each group of consumers.</p><p>To carry out this verification, the solution presented in the method <a href="https://github.com/avvero/kafka-test-support/blob/8e9bdbfebe865f3de48972ab9a7f33cb34326348/kafka-support/src/main/java/pw/avvero/test/kafka/KafkaSupport.java#L105">KafkaSupport#waitForPartitionOffsetCommit</a> can be used.</p><h4>Result Verification (Assert)</h4><p>The final stage involves analyzing the results of the test scenario. This process includes checking the state of mocks and analyzing the messages caught in RecordCaptor.</p><h4>Key Solution Elements Summarized</h4><p>Here is a brief summary of the key components of the proposed solution for effective testing with Kafka:</p><p>1. One partition per topic.</p><p>2. The policy for message reading start for consumers — <em>spring.kafka.consumer.auto-offset-reset=latest</em>.</p><p>3. Acknowledgment policy for producers — <em>spring.kafka.producer.acks=1</em>.</p><p>4. Synchronous message sending — <em>kafkaTemplate.send(message).get()</em>.</p><p>5. Manual offset control — <em>spring.kafka.consumer.enable-auto-commit=false</em>, <em>spring.kafka.listener.ack-mode=record</em>.</p><p>6. Waiting for partition assignment before starting the test scenario — <em>KafkaSupport#waitForPartitionAssignment</em>.</p><p>7. Waiting for offset commitment before verifying test results — <em>KafkaSupport#waitForPartitionOffsetCommit</em>.</p><h4>Result</h4><p>The application code is available in the <a href="https://github.com/avvero/kafka-test-support/tree/sb3/example-testcontainers/src/main/java/pw/avvero/example/feature1">examples module</a>. The test code is as follows.</p><pre>def &quot;User Message Processing with OpenAI&quot;() {<br>    setup:<br>(1) KafkaSupport.waitForPartitionAssignment(applicationContext)                           <br>    and:<br>    def openaiRequestCaptor = new RequestCaptor()<br>(2) restMock.expect(manyTimes(), requestTo(&quot;https://api.openai.com/v1/chat/completions&quot;)) <br>            .andExpect(method(HttpMethod.POST))<br>            .andExpect(openaiRequestCaptor)<br>            .andRespond(withSuccess(&#39;{&quot;content&quot;: &quot;Hi, how can i help you?&quot;}&#39;, MediaType.APPLICATION_JSON))<br>    and:<br>    def telegramRequestCaptor = new RequestCaptor()<br>(3) restMock.expect(manyTimes(), requestTo(&quot;https://api.telegram.org/sendMessage&quot;))      <br>            .andExpect(method(HttpMethod.POST))<br>            .andExpect(telegramRequestCaptor)<br>            .andRespond(withSuccess(&#39;{}&#39;, MediaType.APPLICATION_JSON))<br>    when:<br>(4) mockMvc.perform(post(&quot;/telegram/webhook&quot;)                                           <br>            .contentType(APPLICATION_JSON_VALUE)<br>            .content(&quot;&quot;&quot;{<br>              &quot;message&quot;: {<br>                &quot;from&quot;: {<br>                  &quot;id&quot;: 10000000<br>                },<br>                &quot;chat&quot;: {<br>                  &quot;id&quot;: 20000000<br>                },<br>                &quot;text&quot;: &quot;Hello!&quot;<br>              }<br>            }&quot;&quot;&quot;.toString())<br>            .accept(APPLICATION_JSON_VALUE))<br>            .andExpect(status().isOk())<br>(5) KafkaSupport.waitForPartitionOffsetCommit(applicationContext)                     <br>    then:<br>(6) openaiRequestCaptor.times == 1                                                      <br>    JSONAssert.assertEquals(&quot;&quot;&quot;{<br>        &quot;content&quot;: &quot;Hello!&quot;<br>    }&quot;&quot;&quot;, openaiRequestCaptor.bodyString, false)<br>    and:<br>    telegramRequestCaptor.times == 1<br>    JSONAssert.assertEquals(&quot;&quot;&quot;{<br>        &quot;chatId&quot;: &quot;20000000&quot;,<br>        &quot;text&quot;: &quot;Hi, how can i help you?&quot;<br>    }&quot;&quot;&quot;, telegramRequestCaptor.bodyString, false)<br>}</pre><p>Here are the key steps described in the test scenario:</p><p>1. Waiting for partition assignment before starting the test scenario.</p><p>2. Mocking requests to OpenAI.</p><p>3. Mocking requests to Telegram.</p><p>4. Executing the test scenario.</p><p>5. Waiting for offset confirmation.</p><p>6. Result verification.</p><p>Additional tests, including scenarios with intensive message sending and the use of the `@RetryableTopic` mechanism for retries, are also available in the <a href="https://github.com/avvero/kafka-test-support/tree/sb3/kafka-support/src/test/groovy/pw/avvero/test/kafka">project repository</a>, providing opportunities for study and adaptation to your own development needs.</p><h4>Conclusion</h4><p>Successful testing of interactions with Kafka requires a careful approach to test isolation and environment control. The use of Testcontainers and the capabilities of Spring Boot 3.1 greatly simplifies this process, and the application of the proposed techniques and tools allows developers to focus on the application logic, making development more efficient and less prone to errors.</p><p>Thank you for paying attention to the article, and good luck in your pursuit of writing convenient and reliable tests!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16e00f5d5d7e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Reduce Memory Consumption in Integration Tests with Kafka Using GraalVM]]></title>
            <link>https://medium.com/@avvero.abernathy/how-to-reduce-memory-consumption-in-integration-tests-with-kafka-using-graalvm-de2393f7fe8a?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/de2393f7fe8a</guid>
            <category><![CDATA[spring]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[testcontainer]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Mon, 29 Jan 2024 04:50:56 GMT</pubDate>
            <atom:updated>2024-06-11T07:32:53.224Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrXga0_U6OsexlN37pHIpQ.jpeg" /></figure><p>In this article, I want to share my experience with creating a native image for EmbeddedKafka using GraalVM. Utilizing this image in integration tests not only speeds up the execution of test scenarios but also reduces memory consumption. Interestingly, when compared to using <strong>confluentinc/cp-kafka</strong> in Testcontainers, there is a noticeable difference in both speed and memory usage — and it’s not in favor of the latter.</p><h4>EmbeddedKafka, Testcontainers, GraalVM</h4><p>A brief overview of the key components used in the project:</p><p>EmbeddedKafka is a tool that allows embedding a Kafka server directly into a JVM application or testing environment. This is useful for integration testing of applications that use Apache Kafka for data stream processing or as a messaging system. Embedded Kafka is primarily used for isolated testing of interactions with Kafka, simplifying setup and management of tests due to its quick startup and shutdown. This ensures test reproducibility in different environments and provides control over the Kafka configuration. However, running Embedded Kafka in a JVM application increases memory usage due to Kafka’s resource-intensive nature and the need for data storage. This is a trade-off between development convenience and additional memory load, making an external Kafka server (relative to the JVM application) more preferable for production environments or in cases of limited memory resources.</p><p>Testcontainers is a framework used to support automated integration tests using Docker containers. It allows the creation, management, and deletion of containers during test execution. Using Docker containers ensures consistency of the testing environment across different machines and platforms, simplifying local development and testing. Using Kafka in Testcontainers offers several advantages, especially when it comes to testing in a more realistic and flexible environment. Testcontainers runs Kafka instances based on the official Confluent OSS Platform images.</p><p>GraalVM is a platform for running programs, supporting various programming languages and technologies. Among other things, it allows compiling Java applications into statically linked executable files (native binaries). These native executable files run faster, require less memory, and do not require an installed JVM.</p><h4>Test Scenario</h4><p>To illustrate the approaches to writing tests, I have prepared code examples corresponding to a simple test scenario:</p><p>- Send a message <strong>value1</strong> to the topic <strong>topic1</strong>.</p><p>- Read the message from the topic <strong>topic1</strong>.</p><p>- Verify that the value is equal to <strong>value1</strong>.</p><p>Examples can be found in the <a href="https://github.com/avvero/embedded-kafka">project repository</a>:</p><p>- Tests using EmbeddedKafka are located in the <a href="https://github.com/avvero/embedded-kafka/tree/sb3/example-spring-embedded-kafka">example-spring-embedded-kafka</a> module.</p><p>- Tests using Testcontainers are in the <a href="https://github.com/avvero/embedded-kafka/tree/sb3/example-testcontainers">example-testcontainers</a> module.</p><p>The repository structure facilitates the comparison of modules to evaluate the differences in structure and composition of code when using each approach.</p><h4>Wrapping EmbeddedKafka in a Container</h4><p>The first task was to implement the launch of EmbeddedKafka within a separate container. To do this, I took the following straightforward steps:</p><p>- Created a standard Spring Boot application using <a href="https://start.spring.io/">https://start.spring.io/</a>;</p><p>- In the application, I set up the launch of an instance of the <strong>org.springframework.kafka.test.EmbeddedKafkaZKBroker </strong>class with the necessary parameters;</p><p>- Described a Dockerfile and built the image.</p><p>All the above actions are reflected in the code of the <a href="https://github.com/avvero/embedded-kafka/tree/sb3/emk-application">emk-application</a> module in the project repository.</p><h4>Launching EmbeddedKafka in a Container</h4><p>The Testcontainers documentation provides a guide to launching a Kafka container using the <strong>KafkaContainer </strong>class as follows:</p><pre>KafkaContainer kafka = new KafkaContainer(DockerImageName.parse(&quot;confluentinc/cp-kafka:6.2.1&quot;))</pre><p>However, this class was not suitable for my needs, as it is intended for use with <strong>confluentinc/cp-kafka</strong> compatible images. Nevertheless, examining it was beneficial as it revealed an interesting logic around the handling of the KAFKA_ADVERTISED_LISTENERS parameter:</p><p>1. At the start of the container, a substitution of ENTRYPOINT/COMMAND instructions is performed.</p><p>2. After starting, the KAFKA_ADVERTISED_LISTENERS parameters are passed to the container along with instructions for starting Kafka.</p><p>This process is detailed in the accompanying diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/0*F7wp50Tmk1zaETQA.png" /></figure><p>Why is this necessary? During operation, a client may connect to any Kafka node to obtain the address for performing read/write operations, even if Kafka is represented by a single node. An external address is required for an external user, and similarly, an internal address for an internal user. By specifying KAFKA_ADVERTISED_LISTENERS, we provide the broker with information about its external address, which the broker then communicates to the client. The clients will be external relative to the broker since the broker is running in a container.</p><p>I implemented the logic described above in a new class — <a href="https://github.com/avvero/embedded-kafka/blob/sb3/emk-testcontainers/src/main/java/pw/avvero/emk/EmbeddedKafkaContainer.java">EmbeddedKafkaContainer.java</a>.</p><h4>Creating a Native Image for EmbeddedKafka</h4><p>The simplest way to start a new Spring Boot project for GraalVM is to visit the site <a href="https://start.spring.io/">start.spring.io</a>, add the “GraalVM Native Support” dependency, and generate the project. The project comes with a HELP.md file, which provides useful tips for getting started.</p><h4>Metadata Collection</h4><p>The tool for building a native image relies on static analysis available during the execution of the application code. However, this analysis is not always capable of fully predicting all use cases of Java Native Interface (JNI), Java reflection, dynamic proxy objects, etc. Therefore, these use cases of dynamic functions need to be explicitly specified to the native image build tool in the form of metadata. One way to provide such metadata is through JSON files placed in the project directory <em>META-INF/native-image/&lt;group.id&gt;/&lt;artifact.id&gt;</em>.</p><p>GraalVM offers a Tracing Agent for convenient collection of metadata and preparation of configuration files. This agent tracks all instances of dynamic function usage during the application’s execution on a standard Java VM.</p><p>My approach was as follows:</p><p>- I launched an instance of a Spring application with Embedded Kafka under the JVM with the Tracing Agent.</p><p>- I ran a large set of tests from one of my projects, using the launched application as the main Kafka broker.</p><p>The files generated during this process were placed in the project directory <a href="https://github.com/avvero/embedded-kafka/tree/sb3/emk-application/src/main/resources/META-INF/native-image">META-INF/native-image</a>.</p><h4>Launch and Usage</h4><p>To demonstrate the results, I prepared the following artifacts:</p><p>- A library with the <strong>EmbeddedKafkaContainer </strong>class — <strong>pw.avvero:emk-testcontainers:1.0.0</strong>.</p><p>- Docker images: <strong>avvero/emk</strong> (JVM) and <strong>avvero/emk-native</strong> (native, platform=linux/arm64).</p><p>- An example usage corresponding to the test scenario can be found in the <a href="https://github.com/avvero/embedded-kafka/tree/sb3/example-embedded-kafka-container">example-embedded-kafka-container</a> module.</p><p>The <a href="https://github.com/avvero/embedded-kafka/blob/sb3/example-embedded-kafka-container/src/test/java/pw/avvero/emk/KafkaContainerConfiguration.java">KafkaContainerConfiguration</a> is configured as follows:</p><pre>@TestConfiguration(proxyBeanMethods = false)<br>public class KafkaContainerConfiguration {<br><br>    @Bean<br>    @RestartScope<br>    @ServiceConnection<br>    EmbeddedKafkaContainer kafkaContainer() {<br>        return new EmbeddedKafkaContainer(&quot;avvero/emk-native:1.0.0&quot;);<br>    }<br>}</pre><p>To assess memory utilization, I ran tests for about 7 minutes from one of my projects. Based on observations in `docker stats`, I noticed the following trends in memory consumption:</p><pre>- confluentinc/cp-kafka:7.3.3    1.331GiB<br>- avvero/emk                     677.3MiB<br>- avvero/emk-native              126.4MiB</pre><p>Memory analysis (Young + Old + Meta space) through GC logs using <a href="https://gceasy.io/">https://gceasy.io</a> showed the following:</p><pre>- confluentinc/cp-kafka:7.3.3     1.06 gb/866.92 mb (Allocated/Peak)<br>- avvero/emk                      567.62 mb/241.74 mb (Allocated/Peak)<br>- avvero/emk-native               20.00M -&gt; 15.50M ?</pre><p>Analyzing GC logs for the native image presents a more complex task since the format and composition of data differ from “standard” GC logs. Unfortunately, I could not find an analysis tool suitable for this purpose that could provide ready analytics. Therefore, below is a fragment of the log, which helps to estimate the general order of memory utilization in my case.</p><pre>[497.519s] GC(11371) Collect on allocation<br>[497.520s] GC(11371)   Eden: 4.50M-&gt;0.00M<br>[497.520s] GC(11371)   Survivor: 0.00M-&gt;0.00M<br>[497.520s] GC(11371)   Old: 15.50M-&gt;15.50M<br>[497.520s] GC(11371)   Free: 3.50M-&gt;8.00M<br>[497.520s] GC(11371) Incremental GC (Collect on allocation) 20.00M-&gt;15.50M 0.896ms</pre><p>GC log files are attached to the <a href="https://github.com/avvero/embedded-kafka/tree/sb3/benchmark">performance test module</a>.</p><p>Regarding startup time, I conducted a series of <a href="https://github.com/avvero/embedded-kafka/tree/sb3/benchmark">performance tests using JMH</a> to assess the launch time and operational readiness of different Kafka container configurations:</p><p>- <strong>testContainersKafkaStartAndReady </strong>— Testcontainers with confluentinc/cp-kafka:7.3.3.</p><p>- <strong>emkJvmKafkaStartAndReady</strong> — avvero/emk(JVM).</p><p>- <strong>emkNativeKafkaStartAndReady </strong>— avvero/emk-native (native, platform=linux/arm64).</p><p>The tests focus on verifying both startup and readiness. Simply launching a Kafka container does not always mean it is ready for operations. The readiness check simulates a real-world scenario in which Kafka is not only launched but also fully operationally ready. This provides a more comprehensive view of the time required for Kafka to be fully operational in various containerized environments.</p><p>The performance test results are as follows:</p><pre>Benchmark                                                 Mode  Cnt  Score   Error  Units<br>TestContainersBenchmark.testContainersKafkaStartAndReady    ss   10  3,091 ± 0,354   s/op<br>TestContainersBenchmark.emkJvmKafkaStartAndReady            ss   10  2,659 ± 0,708   s/op<br>TestContainersBenchmark.emkNativeKafkaStartAndReady         ss   10  0,521 ± 0,055   s/op</pre><p>The <strong>avvero/emk-native:1.0.0</strong> container demonstrates higher performance, showing an average startup and readiness check time of just 0.521 seconds with a deviation of ±0,055.</p><h4>Conclusion</h4><p>The use of a native image for EmbeddedKafka with GraalVM in integration tests accelerates test startup time and reduces memory consumption, making it an efficient solution compared to traditional methods, such as using <strong>confluentinc/cp-kafka</strong> in Testcontainers.</p><p>The use of GraalVM opens up new opportunities for developers aiming to improve the performance and efficiency of integration tests. This approach can be adapted and expanded for other similar tasks and technologies, underscoring its versatility and potential in the field of software development.</p><p>Link to the project repository with test demonstrations — <a href="https://github.com/avvero/embedded-kafka">https://github.com/avvero/embedded-kafka</a>.</p><p>Thank you for your attention to the article, and good luck in your endeavor to write effective and fast tests!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de2393f7fe8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ordering Chaos: Arranging HTTP Request Testing in Spring]]></title>
            <link>https://medium.com/@avvero.abernathy/ordering-chaos-arranging-http-request-testing-in-spring-c625520d2418?source=rss-6d700debbbbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c625520d2418</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[spock-framework]]></category>
            <category><![CDATA[spring]]></category>
            <dc:creator><![CDATA[Anton Belyaev]]></dc:creator>
            <pubDate>Wed, 20 Dec 2023 13:10:06 GMT</pubDate>
            <atom:updated>2024-06-11T07:31:54.288Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OPuzjZ7HPbZbsOtFGRO5xQ.jpeg" /></figure><p>In this article, I would like to describe an approach to writing tests with a clear division into separate stages, each performing its specific role. This facilitates the creation of tests that are easier to read, understand, and maintain.</p><p>The discussion will focus on using the Arrange-Act-Assert methodology for integration testing in the Spring Framework with mocking of HTTP requests to external resources encountered during the execution of the tested code within the system behavior. The tests under consideration are written using the Spock Framework in the Groovy language. MockRestServiceServer will be used as the mocking mechanism. There will also be a few words about WireMock.</p><h3>Problem Description</h3><p>When studying how to write integration tests for Spring, I often referred to materials on the topic. Examples for MockRestServiceServer mostly described an approach with the declaration of expectations as follows:</p><ul><li>expected URI</li><li>number of requests to the expected URI</li><li>expectations for the structure and content of the request body</li><li>response to the request</li></ul><p>The code looked something like this:</p><pre>@Test<br>public void testWeatherRequest() {<br>    mockServer.expect(once(), requestTo(&quot;https://external-weather-api.com/forecast&quot;))         <br>            .andExpect(method(HttpMethod.POST))<br>            .andExpect(jsonPath(&quot;$.field1&quot;, equalTo(&quot;value1&quot;)))<br>            .andExpect(jsonPath(&quot;$.field2&quot;, equalTo(&quot;value2&quot;)))<br>            .andExpect(jsonPath(&quot;$.field3&quot;, equalTo(&quot;value3&quot;)))<br>            .andRespond(withSuccess(&#39;{&quot;result&quot;: &quot;42&quot;}&#39;, MediaType.APPLICATION_JSON));<br>    weatherService.getForecast(&quot;London&quot;)<br>    mockServer.verify()<br>    assert ..<br>    assert ..<br>}</pre><p>When applying this approach, I encountered a number of difficulties:</p><p>1. Ambiguity in determining the reasons for AssertionError by the log text — the log text is the same for different scenarios:</p><ul><li>the HTTP call code is missing/not executed according to business logic;</li><li>the HTTP call code is executed with an error;</li><li>the HTTP call code is executed correctly, but there is an error in the mock description.</li></ul><p>2. Difficulty in determining the scope of the tested states due to their dispersion throughout the test code. Formally, the result verification is carried out at the end of the test (mockServer.verify()), but the verification assertions regarding the composition and structure of the request are described at the beginning of the test (as part of creating the mock). At the same time, verification assertions not related to the mock were presented at the end of the test.</p><p><strong>Important clarification</strong>: using RequestMatcher for the purpose of isolating mocks within many requests seems like the right solution.</p><h3>Proposed Solution</h3><p>Clear division of test code into separate stages, according to the Arrange-Act-Assert pattern.</p><h4>Arrange-Act-Assert</h4><p>Arrange-Act-Assert is a widely used pattern in writing tests, especially in unit testing. Let’s take a closer look at each of these steps:</p><p>Arrange (Preparation): At this stage, you set up the test environment. This includes initializing objects, creating mocks, setting up necessary data, etc. The goal of this step is to prepare everything needed for the execution of the action being tested.</p><p>Act (Execution): Here you perform the action you want to test. This could be a method call or a series of actions leading to a certain state or result to be tested.</p><p>Assert (Result Verification): At the final stage, you check the results of the action. This includes assertions about the state of objects, returned values, changes in the database, messages sent, etc. The goal of this step is to ensure that the tested action has produced the expected result.</p><h3>Demonstration Scenarios</h3><p>The business logic of the service for which the tests will be provided can be described as follows:</p><pre>given: The weather service provides information that the weather in city A equals B<br>when: We request weather data from the service for city A<br>then: We receive B</pre><p>Sequence Diagram</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/0*3zBj6LDTEOX2xplq.png" /></figure><h3>Example Implementation for MockRestServiceServer Before Proposed Changes</h3><p>Tests for the above scenario will be described using MockRestServiceServer.</p><h4>Difficulty in Determining the Scope of Tested States Due to Their Dispersion Throughout the Test Code</h4><pre>def &quot;Forecast for provided city London is 42&quot;() {<br>    setup:          // (1)<br>    mockServer.expect(once(), requestTo(&quot;https://external-weather-api.com/forecast&quot;)) // (2)<br>            .andExpect(method(HttpMethod.POST))<br>            .andExpect(jsonPath(&#39;$.city&#39;, Matchers.equalTo(&quot;London&quot;)))                // (3)<br>            .andRespond(withSuccess(&#39;{&quot;result&quot;: &quot;42&quot;}&#39;, MediaType.APPLICATION_JSON)); // (4)<br>    when:          // (5)<br>    def forecast = weatherService.getForecast(&quot;London&quot;)<br>    then:          // (6)<br>    forecast == &quot;42&quot;     // (7)<br>    mockServer.verify()  // (8)<br>}</pre><p>1. Setup stage: describing the mock.</p><p>2. Indicating that exactly one call is expected to <a href="https://external-weather-api.com./">https://external-weather-api.com.</a></p><p>3. Specifying expected request parameters.</p><p>4. Describing the response to return.</p><p>5. Execution stage, where the main call to get the weather for the specified city occurs.</p><p>6. Verification stage. Here, mockServer.verify() is also called to check the request (see item 3).</p><p>7. Verification assertion regarding the returned value.</p><p>8. Calling to verify the mock’s state.</p><p>Here we can observe the problem described earlier as “Difficulty in determining the scope of tested states due to their dispersion throughout the test code” — some of the verification assertions are in the then block, some in the setup block.</p><h4>Ambiguity in Determining the Causes of AssertionError</h4><p>To demonstrate the problem, let’s model different error scenarios in the code. Below are the situations and corresponding error logs.</p><ol><li>Scenario: passed an unknown city name: <em>def forecast = weatherService.getForecast(“Unknown”)</em></li></ol><pre>java.lang.AssertionError: No further requests expected: HTTP POST https://external-weather-api.com<br>0 request(s) executed.<br><br> at org.springframework.test.web.client.AbstractRequestExpectationManager.createUnexpectedRequestError(AbstractRequestExpectationManager.java:193)</pre><p>2. Scenario: incorrect URI declaration for the mock, for example <em>mockServer.expect(once(), requestTo(“https://foo.com&quot;))</em></p><pre>java.lang.AssertionError: No further requests expected: HTTP POST https://external-weather-api.com<br>0 request(s) executed.<br><br> at org.springframework.test.web.client.AbstractRequestExpectationManager.createUnexpectedRequestError(AbstractRequestExpectationManager.java:193)</pre><p>3. Scenario: no HTTP calls in the code</p><pre>java.lang.AssertionError: Further request(s) expected leaving 1 unsatisfied expectation(s).<br>0 request(s) executed.</pre><p>The main observation: all errors are similar, and the stack trace is more or less the same.</p><h3>Example Implementation for MockRestServiceServer with Proposed Changes</h3><h4>Ease of Determining the Scope of Tested States Due to Their Dispersion Throughout the Test Code</h4><pre>def &quot;Forecast for provided city London is 42&quot;() {<br>    setup:          // (1)<br>    def requestCaptor = new RequestCaptor()<br>    mockServer.expect(manyTimes(), requestTo(&quot;https://external-weather-api.com&quot;))          // (2)<br>            .andExpect(method(HttpMethod.POST))<br>            .andExpect(requestCaptor)                                                      // (3)<br>            .andRespond(withSuccess(&#39;{&quot;result&quot;: &quot;42&quot;}&#39;, MediaType.APPLICATION_JSON));      // (4)<br>    when:          // (5)<br>    def forecast = weatherService.getForecast(&quot;London&quot;)<br>    then:          // (6)<br>    forecast == &quot;42&quot;<br>    requestCaptor.times == 1              // (7)<br>    requestCaptor.entity.city == &quot;London&quot; // (8)<br>}</pre><p>3 — Data capture object, code described <a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/java/pw/avvero/spring/sandbox/weather/RequestCaptor.java">here</a></p><p>7 — Verification assertion regarding the number of calls to the URI</p><p>8 — Verification assertion regarding the composition of the request to the URI</p><p>In this implementation, we can see that all the verification assertions are in the then block.</p><p>The above scenario (or the author’s laziness) does not allow showing the convenience of checking headers, but such a possibility also exists in the expected form:</p><pre>forecast == &quot;42&quot;<br>requestCaptor.times == 1           <br>requestCaptor.body.city == &quot;London&quot; <br>requestCaptor.headers.get(&quot;Content-Type&quot;) == [&quot;application/json&quot;]</pre><h4>Unambiguity in Identifying the Causes of AssertionError</h4><p>To demonstrate the problem, let’s attempt to model different error scenarios in the code. Below are the situations and corresponding error logs.</p><ol><li>Scenario: An unknown city name was provided <em>def forecast = weatherService.getForecast(“Unknown”)</em></li></ol><pre>requestCaptor.entity.city == &quot;London&quot;<br>|             |      |    |<br>|             |      |    false<br>|             |      |    5 differences (28% similarity)<br>|             |      |    (Unk)n(-)o(w)n<br>|             |      |    (Lo-)n(d)o(-)n<br>|             |      Unknown<br>|             [city:Unknown]<br>&lt;pw.avvero.spring.sandbox.weather.RequestCaptor@6f77917c times=1 bodyString={&quot;city&quot;:&quot;Unknown&quot;} entity=[city:Unknown] headers=[Accept:[application/json, application/*+json], Content-Type:[application/json], Content-Length:[18]]&gt;</pre><p>2. Scenario: Incorrect URI declaration for the mock, for example <em>mockServer.expect(once(), requestTo(“https://foo.com&quot;))</em></p><pre>java.lang.AssertionError: No further requests expected: HTTP POST https://external-weather-api.com<br>0 request(s) executed.</pre><p>3. Scenario: No HTTP calls in the code</p><pre>Condition not satisfied:<br><br>requestCaptor.times == 1<br>|             |     |<br>|             0     false<br>&lt;pw.avvero.spring.sandbox.weather.RequestCaptor@474a63d9 times=0 bodyString=null entity=null headers=[:]&gt;</pre><h3>Using WireMock</h3><p>WireMock provides the ability to describe verifiable expressions in the Assert block.</p><pre>def &quot;Forecast for provided city London is 42&quot;() {<br>    setup:          // (1)<br>    wireMockServer.stubFor(post(urlEqualTo(&quot;/forecast&quot;))                              // (2)<br>            .willReturn(aResponse()                                                   // (4)<br>                    .withBody(&#39;{&quot;result&quot;: &quot;42&quot;}&#39;)<br>                    .withStatus(200)<br>                    .withHeader(&quot;Content-Type&quot;, &quot;application/json&quot;)))<br>    when:          // (5)<br>    def forecast = weatherService.getForecast(&quot;London&quot;)<br>    then:          // (6)<br>    forecast == &quot;42&quot;<br>    wireMockServer.verify(postRequestedFor(urlEqualTo(&quot;/forecast&quot;))<br>            .withRequestBody(matchingJsonPath(&#39;$.city&#39;, equalTo(&quot;London&quot;))))          // (7)<br>}</pre><p>The above approach can also be used here, by describing the <a href="https://github.com/avvero/spring-sandbox/blob/main/src/test/java/pw/avvero/spring/sandbox/weather/WiredRequestCaptor.java">WiredRequestCaptor class</a>.</p><pre>def &quot;Forecast for provided city London is 42&quot;() {<br>    setup:<br>    StubMapping forecastMapping = wireMockServer.stubFor(post(urlEqualTo(&quot;/forecast&quot;))<br>            .willReturn(aResponse()<br>                    .withBody(&#39;{&quot;result&quot;: &quot;42&quot;}&#39;)<br>                    .withStatus(200)<br>                    .withHeader(&quot;Content-Type&quot;, &quot;application/json&quot;)))<br>    def requestCaptor = new WiredRequestCaptor(wireMockServer, forecastMapping)<br>    when:<br>    def forecast = weatherService.getForecast(&quot;London&quot;)<br>    then:<br>    forecast == &quot;42&quot;<br>    requestCaptor.times == 1<br>    requestCaptor.body.city == &quot;London&quot;<br>}</pre><p>This allows us to simplify expressions and enhance the idiomaticity of the code, making the tests more readable and easier to maintain.</p><h3>Conclusion</h3><p>Throughout this article, I have dissected the stages of testing HTTP requests in Spring, using the Arrange-Act-Assert methodology and mocking tools such as MockRestServiceServer and WireMock. The primary goal was to demonstrate how clearly dividing the test into separate stages significantly enhances readability, understanding, and maintainability.</p><p>I highlighted the problems associated with the ambiguity of error determination and the difficulty of defining the scope of tested states, and presented ways to solve them through a more structured approach to test writing. This approach is particularly important in complex integration tests, where every aspect is critical to ensuring accuracy and reliability of the system.</p><p>Furthermore, I showed how the use of tools like RequestCaptor and WiredRequestCaptor simplifies the test-writing process and improves their idiomaticity and readability, thereby facilitating easier support and modification.</p><p>In conclusion, I want to emphasize that the choice of testing approach and corresponding tools should be based on specific tasks and the context of development. The approach to testing HTTP requests in Spring presented in this article is intended to assist developers facing similar challenges.</p><p>Link to the project repository with demonstration tests — <a href="https://github.com/avvero/spring-sandbox/tree/main/src/test/groovy/pw/avvero/spring/sandbox/weather">here</a><a href="https://github.com/avvero/spring-sandbox/tree/main/src/test/groovy/pw/avvero/spring/sandbox/weather.">.</a></p><p>Thank you for your attention to the article, and good luck in your pursuit of writing effective and reliable tests!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c625520d2418" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>