<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amit Pe&#39;er on Medium]]></title>
        <description><![CDATA[Stories by Amit Pe&#39;er on Medium]]></description>
        <link>https://medium.com/@amitpeer05?source=rss-50119598efed------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kMNPfESKWva6NgMZQB39Vw.jpeg</url>
            <title>Stories by Amit Pe&amp;#39;er on Medium</title>
            <link>https://medium.com/@amitpeer05?source=rss-50119598efed------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:30:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amitpeer05/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[“Shall I Compare Thee to a Summer’s Day”: Exploring the Poetry in Code]]></title>
            <link>https://medium.com/wix-engineering/shall-i-compare-thee-to-a-summers-day-exploring-the-poetry-in-code-89a3b7e12b7e?source=rss-50119598efed------2</link>
            <guid isPermaLink="false">https://medium.com/p/89a3b7e12b7e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[poetry]]></category>
            <dc:creator><![CDATA[Amit Pe'er]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 08:38:32 GMT</pubDate>
            <atom:updated>2025-02-25T08:38:32.735Z</atom:updated>
            <content:encoded><![CDATA[<p>In high school, I fell in love with poetry and the works of Shakespeare. Later in life, I discovered Uncle Bob’s “Clean Code” and realized that the skills required in both disciplines are remarkably similar. This article explores the beauty of software development by drawing parallels between poetic expression and the craftsmanship of writing code. Inspired by Shakespeare’s timeless sonnets, we will see how coding can achieve similar artistic brilliance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*WC-TkkLaIywejcRq" /></figure><h3>Reading Code Like Poetry</h3><p>Recent studies show that developers spend more time reading code than writing it, much like how poetry is meant to be read and appreciated by an audience. Just as a poet carefully selects words for their impact on readers, developers should write code with clarity and consideration for those who will read it later.</p><h3>The Elegance of Minimalism</h3><p>Haiku is a Japanese poetic form with a strict structure and minimalistic approach: it captures a moment or feeling with brevity and clarity. Similarly, in coding, simplicity often leads to elegance and efficiency. Uncle Bob’s “Clean Code” suggests writing code that is clear, concise, and easy to understand.</p><p>Example haiku — <em>The Old Pond by Matsuo Bashō</em></p><blockquote>An old silent pond…</blockquote><blockquote>A frog jumps into the pond,</blockquote><blockquote>splash! Silence again.</blockquote><p>Let’s take a piece of code and try to simplify it so it would resemble a haiku:</p><p>Before Simplification:</p><pre>class Inventory:<br>  def get_total_value(self):<br>      total_value = 0<br>      for item in self.items:<br>          total_value += item[&#39;price&#39;]<br>      return total_value</pre><p>After Simplification:</p><pre>class Inventory:<br>   def get_total_value(self):<br>       return sum(item[&#39;price&#39;] for item in self.items)</pre><h3>Structure and Rhythm</h3><p>Shakespeare’s sonnets are famous for their structured form — 14 lines with a specific rhyme scheme. This structured approach provides rhythm that guides the reader through the poem. In software development, similar principles apply. Well-structured code follows a logical flow, making it easier to read, maintain, and extend. Just as every word in a poem is chosen carefully for its impact, the names of variables and functions in code are crucial for clarity and understanding.</p><p>Here’s a bad example of a simple banking system, with suboptimal naming conventions and structure:</p><pre>public class Account {<br>   private String num;<br>   private double bal;<br><br><br>   public Account(String num, double bal) {<br>       this.num = num;<br>       this.bal = bal;<br>   }<br><br><br>   public String getNum() {<br>       return num;<br>   }<br><br><br>   public double getBal() {<br>       return bal;<br>   }<br><br><br>   public void addBal(double amount) {<br>       if (amount &gt; 0) {<br>           bal += amount;<br>           System.out.println(&quot;Added: &quot; + amount);<br>       } else {<br>           System.out.println(&quot;Invalid amount.&quot;);<br>       }<br>   }<br><br><br>   public boolean takeOut(double amount) {<br>       if (amount &gt; 0 &amp;&amp; amount &lt;= bal) {<br>           bal -= amount;<br>           System.out.println(&quot;Taken out: &quot; + amount);<br>           return true;<br>       } else {<br>           System.out.println(&quot;Invalid amount or insufficient funds.&quot;);<br>           return false;<br>       }<br>   }<br>}</pre><p>In this version, the variable names (<em>num, bal</em>) and method names (<em>addBal, takeOut</em>) are not fully descriptive, which can lead to confusion.</p><p>Now, compare this with an improved example, which demonstrates structured and elegant code with a focus on meaningful naming and clean organization:</p><pre>public class BankAccount {<br>   private String accountNumber;<br>   private double balance;<br><br><br>   public BankAccount(String accountNumber, double initialBalance) {<br>       this.accountNumber = accountNumber;<br>       this.balance = initialBalance;<br>   }<br><br><br>   public String getAccountNumber() {<br>       return accountNumber;<br>   }<br><br><br>   public double getBalance() {<br>       return balance;<br>   }<br><br><br>   public void deposit(double amount) {<br>       if (amount &gt; 0) {<br>           balance += amount;<br>           System.out.println(&quot;Deposited: &quot; + amount);<br>       } else {<br>           System.out.println(&quot;Invalid deposit amount.&quot;);<br>       }<br>   }<br><br><br>   public boolean withdraw(double amount) {<br>       if (amount &gt; 0 &amp;&amp; amount &lt;= balance) {<br>           balance -= amount;<br>           System.out.println(&quot;Withdrew: &quot; + amount);<br>           return true;<br>       } else {<br>           System.out.println(&quot;Invalid withdrawal amount or insufficient funds.&quot;);<br>           return false;<br>       }<br>   }<br>}</pre><p>The names of the methods (<em>deposit, withdraw, getBalance, getAccountNumber</em>) and variables (<em>accountNumber, balance</em>) were selected to be self-explanatory, making the code easy to read and understand. The structure of the <em>BankAccount</em> class is clean and logical, guiding the reader through the flow of operations just as a sonnet guides the reader through its verses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*N7NQJfo5_wi5edBt" /></figure><h3>Creativity and Innovation</h3><p>Poetry is not just about following rules; it’s about breaking them creatively. Poets use literary techniques like metaphor, simile, and allegory to express deeper meanings. Similarly, innovative problem-solving in coding often requires thinking outside the box. By applying creative techniques, developers can find elegant solutions to complex problems.</p><p>This can be demonstrated by the Builder design pattern, which allows developers to create complex objects step by step, emphasizing flexibility and clarity in code construction.</p><pre>public class Cake {<br>   private String flavor;<br>   private int layers;<br>   private boolean hasFrosting;<br><br><br>   public Cake(Builder builder) {<br>       this.flavor = builder.flavor;<br>       this.layers = builder.layers;<br>       this.hasFrosting = builder.hasFrosting;<br>   }<br><br><br>   public static class Builder {<br>       private String flavor;<br>       private int layers;<br>       private boolean hasFrosting;<br><br><br>       public Builder(String flavor) {<br>           this.flavor = flavor;<br>       }<br><br><br>       public Builder layers(int layers) {<br>           this.layers = layers;<br>           return this;<br>       }<br><br><br>       public Builder frosting(boolean hasFrosting) {<br>           this.hasFrosting = hasFrosting;<br>           return this;<br>       }<br><br><br>       public Cake build() {<br>           return new Cake(this);<br>       }<br>   }<br>}</pre><p>Now we can elegantly create a cake by using this builder class:</p><pre>Cake chocolateCake = new Cake.Builder(&quot;Chocolate&quot;)<br>                             .layers(3)<br>                             .frosting(true)<br>                             .build();</pre><h3>Abstraction Levels and Metaphors</h3><p>In poetry, abstraction often involves using metaphors to express deeper meanings without explicitly stating them. This technique lets readers use their imagination and understand the poem in their own way. In software engineering, abstraction levels help manage complexity by hiding unnecessary details and exposing only the relevant parts of an object or process.</p><p>Consider this metaphorical poem by Emily Dickinson:</p><blockquote>“Hope” is the thing with feathers -</blockquote><blockquote>That perches in the soul -</blockquote><blockquote>And sings the tune without the words -</blockquote><blockquote>And never stops — at all -</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*POz0pbeXpSWMqcyM" /></figure><p>Just as Dickinson uses the metaphor of a bird to represent hope, abstraction in code hides complexity while providing a simple interface for the user. It allows developers to work with higher-level concepts without worrying about underlying details:</p><pre>public abstract class Vehicle {<br>   public abstract void startEngine();<br>}<br><br><br>public class Car extends Vehicle {<br>   @Override<br>   public void startEngine() {<br>       System.out.println(&quot;Car engine started.&quot;);<br>   }<br>}<br>public class Motorcycle extends Vehicle {<br>   @Override<br>   public void startEngine() {<br>       System.out.println(&quot;Motorcycle engine started.&quot;);<br>   }<br>}</pre><h3>The Art of Clear and Efficient Code: “Ars Poetica”</h3><p>The term “Ars Poetica” refers to a genre of poetry that discusses the art of writing poetry itself. It is a meta-poetic form where the poet reflects on their own craft, creating a self-referential work that examines the nature and purpose of poetry.</p><p>Archibald MacLeish’s poem, “Ars Poetica,” demonstrates this concept:</p><blockquote>A poem should be palpable and mute</blockquote><blockquote>As a globed fruit,</blockquote><blockquote>Dumb</blockquote><blockquote>As old medallions to the thumb,</blockquote><blockquote>Silent as the sleeve-worn stone</blockquote><blockquote>Of casement ledges where the moss has grown —</blockquote><blockquote>A poem should be wordless</blockquote><blockquote>As the flight of birds.</blockquote><p>In coding, Java Reflections can be compared to the meta-nature of “Ars Poetica”. Just as the poem reflects on the art of poetry, Java Reflections allow a program to modify its own structure and behavior at runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*CJAopwtCzbFHuA5p" /><figcaption>Reflections</figcaption></figure><p>Here’s an example of using Java Reflections to dynamically invoke methods:</p><pre>import java.lang.reflect.Method;<br><br><br>public class ReflectionExample {<br><br><br>   public void sayHello() {<br>       System.out.println(&quot;Hello, world!&quot;);<br>   }<br><br><br>   public static void main(String[] args) {<br>       try {<br>           Class&lt;?&gt; clazz = ReflectionExample.class;<br>           Method method = clazz.getMethod(&quot;sayHello&quot;);<br>           ReflectionExample instance = new ReflectionExample();<br>           method.invoke(instance);<br>       } catch (Exception e) {<br>           e.printStackTrace();<br>       }<br>   }<br>}</pre><p>This example demonstrates the power of reflections by allowing the program to inspect its own class and invoke a method dynamically. The <em>ReflectionExample</em> class has a <em>sayHello</em> method, and through reflections, we can dynamically call this method at runtime.</p><h3>The Joy of Craftsmanship</h3><p>Finally, both poetry and coding are about craftsmanship. The satisfaction from writing a beautiful sonnet or an elegant piece of code arrives when we solve complex problems with well-crafted solutions. When we feel a sense of accomplishment similar to that of a poet completing a masterpiece.</p><h3>Disclaimer</h3><p>While many argue that developers should treat their code as a piece of art, others believe they should avoid becoming too attached to it. Code is inherently temporary and constantly evolves, either by the original developer or someone else. Unlike art, which aims for timelessness, code is meant to be functional and adaptable. The short-lived nature of code demands flexibility and openness to change, reflecting the fast-paced and collaborative world of software development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*dnYQR_8J7WWIgSuC" /><figcaption>Art is timeless; Code isn’t</figcaption></figure><p>Thanks for reading! Feel free to explore more articles on my Medium <a href="https://medium.com/@amitpeer05">profile</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89a3b7e12b7e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wix-engineering/shall-i-compare-thee-to-a-summers-day-exploring-the-poetry-in-code-89a3b7e12b7e">“Shall I Compare Thee to a Summer’s Day”: Exploring the Poetry in Code</a> was originally published in <a href="https://medium.com/wix-engineering">Wix Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let Others Solve Your Problems: Change Your Mindset From APIs to SPIs]]></title>
            <link>https://medium.com/wix-engineering/let-others-solve-your-problems-change-your-mindset-from-apis-to-spis-781afa48574f?source=rss-50119598efed------2</link>
            <guid isPermaLink="false">https://medium.com/p/781afa48574f</guid>
            <category><![CDATA[system-architecture]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[scale]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Amit Pe'er]]></dc:creator>
            <pubDate>Wed, 09 Feb 2022 13:40:21 GMT</pubDate>
            <atom:updated>2022-07-10T07:26:54.630Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*i2d6lXLX_ZjFH-IR" /></figure><p>What if I told you there is a way to scale your system to infinity without writing a single line of code?</p><p>Moreover, by adopting this paradigm your system can become a platform, that is not limited only to what you develop, but can be plugged in with further business logic from the outside?</p><p>As engineers, our job is to solve problems, and we’re probably OK at solving them (give or take a few extra tricky ones). However, sometimes, these problems can be delegated. All we need to do is to change the mindset from writing APIs, to writing <em>“reverse APIs”,</em> also known as SPIs.</p><p>A year ago, we decided at Wix Stores that we need a way to give our users the capability to provide their store’s customers with real-time shipping rates, from as many delivery services as possible during the checkout phase.</p><p>Wix Stores is Wix’s eCommerce solution. We provide a platform for merchants to sell goods online and give them tools to help build a stunning store that contains everything an online shop needs: cart, checkout, catalog, order services, shipping management, and much more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*GH5N6wRvJFkn4Td__tMWwA.png" /><figcaption>Real-time shipping rates at the checkout phase</figcaption></figure><p>To accomplish the shipping rates challenge, we took advantage of the SPI pattern and established a platform that enables seamless integration between us and various shipping rates providers (i.e “delivery/carrier services”). Today, we have a fully functioning platform with multiple integrated partners. For us, plugging in a new service is a no-brainer.</p><p>In this post, I will present the SPI approach and the paradigm shift it brings with it. I’ll share our takeaways and the important lessons we learned from the process. By the end of it, you will have yet another tool in your toolbox that may help you (or someone else) solve tomorrow’s problem.</p><p>So what on earth does SPI mean? Good question. It stands for “Service Provider Interface”. Technically it is an API — a contract. However, contrary to “normal” cases in which <em>you</em> define and <em>you</em> implement, with SPIs, <em>you</em> define and let <em>others</em> implement. A form of “reverse API”, if you will, where each implementation is an external extension to your system.</p><p>SPI has been around for quite some time within the world of software engineering. As <a href="https://stackoverflow.com/a/2956803">answered</a> in Stack Overflow:</p><blockquote>“The API is the description of classes/interfaces/methods/… that you <em>call and use</em> to achieve a goal, and the SPI is the description of classes/interfaces/methods/… that you <em>extend and implement</em> to achieve a goal.</blockquote><blockquote>Put differently, the API tells you what a specific class/method does for you, and the SPI tells you what you must do to conform.</blockquote><p>In this article I’ll talk about the SPI within the context <em>web architecture</em>, which is the young, hipster brother of the classic SPI presented in software engineering classic literature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*bgG2cZtECnsF5WsI0ayZ3A.png" /><figcaption>“classic” API — each 3rd party defines its own contract</figcaption></figure><p>As previously mentioned, our use case was to provide real-time shipping rates from any delivery service (the “service provider” within the context of <em>Service Provider Interface</em>). The naive approach would be to go ahead and study the API of each one of the different services, and call each one separately.</p><p>Obviously, this does not scale.</p><p>Let’s say that today we want to retrieve shipping rates from USPS and Canada Post. Studying their APIs, implementing the code, testing, rolling out — this whole cycle takes time. Now let’s say we wish to add an additional provider — Royal Mail. We would have to start all over again. Now, imagine we need to add 20 more providers — this will never end.</p><p>Instead, we wanted to have a single API, let’s call it <em>getShippingRates</em>, which all the delivery services would implement on <em>their</em> side. Then we would just have to call each one with the same request, expect the same response, collect all the results, and finally show them to the user. This way of adding a new service requires zero effort from our perspective. Our application works with a single contract — the SPI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*VDvvusfhNnO9ZV0uWDmzqQ.png" /><figcaption>SPI — single contract to rule them all</figcaption></figure><p><em>So how do we start? API first.</em></p><p>The first thing we had to do, which was the most difficult part of the process, was to define the so-called <em>getShippingRates</em> API. Think about it — it has to be good enough that the different partners would actually agree to implement.</p><p><em>getShippingRates’s request</em> is the payload we send to each one, and the response is what we expect them to return. Thus, the request must contain all the relevant information the partners need in order to calculate shipping rates, and the response needs to make sense so it is easier for them to answer.</p><p>This was not an easy task at all. We went back and forth with the engineering and business departments of the different partners, prepared countless drafts and made changes over and over again until we came up with the final version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*6ob6I3bLbpbH2ne9" /><figcaption>Agree to agree</figcaption></figure><p><em>And once the API is defined?</em></p><p>Once we came up with the final version of the API, and at least one partner willing to implement it and work with us, we had to design and write the software infrastructure for the two main components of the system: the SPI discovery mechanism and the aggregation logic.</p><p>Before diving into the implementation, let’s go over the two main requirements of the feature:</p><ol><li>A merchant (store owner) can enable a delivery service within a shipping region. For example, USPS in New York. Now, all New-York-based customers that shop in her store should be able to receive shipping rates from USPS during the checkout phase.</li><li>During the checkout phase and having entered a shipping address, the buyer will receive shipping rates from all the providers that the merchant has enabled for that region. To further our example, if I’m a customer in the checkout phase and I enter a New-York address, I will receive shipping rates from USPS.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*N4EwbhF7dsl4EiQM" /><figcaption>One contract; many delivery services</figcaption></figure><p><strong>Extensions Discovery Mechanism</strong></p><p>The first component of the system is the e<em>xtensions</em> <em>discovery mechanism</em>. Decent infrastructure is required to allow for adding and removing extensions from the system. The end-game is to offer some form of self-service portal so that each extension could implement the SPI and register itself to the system by providing the endpoint URL it’s listening to. Then, merchants will be able to search through all of the registered services and choose the ones they want.</p><p>That being said, we decided to begin with a simpler and faster solution and keep a hard-coded list of providers &amp; URLs. Each service had to manually communicate with us and provide its URL, and only then could we add it to the system. It worked well in the beginning, but we realized very quickly that it does not scale and began to head towards the self-service solution. The reason we started with the hard coded solution is because we didn’t want this development to be a dependency, and didn’t want this to effect the time-to-market. Also, a generic solution by Wix Framework team was in the making.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*BsBwlaAdcpYOMzqU" /><figcaption>Self-service is the end-game</figcaption></figure><p><strong>Aggregation</strong></p><p>The second component of the system is the <em>aggregator</em>. It calls the relevant extensions during runtime, collects all the responses, and returns those to the client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ItgQXMxu72gcXmiWwOhuMg.png" /><figcaption>High level design for the entire system</figcaption></figure><p>Some things to consider:</p><ul><li><em>Timeouts</em> — we cannot afford a single extension becoming a bottleneck and thus impinging on the end-user’s experience. We had to define a timeout for all the requests, and those that did not respect it were simply cut out. It is worth mentioning that even though we called the extensions in parallel, it was still synchronous.</li><li><em>Security</em> — the extensions would probably want to know that the request indeed came from you. Consider adding an authentication header to each request, and according to your favorite security protocol allow each service to verify the request’s origin.</li><li><em>Failures</em> — different extensions can throw errors. Don’t let them explode in your face and have the entire request fail — collect only the successful responses.</li><li><em>Monitoring</em> — this is practically a <em>must.</em> You really want to have visibility on what’s going on with each extension: log every request/response; have success rates &amp; response time metrics; define alerts. This is gold.</li><li><em>Bugs</em> — never forget that the extension’s logic can contain bugs as well. You must accept that you can’t control it and can’t expect everything to happen as planned. You need to protect your system from faulty and incorrect responses from the extensions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*5HQT7_b8Zo55Cbs-" /><figcaption>Don’t keep your users waiting. Define timeouts.</figcaption></figure><p><strong>Summary</strong></p><p>When designing software architecture, try identify the “extension points” of your system. These are sweet spots that, if defined correctly, can take your system to the next level in becoming a <em>platform</em>. In the best case scenario your system will become much more scalable. Worst-case scenario? “<em>Platform” </em>is a<em> </em>buzzword these days, so you may just impress your bosses while you’re at it. Do consider that I have presented only one use-case and implementation for the SPI pattern — remember that there are many, many more.</p><p>Thanks for reading! As always, you’re welcome <a href="https://medium.com/@amitpeer05">here</a> to read more of my posts.</p><p><a href="https://medium.com/u/93bc38e895ee"><em>Roy Shachori</em></a><em> contributed to this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=781afa48574f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wix-engineering/let-others-solve-your-problems-change-your-mindset-from-apis-to-spis-781afa48574f">Let Others Solve Your Problems: Change Your Mindset From APIs to SPIs</a> was originally published in <a href="https://medium.com/wix-engineering">Wix Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to integrate 3rd Party APIs without Fear (Part 2)]]></title>
            <link>https://medium.com/wix-engineering/how-to-integrate-3rd-party-apis-without-fear-part-2-1abfe35a4fa4?source=rss-50119598efed------2</link>
            <guid isPermaLink="false">https://medium.com/p/1abfe35a4fa4</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[tdd]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Amit Pe'er]]></dc:creator>
            <pubDate>Fri, 16 Jul 2021 10:22:11 GMT</pubDate>
            <atom:updated>2021-07-16T10:22:11.321Z</atom:updated>
            <content:encoded><![CDATA[<h3>A journey between TDD, contract-tests and clean code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y0aJT_dgkEUmBzBn1CXDHQ.jpeg" /></figure><p>Welcome to the second part of ‘<em>How to integrate third-party APIs without fear’. It’s recommended to read the </em><a href="https://medium.com/@amitpeer05/how-to-integrate-3rd-party-apis-without-fear-part-1-a3dd4bf42cd6"><em>previous part</em></a><em> before proceeding with this one.</em></p><p><strong>A quick recap:</strong> In the previous part we had introduced the <em>Contract Testing</em> concept. Our toy example was a Scala client library we wanted to write in order to send messages to Slack channels. We defined a list of requirements for that client and called it the ‘<em>Contract</em>’. This contract was an abstract class containing a few specifications (<em>requirements; tests</em>). It was an abstract class since it’s going to have <em>two</em> different implementations: one “real” and one “fake”.</p><p><strong><em>You promised we’d see more code, right?</em></strong></p><p>To satisfy the contract’s requirements, we decide that this is the trait (equivalent to Java’s interface, more or less) for our Slack Client:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebd9ff5d7766b29399b8d466a4cef2bc/href">https://medium.com/media/ebd9ff5d7766b29399b8d466a4cef2bc/href</a></iframe><p>The method returns a <em>Try</em> since we want to imply to the user of this class that bad things might happen in there. This way, we elegantly force the API consumer to consider failures rather than ignoring them — we keep her safe. We return a u<em>nit</em> since posting a message is a <em>side effect.</em></p><p>According to the first law of TDD — <em>You are not allowed to write any production code unless it is to make a failing unit test pass </em>— we’ll go ahead implement the first test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7be1b0b5b0ecdb9d4a6d0d9bc62342b6/href">https://medium.com/media/7be1b0b5b0ecdb9d4a6d0d9bc62342b6/href</a></iframe><p><em>slackClient </em>is an abstract member that would be overridden by each one of the contract’s implementations. <em>channelId </em>is, for now, the ID of a real channel I had created for testing purposes.</p><p>Alas, when we try to run this test class we get something like:</p><pre>org.specs2.control.UserException: cannot create an instance for class contract.slack.sync.SlackClientContract</pre><p>That’s because it’s an <em>abstract</em> class. We must provide a concrete implementation if we wish to execute the tests. But which one will we choose? the Real or the Fake?</p><p>I prefer to begin with the real one. When a real test passes, I’m <em>sure</em> that everything is wired up correctly and the integration is perfect. Only then would I go ahead and write the fake as well.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3408edd6b464d78730078d1b0051b10e/href">https://medium.com/media/3408edd6b464d78730078d1b0051b10e/href</a></iframe><p>A word regarding <em>TokenProvider: </em>we don’t want the API credentials to sit around in our code, rot, and god forbid — be committed to the VCS. So we’ll set it as a system property. Remember that this is the <em>real</em> API token we were given by Slack.</p><p>However, we have a <em>compilation error</em> in line 3. <em>HttpSlackClient</em> does not yet exist. Luckily enough, according to the second law of TDD — <em>You are not allowed to write any more of a unit test than is sufficient to fail, and compilation failures are failures — </em>we can now go ahead and write some <em>actual</em> production code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8cdc74c5517a77ce49b78d2f7277a13/href">https://medium.com/media/c8cdc74c5517a77ce49b78d2f7277a13/href</a></iframe><p>Just to make sure everything is wired correctly, let’s run the test. As required, it fails with no less than my favorite exception. It ought to be meaning we’re doing something good (in Scala, three question marks are syntax sugar for <em>throw new NotImplementedError).</em></p><pre>java.lang.Exception: an implementation is missing</pre><p>We’re set to go and can finally write some <em>actual</em> production code. I used <a href="https://github.com/scalaj/scalaj-http">scalaj-http</a> to perform the HTTP request (because it’s a dead-simple library), and a standard Jackon’s <a href="https://github.com/Amitpe/contract-test-example/blob/master/src/main/scala/contract/json/DefaultObjectMapper.scala">object mapper</a> to parse the response body. While doing so, try to remember the third and final law of TDD: <em>You are not allowed to write any more production code than is sufficient to pass the one failing unit test.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4dea4dc6d1fbb0c4d365d26d8e9e27e/href">https://medium.com/media/e4dea4dc6d1fbb0c4d365d26d8e9e27e/href</a></iframe><p>Not only is the test green, but we can also see the message in the test channel we created in Slack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXy7uZ4Siekw-mX8zkGxlA.png" /></figure><p>Indeed, I had to play around until I got to this version of the code. It took me some time; I gradually built the HTTP request, debugged, and checked Slack’s response over and over again, and only after a few attempts have I managed to get it working.</p><p>But how does it differ from any other integration we had written in the past? That’s what we always do — play around with the code until we manage to get it all wired up correctly. Well, <strong>now we have a test. </strong>We can always go back and run it, and this way be sure the integration is still correct.</p><p><strong><em>So far so code. But you mentioned that ‘real’ tests won’t even run within the CI build. What’s next?</em></strong></p><p>Now we know for sure that our integration is correct. We have a ‘real’ test that gives us the strong validation we needed. The problem with this test is that it’s not isolated — it’s interacting with the real world. It might be flaky, slow, and probably shouldn’t run within our CI systems. Indeed, now it’s time to write the ‘fake’ tests class — which implements the <strong><em>same</em></strong> contract.</p><p>At this point, we stop, scratch our heads, and think. Our production code sends an HTTP request out to the world. That’s a <em>side effect</em>. How can we mimic the response of an HTTP request? We don’t have an object to mock &amp; verify upon. If so, how can we write a test that would eventually pass?</p><p><strong>For that, we need a Mock Server.</strong></p><p>Luckily enough, this can be easily set up by using <a href="https://github.com/wix/wix-http-testkit">wix-http-testkit</a>, which is an open-source library by Wix that helps exactly for these scenarios and is using <a href="https://github.com/akka/akka-http">akka-http</a> under-the-hood.</p><p>Therefore, we’ll create a <em>FakeSlackServer</em>, which would resemble Slack within our test environment. The fake server would listen to the HTTP calls made by production code during tests execution, respond correspondingly, and satisfy our Slack client’s specifications. <strong>It will mimic Slack behavior just as much as we need it to — no less, no more.</strong></p><p>Following is an initial implementation of <em>FakeSlackServer</em>, just enough to support the first requirement:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eee2a14ff6fcc6ccd0379507b36e4f94/href">https://medium.com/media/eee2a14ff6fcc6ccd0379507b36e4f94/href</a></iframe><p>Pay special attention to the <em>port</em> our class receives as a member — that’s the port it would listen to. We will have to make sure that during the execution of the fake tests, our client would perform the HTTP calls to this <em>exact</em> port.</p><p>After having initiated the server on that port, we define a <em>RequestHandler. RequestHandler </em>is merely a partial function<em> </em>between <em>HttpRequest</em> and <em>HttpResponse</em>. We pattern match the request against the type of request we’d expect when posting a message to Slack, and return a valid response if indeed this is the case. For any other case, our server returns 500 by default.</p><p>Obviously, that’s a very thin implementation and there’s a lot more behavior we can resemble. We hadn’t fully validated the request body, nor did we verify the API token. but that’s just enough to make the first test green. <strong>The more tests we add, the more behavior we’d have to implement within <em>FakeSlackServer</em>.</strong></p><p>Finally, we’re ready to write down the fake test class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43e951fe0f9a62be114cd5cf187f17a4/href">https://medium.com/media/43e951fe0f9a62be114cd5cf187f17a4/href</a></iframe><p>Take a look at how <em>baseUrl</em> is being accepted as a class member of <em>HttpSlackClient</em>. That’s because we wish to control where the HTTP calls would go to — real Slack or localhost. Executing the following class would result in a <em>green</em> test against the <em>FakeSlackSever</em> we had written.</p><p>It had cost us a few drops of sweat but we had finally completed a full TDD cycle. We have two green tests that cover the first requirement — one executes against ‘real’ Slack and the other against the ‘fake’ one. From now on, things will flow more smoothly.</p><p>Indeed, It’s much easier to continue now since we’re not afraid to change existing code — we have tests to verify there’s no regression. Also, the foundation of <em>FakeSlackSever</em> is already set up — we would just have to gradually extend it to support every new requirement we’re about to implement.</p><p>At any given point in time, we can go ahead and execute each of the concrete test classes we choose. If we need a strong validation we’ll run the real one. We’ll probably do that less, maybe after big changes, or before committing the code. When we don’t need as much assurance and want a quick feedback, we’ll execute the fake tests class. We’ll probably do that a lot.</p><p>For every new specification we write down in <em>SlackClientContract, </em>we<em> must </em>implement both concrete classes. We cannot escape or postpone it — we are <em>forced</em> to support the requirement for the two different environments. It keeps us safe and assured that each requirement we demand is correctly backed up in the client.</p><p>We’re done! check out the full code in <a href="https://github.com/Amitpe/contract-test-example">GitHub</a> to see the implementation for all of the requirements.</p><h3>Summary</h3><p>Third-party integrations can be weak spots within our systems. But they don’t <em>have</em> to. While reading this article, I hope you understood the importance of testing these integrations, and are eager to adopt the technique I had introduced.</p><p>You might argue that a lot of boiler platting is involved and that the process is too long. I would respond that after a few times you get the hang of it, and it flows much easier than at the beginning. <strong>The value is indeed higher than the cost.</strong></p><p>Of course, you may say that you <a href="https://dev.to/d_broadbridge/you-might-not-need-integration-tests-14a8">might not need integration tests</a> at all. I’d say that there’s no single truth. Remember that each case is different, and try to judge every trade on its own merits.</p><p>Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1abfe35a4fa4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wix-engineering/how-to-integrate-3rd-party-apis-without-fear-part-2-1abfe35a4fa4">How to integrate 3rd Party APIs without Fear (Part 2)</a> was originally published in <a href="https://medium.com/wix-engineering">Wix Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chunks Producer/Consumer]]></title>
            <link>https://medium.com/wix-engineering/chunks-producer-consumer-f97a834df00d?source=rss-50119598efed------2</link>
            <guid isPermaLink="false">https://medium.com/p/f97a834df00d</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[design-patterns]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Amit Pe'er]]></dc:creator>
            <pubDate>Tue, 28 Jul 2020 10:35:32 GMT</pubDate>
            <atom:updated>2020-08-04T09:08:41.029Z</atom:updated>
            <content:encoded><![CDATA[<h4>Handling Large Messages Within Kafka</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xFwzh2QX1JETRnOCTv0ohA.jpeg" /></figure><p>Hello, Kafka fans. Have you ever encountered the following exception?</p><pre>org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.</pre><p>Or this?</p><pre>The message is 37455369 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.</pre><p>If the answer is <em>yes — </em>this post is for you. If the answer is <em>no, </em>then this post may be helpful for your future selves.</p><p>At Wix, we widely use Kafka in various use cases, across multiple micro-services. One day, I started to occasionally notice one of these exceptions. Indeed, in one particular use case, my application would <em>sometimes</em> try to produce a message larger than the maximum size allowed (which was 1 MB).</p><p>— So what do I do now?</p><p><strong><em>Disclaimer — </em></strong><em>This article requires some basic knowledge about Kafka. Yet, the solutions provided below are somewhat higher in the abstraction level and do not dive deep into Kafka low-levels API.</em></p><h4>Easy — remove the limitation.</h4><p>As the typical developer I am, the first thing I tried to do is to figure out how to remove the limitation, or at least extend it. I poked around, googled it, and asked a few experts. After some investigation, I figured out that it’s not recommended, might slow down the whole Kafka cluster, and cause issues.</p><p>I had no choice but to scratch my head and try a different approach.</p><h4>First thought — refactor</h4><p>Of course, I could introduce a refactor within my application so that it would never attempt to produce large messages <em>at all</em>. Naturally, It was my least favorite choice, as it would consume too much time and effort. I didn’t want to change the architecture just to solve somewhat of an edge case.</p><h4>Coming up next — external storage</h4><p>The next idea was as follows: when a large message is encountered — upload it to external storage (such as S3), and produce a link to download it. The consumer would then have to go ahead and download the message from that storage. It sounded like a decent solution, but I was annoyed by the fact I introduced another point of failure to my application. I had to deal with a 3rd party storage solution, study its API, and it might be an opening for a whole new set of bugs. I preferred to remain within the Kafka context and enjoy its resilience.</p><h4>Last but not least — Chunks Producer</h4><p>I initially thought about is as <em>pieces </em>producer, but then someone suggested the word <em>chunks</em> and it sounded much better. The initial idea was the following:</p><p><strong>Producer:</strong></p><ol><li>Whenever a large message is encountered, split it into chunks such that no chunk is larger than X bytes (X = message.max.bytes property).</li><li>Give each chunk a unique ID.</li><li>Store each chunk in persistent storage. Luckily enough, I could use an in-memory key-value persistent store backed by Kafka, which was implemented here at Wix. This way, I could stay within Kafka-context as I desired, such that each chunk is actually produced to a <a href="https://towardsdatascience.com/log-compacted-topics-in-apache-kafka-b1aa1e4665a7">compacted topic</a>. Read more about it in bullet #3 <a href="https://medium.com/wix-engineering/6-event-driven-architecture-patterns-part-1-93758b253f47">here</a>.</li><li>Produce an array of all the chunk IDs.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Een_2TruEE4MFYsLQTtMg.png" /></figure><p><strong>Consumer:</strong></p><ol><li>Consume the chunk IDs.</li><li>For each ID, fetch the chunk from the datastore <em>(eventually consistent spoiler: what happens if the chunk is not there yet?).</em></li><li>Aggregate the chunks to the complete (original) message.</li><li>Invoke the original handler with the original message.</li><li>Delete the chunks from the store.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l4vMpl5-MHW6kBdS_s2JsA.png" /></figure><h4>Fine-tuning &amp; Pitfalls</h4><p>As this was merely the draft, while I was implementing that feature (which was available as an opt-in only for other developers too), a few ideas for improvements came up.</p><p><strong>Consider eventual consistency</strong><br>Bear in mind that Kafka guarantees eventual consistency. Consider the following scenario:</p><ol><li><em>Producer:</em> Chunks are delivered to the store in an async and eventual-consistent fashion (via Kafka or any other async framework).</li><li><em>Producer:</em> Chunk ID’s are produced.</li><li><em>Consumer:</em> Chunk ID’s are consumed.</li><li><em>Consumer: </em>Fails to get a chunk from the store — it’s not there yet<em>. </em>Bullet #2 didn’t wait until bullet #1 is finished.</li></ol><p>This problem can be solved in different ways. I chose to retry on the consumer’s side until all the chunks have indeed arrived: for each chunk ID, try to fetch the chunk. If not found — throw a <em>“blocking”</em> exception which will cause a retry for the consumer and block the consumption of further messages.</p><p><strong>Compression<em><br></em></strong>Of course, if compressing the data helps you get below the threshold without having to split it— go for it and throw this post to the trash. But if compressing doesn’t help, I see no reason to do it <em>and</em> to split into chunks. Perhaps if your large messages throughput is <em>extremely</em> high and you really need to save space.</p><p><strong>Disk &gt; Memory<em><br></em></strong>I had initially used an in-memory store to persist the chunks. The problem was that for a high latency of large messages my application memory could blow up. So I switched to using disk space: the same K/V store, backed by Kafka, but instead of reading/writing from an in-memory data structure — read/write from the disk (powered by H2 DB engine).</p><p><strong>At least once<em><br></em></strong>If you guarantee an <a href="https://www.cloudcomputingpatterns.org/at_least_once_delivery/">at least once</a> delivery, beware of the following pitfall:</p><ul><li>Chunks consumer composes the original message.</li><li>The original handler is triggered.</li><li>Chunks are deleted from the store.</li><li><strong><em>**RETRY.**</em></strong> for some reason, the chunks handler retries (due to the “at least once” guarantee - this can happen for different reasons).</li><li>Chunks consumer starts over again and tries to fetch the chunks from the store, but they are <em>already deleted</em>.</li></ul><p>You could solve that in a few different ways. I chose not to discuss it in this post, since its a very unique edge case. Although I do hope it intrigues your imagination.</p><h4>Summary</h4><p>In this short post, I had introduced a few ideas as to how to handle large messages within Kafka world and a detailed implementation for one of them. The implementation details can vary, but the idea is pretty much the same: split to chunks; persist the chunks somewhere (preferably by using Kafka); produce a pointer/reference to the chunks; aggregate them on the other side; trigger the original handler.</p><p>As always, you’re welcomed <a href="https://medium.com/@amitpeer05">here</a> to read more of my posts. Thanks for reading, and see you next time.</p><p>Many thanks to <a href="https://medium.com/u/32e883029628">Natan Silnitsky</a> and <a href="https://medium.com/u/7be0af315855">Noam Berman</a> for the support.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f97a834df00d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wix-engineering/chunks-producer-consumer-f97a834df00d">Chunks Producer/Consumer</a> was originally published in <a href="https://medium.com/wix-engineering">Wix Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to integrate 3rd Party APIs without Fear (Part 1)]]></title>
            <link>https://medium.com/wix-engineering/how-to-integrate-3rd-party-apis-without-fear-part-1-a3dd4bf42cd6?source=rss-50119598efed------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3dd4bf42cd6</guid>
            <category><![CDATA[contract-testing]]></category>
            <category><![CDATA[api-integration]]></category>
            <category><![CDATA[test-driven-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Amit Pe'er]]></dc:creator>
            <pubDate>Fri, 24 Apr 2020 10:27:56 GMT</pubDate>
            <atom:updated>2021-07-16T10:23:49.455Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to integrate 3rd Party APIs without Fear</h3><h3>A journey between TDD, contract-tests and clean code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Dip-HU6DAhiEE3jY" /></figure><p>Third-party APIs. How can we write our programs without them?<em> </em>It feels like we’ve reached a point in humanity where for almost every common problem, someone has already found a solution. So why not just use it? Why waste time over solving <em>these</em> problems, when we could be focusing 100% on our own business logic?</p><p>This is the first post out of a two parts article. If you have already read this one, feel free to jump to the <a href="https://medium.com/@amitpeer05/how-to-integrate-3rd-party-apis-without-fear-part-2-1abfe35a4fa4">second</a> part.</p><p>Indeed, when writing an application nowadays we almost always find ourselves integrating with some external API. Whether it’s from a third-party, a micro-service someone from our company wrote, or even our own.</p><p>These kinds of integrations can be tricky. Tricky in such a way that we sometimes find ourselves tempted not to test them. That’s because writing integration tests for 3rd party API’s is <em>not</em> an easy task. ‘<em>It’s an overhead’</em> we tell ourselves. Gloomy thoughts such as ‘<em>Testing it will take too much time anyway’</em> often lurk inside our minds. The problem is that when we choose not to test, the next time we introduce a change within the integration code we’ll doom our future selves to deploy with fear, trembling in case we break that integration in production.</p><blockquote><em>Learning the third-party code is hard. Integrating the third-party code is hard too. Doing both at the same time is doubly hard. What if we took a different approach? Instead of experimenting and trying out the new stuff in our production code, we could write some tests to explore our understanding of the third-party code.</em></blockquote><blockquote>— Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship</blockquote><p>Here at Wix Engineering we respect our future selves, care deeply for resiliency, and embrace <a href="https://en.wikipedia.org/wiki/Test-driven_development">TDD</a> (<em>Test Driven Development</em>) as a way of life. This means we have already adopted a methodology for these exact kinds of interactions. In this post, I’ll present one of our approaches to solving this problem, seasoned with a touch of my own ideology. Code examples are in <em>Scala</em>, which is a JVM, somewhat functional language. I chose it since that’s the language we mostly practice here at Wix Server guild, and also because, in my opinion, it’s one of the coolest programming languages out there.</p><p><strong><em>What’s In It For Me? (Or: why should I read this?)</em></strong></p><p>Once you’ve read this article, I hope you will:</p><ul><li>Fearlessly integrate with third-party APIs</li><li>Sharpen your TDD skills</li><li>Be a cleaner coder</li><li>Earn some useful Scala tips &amp; tricks</li></ul><p><strong><em>O.K., I’m convinced. What’s the first thing I need to know?</em></strong></p><p>In this post, we’ll deep dive into a concept called <em>Contract Testing. </em>There are different <a href="https://medium.com/@liran.tal/a-comprehensive-guide-to-contract-testing-apis-in-a-service-oriented-architecture-5695ccf9ac5a">ways</a> and <a href="https://medium.com/@axelhodler/integration-tests-for-third-party-apis-dab67c52e352">approaches</a> to writing contract tests. Some are more complex, some simpler. I’ll focus on the methodology that I find to be the most convenient.</p><p>For me, contract testing is synonymous with <em>expectation management</em>. It’s like any other contract you sign — whether it’s a contract for an apartment, a job, or whatever. We commit to writing and sign over all of the expectations that both sides of the parties agree on.</p><p>The same thing goes for a third-party API — we have a <em>contract</em> with it, which describes the criteria we expect it to meet; the terms and conditions it must stand for, with respect to <em>our</em> needs.</p><p>A contract test could then be viewed as a list of <em>requirements</em> we expect this API to abide with. Reading the contract test gives us confidence that we know for sure how this API will behave during our application’s lifetime.</p><p><strong><em>Sounds cool! But how exactly does it work?</em></strong></p><p>The main idea is to eventually wind up with <em>two</em> implementations for <em>one </em>contract:</p><blockquote><strong>Real</strong> — <em>code triggered by these tests would perform the HTTP calls to the </em>real<em> API server, out there in the external world, </em>exactly<em> as would happen in our production environment. These tests should </em><strong>not</strong><em> run within the CI system — they’re here to give strong confidence, but they don’t fit regular CI builds.</em></blockquote><blockquote><strong>Fake</strong> —<em> tests that would run against a </em>fake<em> server we write ourselves, or in some cases supplied by the SDK of the API provider (which is a good indicator we want to integrate with it). This so-called ‘server’ resembles and mimics the API behavior. HTTP calls would </em><strong>not</strong><em> go out to the external world, but rather to that fake server, started locally as a </em><a href="https://github.com/testdouble/contributing-tests/wiki/Collaboration-Test"><em>test collaborator</em></a><em>.</em></blockquote><p>Having these two kinds of implementations is, in my opinion, very important for the development velocity. ‘Real’ tests are slower to run, but significantly instill confidence in the correctness of our integration. We would probably want to run them after a big change, or before committing our code. They should not really be run as part of the CI build since they interact with the real world and might be flaky and/or too slow.</p><p>The ‘fake’ tests, on the other hand, would be faster to run and still give us a <em>certain</em> amount of confidence in the integration’s validation. Now, how much confidence they give us depends on <em>how</em> the fake server is written. The more it is accurate and resembles the real API, the more confidence we’ll have in these tests. We would probably execute these tests more often than the real ones. They’re fast, hermetic, and reproducible, giving us the confidence to run them as part of our regular CI builds, without bringing external instability.</p><p>The main benefit would be that a developer could easily juggle between executing the ‘real’ and the ‘fake’ tests while developing, without having to change any code or configuration whatsoever. If she needs a strong validation, she shall execute the ‘real’ tests. If she’ll be satisfied with less validation and wants quick feedback — she may execute the ‘fake’ tests. And all without changing a single line of code.</p><p><strong><em>The concept is clear, but I think we need a ‘real world’ example.</em></strong></p><p>In this toy example, let’s try and create an application that integrates with <a href="http://slack.com/">Slack</a>. More specifically, we want to write some code that allows sending messages to channels. Consider the following background story: every time our system detects that some alert fires, it must send a message to our team’s urgent Slack channel.</p><p>For the sake of the example, let us zoom in to the <em>integration point</em> itself — that is the <em>Client</em> class we shall write for Slack. Let’s ignore the fact that there are a few pre-made Java clients for Slack out there for a moment — it’s a privilege we won’t, and don’t always have. And finally, to keep it really simple, the client is going to be synchronous.</p><p>The first thing we’d want to do is to study and play around with Slack’s <a href="https://api.slack.com/methods/chat.postMessage">Post Message</a> endpoint. Then, we can write down the requirements we expect from our Slack Client. Those can be written as a list of <em>specifications</em>, or more specifically — <em>tests</em>. It’s a good hint as to why <a href="http://etorreborre.github.io/specs2/">Specs2</a>, a testing framework for Scala, is called the way it is.</p><p>To make things easier, I’ll sum up the essential information we need to know about that endpoint. I chose <em>application/JSON</em> as the request format.</p><pre>POST <a href="https://slack.com/api/chat.postMessage">https://slack.com/api/chat.postMessage</a></pre><pre><strong><em>Body:<br></em></strong>{<br>   &quot;channel&quot;: {channel ID},<br>   &quot;text&quot;: {text}<br>}</pre><pre><strong><em>Headers</em>:</strong><br>Authorization: Bearer {<a href="https://api.slack.com/apps">token</a>}<br>Content-Type: application/json</pre><p>After studying the API, while considering how we need this client to behave within our application (remember that the client is an actor within a larger system that needs to interact with it), these might be the specifications we come up with. This will be our <em>contract</em> with the Slack:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb60d3b634a410d751ac1d63c5a5fdb9/href">https://medium.com/media/bb60d3b634a410d751ac1d63c5a5fdb9/href</a></iframe><p>Let’s break it down: the first requirement is pretty straight forward — our client simply needs to be able to post messages to channels. The second and the third one verify that it would provide its users (we, or whoever it is to use this client), the correct exception given they had provided a bad token/channel. We don’t want it to simply explode with some arbitrary <em>null pointer</em>/<em>Parsing Exception,</em> which would enforce an intensive debugging session in production.</p><p>At last, we have a <em>contract</em>. Now the sharper ones among you have probably noticed it’s an <em>abstract</em> class. That’s because this class merely defines the <em>requirements</em>, but knows nothing about the implementation details. there would be <em>two</em> concrete test classes to extend it — one for the ‘real’ tests, and the other for the ‘fake’ ones. The key difference between them would be the environment setup — and<strong><em> not</em></strong> the tests themselves.</p><p>The first concrete class <em>(‘real’)</em> will initiate the SUT (system under test) so that it would perform HTTP calls directly to Slack, exactly as would happen in production. When we execute this class, we expect to see the output of our production code in real life with our own eyes— In this case, see a new message in the appropriate Slack channel.</p><p>The second concrete class <em>(‘fake’)</em> would start up a fake server before tests begin to run, and configure our client to make the calls to <em>that</em> server, rather than to Slack itself. Production code triggered by this class would interact with that server rather than Slack. We won’t see any message being sent anywhere, but if we have enough trust in that server, we shall be satisfied when we see these tests green.</p><p>As promised at the beginning — <em>one</em> contract, <em>two</em> implementations.</p><p>Thanks for reading!</p><p>That was the first part of my blog-post ‘<em>How to integrate third-party APIs without fear’. </em>In the next part, we’ll continue <em>TDD’ing</em> our way through the example, and see more code rather than words. It would be more of a technical, Scala-oriented post, and a less conceptual one. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3dd4bf42cd6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wix-engineering/how-to-integrate-3rd-party-apis-without-fear-part-1-a3dd4bf42cd6">How to integrate 3rd Party APIs without Fear (Part 1)</a> was originally published in <a href="https://medium.com/wix-engineering">Wix Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>