<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dor Sever on Medium]]></title>
        <description><![CDATA[Stories by Dor Sever on Medium]]></description>
        <link>https://medium.com/@dorsever?source=rss-827c327d5d0c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*sLB2DMNN7YPAv4sI</url>
            <title>Stories by Dor Sever on Medium</title>
            <link>https://medium.com/@dorsever?source=rss-827c327d5d0c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:20:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dorsever/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to migrate from Elasticsearch 1.7 to 6.8 with zero downtime.]]></title>
            <link>https://medium.com/bigpanda-engineering/how-to-migrate-from-elasticsearch-1-7-to-6-8-with-zero-downtime-27340d668b3?source=rss-827c327d5d0c------2</link>
            <guid isPermaLink="false">https://medium.com/p/27340d668b3</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[availability]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Dor Sever]]></dc:creator>
            <pubDate>Wed, 25 Dec 2019 11:04:27 GMT</pubDate>
            <atom:updated>2019-12-25T11:04:27.218Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to migrate from Elasticsearch 1.7 to 6.8 with zero downtime</h3><p>My last task in <a href="https://medium.com/u/e39dfc44e6">BigPanda Engineering</a> was to upgrade an existing service from using Elasticsearch version 1.7 to a newer Elastic version 6.8.1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20yTn7K6c6vcmEdgwAgjSA.png" /></figure><p>In this post, I will share how did we migrate from elastic 1.6 to elastic 6.8 with harsh constraints like zero downtime, no data loss, zero bugs and provide you with a script that does the migration for you.</p><p>This post contains 6 chapters(one is optional):</p><ul><li>What’s in it for me— What were the new features that led us to upgrade our version?</li><li>The Constraints — What were our business requirements</li><li>How can we Address the constraints</li><li>The Plan</li><li>[Optional chapter] — How did we handle the infamous mapping explosion problem?</li><li>How to do data migration between clusters</li></ul><h3>Chapter 1 — What’s in it for me?</h3><p>What benefits are we expecting to solve by upgrading our data store?</p><p>There were a couple of reasons:</p><ol><li>Performance and stability issues — We were experiencing a huge amount of outages with long MTTR that caused us a lot of headaches. This reflected in frequent high latencies, high CPU usage and more</li><li>Non-existing support in old Elasticsearch versions — We were missing some operative knowledge in Elasticsearch, and when we searched for outside consulting we were encouraged to migrate forward to receive support</li><li>Dynamic mappings in our schema — Our current schema in Elasticsearch 1.7 used a feature called dynamic mappings that made our cluster <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.1/mapping.html#mapping-limit-settings">explode</a> multiple times and we wanted to address this issue</li><li>Poor visibility on our existing cluster — We wanted a better view under the hood and saw that later versions had great metrics exporting tools</li></ol><h3>Chapter 2 — The Constraints</h3><ul><li>ZERO downtime migration — We have active users on our system, and we cannot afford the system to be down while we are migrating</li><li>Recovery plan — We cannot afford to “lose” or or “corrupt” data, no matter what is the cost. So we must prepare a recovery plan in case our migration fails</li><li>Zero bugs — We must not change existing search functionality for end-users</li></ul><h3>Chapter 3 — Thinking of the plan</h3><p>Let’s tackle the constraints from the simples to the hardest:</p><h4>Zero bugs</h4><p>In order to address this requirement, I studied all the possible requests the service receives and what were its outputs and added unit-tests where needed.</p><p>In addition, I added multiple metrics (to the Elasticsearch Indexer and the new Elasticsearch Indexer ) to track latency, throughput, and performance, which allowed me to validate that we only improved them.</p><h4>Recovery plan</h4><ul><li>This means, that I need to address the following situation: I deploy the new code to production and stuff is not working as expected. What can I do about it then? Since I was working in a service that used <a href="https://www.youtube.com/watch?v=STKCRSUsyP0">event-sourcing,</a> I could add another listener (Diagram attached below) and start writing to a new Elasticsearch cluster without affecting production status</li></ul><h4>Zero downtime migration</h4><p>The current service is in live mode, and cannot be “deactivated” for periods longer than 5–10 minutes. The trick in getting this right is this:</p><ul><li>Store a log of all the actions your service is handling (We use Kafka in production)</li><li>Start the migration process offline (and keeping track of the offset before you started the migration).</li><li>When the migration ends, start the new service against the log with the recorded offset and catch up the lag</li><li>When the lag finishes, change your frontend side to query against the new service and you are done</li></ul><h3>Chapter 4 — The Plan</h3><p>Our current service uses the following architecture(based on message passing in Kafka):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*XgXL2my8CpCY1SRbYEY4hg.jpeg" /></figure><ol><li>Event topic contains events produced by other applications (for example, UserId 3 created)</li><li>Command topic contains the translation of these events into specific commands used by this application (for example: Add userId 3)</li><li>Elasticsearch 1.7 — The datastore of the command Topic read by the Elasticsearch Indexer.</li></ol><p>We planned to add another consumer(new Elasticsearch Indexer) to the command topic, which will read the same exact messages and write them in parallel to Elasticsearch 6.8</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*fTCLe3GYY6WhnE6xOX2pDA.jpeg" /></figure><h3>Where should I Start</h3><p>To be honest, I considered myself a newbie Elasticsearch user, and to feel confident to perform this task, I had to think about what’s the best way to approach this topic and learn it. A few things that helped were:</p><ol><li>Documentation — It’s an insanely useful resource to everything Elasticsearch. Take the time to read it and take notes (don’t miss: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html">Mapping</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html">QueryDsl</a>).</li><li>HTTP API — everything under <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cat.html">CAT</a> API. This was super useful to debug things locally and see how Elastic response. (don’t miss: cluster health, cat indices, search, delete index)</li><li>Metrics (❤️) — From the first day, we configured a new and shining dashboard with lots of cool metrics (taken from <a href="https://github.com/justwatchcom/elasticsearch_exporter"><em>elasticsearch-exporter-for-Prometheus</em></a>) that helped and pushed us to understand more about Elastic</li></ol><h3>The code</h3><p>Our codebase was using a library called <a href="https://github.com/sksamuel/elastic4s">elastic4s</a> and was using the oldest release available in the library — A really good reason to migrate on!<br>So the first thing was, to just migrate the versions, and see what breaks</p><p>There are a few tactics on how to do this code migration, and the tactic we choose was to try and restore existing functionality first in the new Elastic version without re-writing the all code from the start. AKA, reach existing functionality but on a newer version of Elasticsearch</p><p>Luckily for us, the code already contained almost full testing coverage so our task was much much simpler, and that took around 2 weeks of development time</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*CtpdXx3btOD8iGgZQVdyLA.jpeg" /></figure><p><em>It’s important to notice, that if that wasn’t the case, we would have had to invest some time in filling that coverage up and only then migrating to a new version (since one of our constraints was: do not break existing functionality)</em></p><h3>Chapter 5 — The mapping explosion problem</h3><p>Let’s describe our use-case in more detail, this is our model</p><p>class InsertMessageCommand(tags: Map[String,String])</p><p>And for example, an instance of this message would be:</p><p>new InsertMessageCommand(Map(&quot;name&quot;-&gt;&quot;dor&quot;,&quot;lastName&quot;-&gt;&quot;sever&quot;))</p><p>And given this model, we needed to support the following query requirements:</p><ol><li>query by value</li><li>query by tag name and value</li></ol><p>The way this was modeled in our Elasticsearch 1.7 schema was using a dynamic template schema (since the tag keys are dynamic, and cannot be modeled in advanced)</p><p>The dynamic template caused us multiple outages due to Mapping explosion problem, and the schema looked like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7dd8d28b54594ef024a6a1238fa8b6ed/href">https://medium.com/media/7dd8d28b54594ef024a6a1238fa8b6ed/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5de0f80ae13e18db75fd1685ea27fec9/href">https://medium.com/media/5de0f80ae13e18db75fd1685ea27fec9/href</a></iframe><h4>Nested documents solution</h4><p>Our first instinct in solving the Mapping explosion problem was to use nested documents.</p><p>We read the nested data type tutorial in Elastic docs and defined the following schema and queries</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a32556113b3933773748be692bc70f2/href">https://medium.com/media/8a32556113b3933773748be692bc70f2/href</a></iframe><p>And this solution worked. However, when we tried to insert real customer data we saw that the number of documents in our index increased by around X 500</p><p>We thoughts about the following problems and went on to find a better solution:</p><ol><li>Amount of documents we had in our cluster was around 500 million documents, and this meant that in the new schema we were going to reach two hundred fifty billion documents (that’s 250,000,000,000 documents 😱)</li><li>We read this really good blog post — <a href="https://blog.gojekengineering.com/elasticsearch-the-trouble-with-nested-documents-e97b33b46194">https://blog.gojekengineering.com/elasticsearch-the-trouble-with-nested-documents-e97b33b46194</a> which highlights that nested documents can cause high latency in queries, and heap usage problems</li><li>Testing — Since we were converting 1 document in the old cluster, to an unknown number of documents in the new cluster, it will be much harder to track if the migration process works without any data loss (if our conversion was 1:1, we could assert that the count in the old cluster equals the count in the new cluster)</li></ol><h4>Avoiding nested documents</h4><p>The real trick in this was to focus on what were the supported queries we were running: search by tag value, and search by tag key and value</p><p>The first query does not require nested documents since it works on a single field, and for the latter, we did the following trick.<br>We created a field that contains the combination of the key and the value, and whenever a user queries on key, value match, we translate his request to the corresponding text and query against that field</p><p>Example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2ae951e5310f370622cd321e8bae964/href">https://medium.com/media/e2ae951e5310f370622cd321e8bae964/href</a></iframe><h3>Chapter 6 — The migration process</h3><p>We planned to migrate about 500 million documents with zero downtime. And to do that we needed:</p><ol><li>A strategy on how to transfer data from the old Elastic to the new Elasticsearch</li><li>A strategy on how to close the lag between the start of the migration, and the end of it</li></ol><p>And our two options in closing the lag:</p><ol><li>Our messaging system is Kafka based, and we could just take the current offset before the migration start, and after the migration ended, start consuming from that specific offset. This solution requires some manual tweaking of offsets and some other stuff but will work</li><li>Another approach to solving this issue was to start consuming messages from the beginning of the topic in Kafka and make our actions on Elasticsearch idempotent — meaning, if the change was “applied” already, nothing will change in Elastic store</li></ol><p>The requests made by our service against Elastic were already idempotent so we choose option 2 because it requires zero manual work (no need to take specific offsets, and then set them afterward in a new consumer group)</p><h4>How can we migrate the data?</h4><p>These were the options we thought of:</p><ol><li>If our Kafka contained all messages from the beginning of time we could just play from the start and the end state would be equal — But since we apply retention to out topics, this was not an option</li><li>Dump messages to disk — and then ingest them to Elastic directly — this solution looked kind of weird, why store them in disk instead of just writing them directly to Elastic?</li><li>Transfer messages between old elastic to new Elastic — This meant, writing some sort of “script” (anyone said python? 😃) that will connect to the old Elasticsearch cluster, query for items, transform them to the new schema, and index them in the cluster</li></ol><p>We choose the last option and these were the design choices we had in mind</p><ol><li>Let’s not try to think about error handling unless we need to. Let’s try to write something super simple, and if errors occur, let’s try to address them. In the end, we did not need to address this issue since no errors occurred during the migration</li><li>It’s a one-off operation, so whatever works first / KISS</li><li>Metrics — Since migration processes can take long hours to days, we wanted the ability from day 1 to be able to monitor error count and to track the current progress and copy rate of the script</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*9H5pYAwy18njy6Sk4GuizA.gif" /></figure><p>We thought long and hard and choose python as our weapon of choice. The final version of the code is attached below</p><pre>dictor==0.1.2 - to copy and transform our Elasticsearch documents<br>elasticsearch==1.9.0 - to connect to &quot;old&quot; Elasticsearch<br>elasticsearch6==6.4.2 - to connect to the &quot;new&quot; Elasticsearch<br>statsd==3.3.0 - to report metrics</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aeb90acdbda2b036b177dc3053645a32/href">https://medium.com/media/aeb90acdbda2b036b177dc3053645a32/href</a></iframe><h3>Conclusion</h3><p>Migrating Data in a live production system is a complicated task that requires a lot of attention and careful planning. I recommend taking the time and working through the steps listed above and figuring out what works best for your needs</p><p>As a rule of thumb, always try to reduce your requirements as much as possible. For example, Is zero downtime migration required? Can you afford data-loss?</p><p>Upgrading data stores is usually a marathon and not a sprint, so take a deep breath and try to enjoy the ride</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*RuRfAyawy6BUgET_O8OeYQ.gif" /></figure><ul><li>The whole process listed above took me around 4 months of work.</li><li>All of the Elasticsearch examples that appear in this post have been tested against version 6.8.1.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27340d668b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigpanda-engineering/how-to-migrate-from-elasticsearch-1-7-to-6-8-with-zero-downtime-27340d668b3">How to migrate from Elasticsearch 1.7 to 6.8 with zero downtime.</a> was originally published in <a href="https://medium.com/bigpanda-engineering">BigPanda Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cross-compiling In Scala]]></title>
            <link>https://medium.com/bigpanda-engineering/cross-compiling-in-scala-bb491682ffcf?source=rss-827c327d5d0c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb491682ffcf</guid>
            <category><![CDATA[cross-compile]]></category>
            <category><![CDATA[bts]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Dor Sever]]></dc:creator>
            <pubDate>Sun, 21 Jul 2019 17:52:38 GMT</pubDate>
            <atom:updated>2019-07-21T18:34:26.318Z</atom:updated>
            <content:encoded><![CDATA[<p>This feature comes up when talking about upgrading Scala versions. In this post, I will try to explain what it means and why should we use it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*2IHtmdq-1fbRostB0nXptg.jpeg" /></figure><h3>Overview</h3><p>The goal of this post is to help you update your application to the latest scala version (2.12, 2.13 and counting), And by the end of this post you will know how to answer the following questions :</p><ul><li>Updating scala version — What’s the catch?</li><li>How to cross-compile my libraries?</li><li>How to update the Scala version in my application?</li></ul><h4>Updating Scala version — What’s the catch?</h4><p>My first task in <a href="https://medium.com/u/3a9656046a20">BigPanda</a> was to upgrade our Scala stack from 2.11 to the (already quite old now) 2.12 version.</p><p>As I learned in this task, Scala is a version sensitive language, which means that code breaks between versions and one cannot use a 2.11 library in a 2.12 application.</p><p>Even more, I found that most of the artifacts we define in our build.sbtare defined using the %% syntax like this</p><pre>groupID %% artifactID % revision</pre><p>which translates to :</p><pre>groupID % artifactID_scalaVersion % revision</pre><p>This <a href="https://www.scala-sbt.org/release/docs/Library-Dependencies.html">means</a>, that our application requires that all it’s dependencies to be published in the same Scala version that it’s trying to use.</p><p>This process of updating to a new Scala version takes some time for libraries(see the famous spark 2.12 support <a href="https://issues.apache.org/jira/browse/SPARK-14220">here</a>) but the real problem lies in the in house dependencies which we have to update first.</p><h4>How to cross-compile my libraries?</h4><p>My initial and naive approach was: let’s upgrade our libraries to 2.12 and update all applications to 2.12 also, but it did not work.</p><p>The reason for the failure was in the numbers: in <a href="https://medium.com/u/3a9656046a20">BigPanda</a>, we’ve got about 10 scala libraries and 10 complex applications that consume them. Migrating all of them at once is an insanely complex and risky operation that requires complex roll-back strategies and a large amount of time which we did not have.</p><p>The more practical way of doing this is by compiling our libraries to both 2.11, 2.12 using a feature called <strong>cross-compilation</strong> available in SBT.</p><p>It’s as simple as adding this line in your build.sbt</p><pre>crossScalaVersions := Seq(“2.11.11”, “2.12.7”)</pre><p>And changing your build\publish command from</p><pre>sbt ++2.11.11 publish</pre><p>to</p><pre>sbt +publish</pre><p>This tells SBT, to publish it twice using the versions in crossScalaVersions . More info can be found <a href="https://www.scala-sbt.org/1.x/docs/Cross-Build.html">here</a>.</p><h4>How to update the Scala version in my application?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*cVGueC5riOfClYWVByN_GA.gif" /></figure><p>Now, this is what we really wanted to do. We choose the least critical service in our pipeline and decided to start there and iterate.</p><p>The process is as simple as writing this in your build.sbt file</p><pre><em>scalaVersion </em>:= &quot;2.12.7&quot;</pre><p>And then making sure all regression tests pass (which is often not as simple).</p><p>After we deployed that service into production without any glitches, we felt much more confident about updating the rest of our services.</p><h4>The happy ending</h4><p>To be honest, it has been almost a year since we have done this task. we are currently at 9\10 applications migrated to 2.12 and we did not yet reach the part where we delete the cross-compilation into 2.11. It’s not an optimal ending, but a happy one in my perspective.</p><h3>Conclusion</h3><ul><li>Don’t be afraid to update your Scala version!</li><li>Cross-compiling in SBT is a super cool feature that works out of the box and provides the ability to update your libraries to a new version while maintaining backward compatibility.</li><li>Hope for the best, plan for the worst. Migrating all your services and libraries at once is a complex job with a lot of uncertainties.</li><li>I recommend cross-compiling your libraries first to the required version and then migrating the applications one at a time in an iterative manner.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb491682ffcf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigpanda-engineering/cross-compiling-in-scala-bb491682ffcf">Cross-compiling In Scala</a> was originally published in <a href="https://medium.com/bigpanda-engineering">BigPanda Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Becoming a better functional programmer]]></title>
            <link>https://medium.com/@dorsever/becoming-a-better-functional-programmer-4e32ad6252c8?source=rss-827c327d5d0c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e32ad6252c8</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[zio]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Dor Sever]]></dc:creator>
            <pubDate>Thu, 11 Jul 2019 09:04:42 GMT</pubDate>
            <atom:updated>2019-07-11T09:07:18.973Z</atom:updated>
            <content:encoded><![CDATA[<h3>Motivation</h3><p>I originally was introduced to FP at work by a colleague and then my curiosity was picked and I started to dig deeper, starting with the excellent <a href="https://www.manning.com/books/functional-programming-in-scala">red book</a>.</p><p>Next, I started watching videos from the top scala FP people like <a href="https://github.com/tpolecat">Rob Norris</a> /<a href="https://github.com/jdegoes">John de Goes</a> /<a href="https://github.com/runarorama">Runar</a> / <a href="https://github.com/odersky">Martin Ordersky</a>/ <a href="https://github.com/alissapajer">Alissa Pajer</a>.</p><p>And each time I finished one of these videos, I asked myself how can I reach this level of mastery.</p><p>And the obvious way to do that is to work directly with them.</p><p>Luckily for us, this is achievable since all of them are contributing to open source software.</p><h3>Why ZIO?</h3><p>It is notoriously known that scala FP libraries are hard to understand and even harder to contribute to. It feels like the barrier to entry is extremely high and almost impossible to jump.</p><p>The <a href="https://zio.dev/">#ZIO</a> project has a different approach to welcoming new contributors and it puts an emphasis on guiding them in their first steps.</p><p>The proof can be found in the ZIO <a href="https://github.com/zio/zio/blob/master/CONTRIBUTING.md">contributing guide</a> :</p><blockquote><em>You too can contribute to ZIO, we believe in you!</em></blockquote><h3>Becoming a ZIO contributor</h3><p>It’s very easy to start. just go to the <a href="https://github.com/zio/zio/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">issues page</a> and filter by ‘good first issue’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5_d190Ft0hyvwXVgddVC9w.png" /></figure><p>It’s important to recognize that not all issues were born equal. In my opinion, the easiest ones to start with are refactoring or test-related tasks.</p><p>Once you have chosen a task, the next thing to do is to take a leap of faith and grab the issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-dHuNU1hLZl6FWM2tDOP6w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*9t_dmqpIFSRranZcZMh44w.jpeg" /></figure><p>Now comes the fun part, start writing code.</p><p>You will get stuck but no worries, that’s where the learning happens.</p><p>To get unstuck, ask a question in the issue page and then something magical happens. <strong>you get reviewed by people you&#39;ve only seen on scala-days videos.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8u_BEbn4FiwmR2q2DLH20g.png" /></figure><p>You can make this task even more fun by making it a collaborative effort with your colleagues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RxeM_rlr5kZ0oTglXBL9dQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNhpat9yzTrS1ueHw1xPVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2RxKIqmRQ0IzvZ_BhLATFg.png" /></figure><h3>Victory — PR merged to master</h3><p>After a few days of hard work, that’s how victory looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TSh8L1guoMilvN_x-Le7Ig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DEcjIBga4pbTGjNTtM9VzQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f_7UT_QHWziM54nVyQDgEQ.png" /></figure><h3>Summary</h3><ol><li>Contributing to open source software will make you a better functional programmer.</li><li>The <a href="https://zio.dev/">#ZIO</a> project is extremely welcoming to newbies and I encourage you to check it out.</li><li>Involving your teammates will make the experience much more fun and rewarding.</li></ol><h3>Credits</h3><p>This post was co-authored with my colleague <a href="https://medium.com/@daniel.sebban">Daniel Sebban</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e32ad6252c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Killing the unit test]]></title>
            <link>https://medium.com/@dorsever/killing-the-unit-test-235c0eddd029?source=rss-827c327d5d0c------2</link>
            <guid isPermaLink="false">https://medium.com/p/235c0eddd029</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[unittest]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Dor Sever]]></dc:creator>
            <pubDate>Tue, 25 Jun 2019 07:47:21 GMT</pubDate>
            <atom:updated>2019-06-25T07:47:21.068Z</atom:updated>
            <content:encoded><![CDATA[<h3>Killing the Unit Test</h3><p>Yesterday I spoke at the scala underscore meetup in Tel Aviv and had so much fun :) <br>The video has just been published and I’m excited to share with you.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXtuv_aduYjM%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXtuv_aduYjM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXtuv_aduYjM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4c4e051fb15823edc1fa5e44f3a4414/href">https://medium.com/media/a4c4e051fb15823edc1fa5e44f3a4414/href</a></iframe><p>There is no doubt that Unit tests are an essential part of the development process today.<br>However, writing unit tests is a tedious and repetitive job that is often unrewarding since bugs slip away.<br>Property-based testing addresses these issues by providing a way to define laws and properties on your data types and generate tests to verify them.</p><p>In this talk, I’ll explain what Property-based testing and Law testing are, and share some concrete examples from the work we do in <a href="https://medium.com/u/3a9656046a20">BigPanda</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QGyBbf1QvmegNS5zXdpu6A.png" /></figure><p>Slides are available here:<br><a href="https://www.slideshare.net/dorsever1/killing-the-unit-test-talk-151464773">https://www.slideshare.net/dorsever1/killing-the-unit-test-talk-151464773</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=235c0eddd029" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An introduction to Law Testing in Scala]]></title>
            <link>https://medium.com/free-code-camp/an-introduction-to-law-testing-in-scala-4243d72272f9?source=rss-827c327d5d0c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4243d72272f9</guid>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Dor Sever]]></dc:creator>
            <pubDate>Mon, 22 Apr 2019 16:48:22 GMT</pubDate>
            <atom:updated>2019-05-13T07:24:40.835Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*HzzqlUARmxESyEDddvfxaw.gif" /></figure><p>Property-based law testing is one of the most powerful tools in the scala ecosystem. In this post, I’ll explain how to use law testing and the value it’ll give you using in-depth code examples.</p><p>This post is aimed for Scala developers who want to improve their testing knowledge and skills. It assumes some knowledge of Scala, cats, and other functional libraries.</p><h4><strong>Introduction</strong></h4><ul><li>You might be familiar with types which are a set of values (for example Int values are : 1,2,3… String values are : “John Doe” etc).</li><li>You also might be familiar with functions which is a mapping from Input type to Output type.</li><li>A property is defined on a type or a function and describes its desired behavior.</li></ul><p>So what is a Law? <strong>Keep on reading!</strong></p><h4><strong>A concrete example</strong></h4><p>Here is our beloved <em>Person</em> data type:</p><pre><strong>case class </strong>Person(name: String, age: Int)</pre><p>And serialization code using the Play-Json , a library that allows transforming your Person type into JSON :</p><pre><strong>val </strong><em>personFormat: </em>OFormat[Person]<em> = </em><strong>new </strong>OFormat[Person] {<br>  <strong>override def </strong>reads(json: JsValue): JsResult[Person] = {<br>    <strong>val </strong>name = (json \ &quot;name&quot;).as[String]<br>    <strong>val </strong>age = (json \ &quot;age&quot;).as[Int]<br>    <em>JsSuccess</em>(<em>Person</em>(name, age))<br>  }</pre><pre><strong>override def </strong>writes(o: Person): JsObject =<br>    <em>JsObject</em>(<em>Seq</em>(&quot;name&quot; -&gt; <em>JsString</em>(o.name), <br>                 &quot;age&quot; -&gt; <em>JsNumber</em>(o.age)))<br>}</pre><p>We can now test this serialization function on a specific input like this:</p><pre><strong>import </strong>org.scalatest._</pre><pre><strong>class </strong>PersonSerdeSpec <strong>extends </strong>WordSpecLike <strong>with </strong>Matchers {<br>  &quot;should serialize and deserialize a person&quot; in {<br>    <strong>val </strong>person = <em>Person</em>(&quot;John Doe&quot;, 32)<br>    <strong>val </strong>actualPerson =<br>      <em>personFormat</em>.reads(<em>personFormat</em>.writes(person))<br>    actualPerson.asOpt.shouldEqual(<em>Some</em>(person))<br>  }<br>}</pre><p>But, we now need to ask ourselves, whether all people will serialize successfully? What about a person with invalid data (such as negative age)? Will we want to repeat this thought process of finding edge-cases for all our test data?</p><p>And most importantly, will this code remain readable over time? (e.g.: changing the person data type [adding a LastName field], repeated tests for other data types, etc)</p><blockquote>“ We can solve any problem by introducing an extra level of indirection”.</blockquote><h3>Property-based testing</h3><p>The first weapon in our disposal is Property-based testing (PBT). PBT works by defining a property, which is a high-level specification of behavior that should hold for all values of the specific type.</p><p>In our example, the <strong>property</strong> will be:</p><ul><li>For every person p, if we serialize and deserialize them, we should get back the same person.</li></ul><p>Writing this property using scala check looks like this:</p><pre><strong>object </strong>PersonSerdeSpec <strong>extends </strong>org.scalacheck.Properties(&quot;PersonCodec&quot;) {<br><em>  property</em>(&quot;round trip consistency&quot;) = <br>org.scalacheck.Prop.<em>forAll </em>{ a: Person =&gt;<br>    <em>personFormat</em>.reads(<em>personFormat</em>.writes(a)).asOpt.contains(a)<br>  }<br>}</pre><p>The property check requires a way to generate Persons. This is done by using an Arbitrary[Person] which can be defined like this:</p><pre><strong>implicit val </strong><em>personArb</em>: Arbitrary[Person] = Arbitrary {<br>  <strong>for </strong>{<br>    name &lt;- Gen.<em>alphaStr<br>    </em>age  &lt;- Gen.<em>chooseNum</em>(0, 120)<br>  } <strong>yield </strong><em>Person</em>(name, age)<br>}</pre><p>Furthermore, we can use “scalacheck-shapeless”- an amazing library which eliminates (almost) all needs for the verbose (quite messy and highly bug-prone) arbitrary type definition by generating it for us!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*ljTFIMmjzU5_IxkLssllAA.gif" /></figure><p>This can be done by adding:</p><pre>libraryDependencies += &quot;com.github.alexarchambault&quot; %% &quot;scalacheck-shapeless_1.14&quot; % &quot;1.2.0&quot;</pre><p>and importing the following in our code:</p><pre><strong>import </strong>org.scalacheck.ScalacheckShapeless._</pre><p>And then we can remove the <em>personArb </em>instance<em> </em>we defined earlier.</p><h3>The Codec Law</h3><p>Let’s try to abstract further, by defining the <strong>laws</strong> of our data type:</p><pre><strong>trait </strong>CodecLaws[A, B] {<br>  <strong>def </strong>serialize: A =&gt; B<br>  <strong>def </strong>deserialize: B =&gt; A<br>  <strong>def </strong>codecRoundTrip(a: A): Boolean = serialize.<br>andThen(deserialize)(a) == a<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*59G7n15irY1CNdjuhLyRNg.png" /></figure><p>This means That given</p><ul><li>The types A, B</li><li>A function from A to B</li><li>A function from B to A</li></ul><p>We define a function called “codecRoundTrip” which takes an “a: A” and takes it through the functions and makes sure we get the same value of type A back.</p><p>This Law states (without giving away<strong> any implementation details</strong>), that the roundtrip we do on the given input does not “lose” any information.</p><blockquote>Another way of saying just that is by claiming that our types A and B are isomorphic.</blockquote><p>We can abstract even more, by using the cats-laws library with the IsEq case-class for defining an Equality description.</p><pre><strong>import </strong>cats.laws._<br><strong>trait </strong>CodecLaws[A, B] {<br>  <strong>def </strong>serialize: A =&gt; B<br>  <strong>def </strong>deserialize: B =&gt; A<br>  <strong>def </strong>codecRoundTrip(a: A): cats.laws.IsEq[A] = serialize.andThen(deserialize)(a) &lt;-&gt; a<br>}</pre><pre><em>/** Represents two values of the same type that are expected to be equal. */<br></em><strong>final case class </strong>IsEq[A](lhs: A, rhs: A)</pre><p>What we get from this type and syntax is a <strong>description</strong> <strong>of equality</strong> between the two values instead of the equality result like before.</p><h3>The Codec Test</h3><p>It is time to test the laws we just defined. In order to do that, we will use the “<a href="https://github.com/typelevel/discipline">discipline</a>” library.</p><pre><strong>import </strong>cats.laws.discipline._</pre><pre><strong>import </strong>org.scalacheck.{ Arbitrary, Prop }</pre><pre><strong>trait </strong>CodecTests[A, B] <strong>extends </strong>org.typelevel.discipline.Laws {<br>  <strong>def </strong>laws: CodecLaws[A, B]<br>  <strong>def </strong>tests(<br>    <strong>implicit<br>    </strong>arbitrary: Arbitrary[A],<br>    eqA: cats.Eq[A]<br>  ): RuleSet =<br>    <strong>new </strong>DefaultRuleSet(<br>      name   = name,<br>      parent = None,<br>      &quot;roundTrip&quot; -&gt; Prop.<em>forAll </em>{ a: A =&gt;<br>        laws.codecRoundTrip(a)<br>      }<br>    )<br>}</pre><p>We define a CodecTest trait that takes 2 type parameters A and B, which in our example will be Person and JsResult.</p><p>The trait holds an instance of the laws and defines a test method that takes an Arbitrary[A] and an equality checker (of type Eq[A]) and returns a rule-set for scalacheck to run.</p><p>Note that no tests actually run here. This gives us the power to run these tests which are defined just once for all the types we want</p><p>We can now commit to a specific type and implementation (like Play-Json serialization) by instantiating a CodecTest with the proper types.</p><pre><strong>object </strong>JsonCodecTests {<br>  <strong>def </strong>apply[A: Arbitrary](<strong>implicit </strong>format: Format[A]): CodecTests[A, JsValue] =<br>    <strong>new </strong>CodecTests[A, JsValue] {<br>      <strong>override def </strong>laws: CodecLaws[A, JsValue] =<br>        CodecLaws[A, JsValue](format.reads, format.writes)<br>    }<br>}</pre><h4>A (type) detour</h4><p>But now we get the error:</p><pre>Error:(11, 38) type mismatch;<br> found   : play.api.libs.json.<strong>JsResult</strong>[A]<br> required: A</pre><p>We expected the types to flow from:</p><pre>  A  =&gt;  B  <strong>=&gt;  A</strong></pre><p>But Play-Json types go from:</p><pre> A  =&gt;  JsValue  <strong>=&gt;  JsResult[A]</strong></pre><p>This means that our deserialize function can succeed or fail and will not always return an A, but rather a container of A.</p><p>In order to abstract over the types, we now need to use the F[_] type constructor syntax:</p><pre><strong>trait </strong>CodecLaws[<strong>F[_],</strong>A, B] {<br>  <strong>def </strong>serialize: A =&gt; B<br>  <strong>def </strong>deserialize: <strong>B =&gt; F[A]</strong><br>  <strong>def </strong>codecRoundTrip(a: A)(implicit<strong> </strong>app:<strong>Applicative[F]</strong>): IsEq[<strong>F[A]</strong>] =<br>    serialize.andThen(deserialize)(a) &lt;-&gt; <strong>app.pure(a)</strong><br>}</pre><p>The Applicative instance is used to take a simple value of type A and lift it into the Applicative context which returns a value of type F[A].</p><blockquote>This process is similar to taking some value x and lifting it to an Option context using Some(x), or in our concrete example taking a value a:A and lifting it to the JsResult type using <a href="https://www.playframework.com/documentation/2.7.0/api/scala/play/api/libs/json/JsSuccess.html">JsSuccess</a>(a).</blockquote><p>We can now finish the implementation for CodecTests and JsonCodecTests like this:</p><pre><strong>trait </strong>CodecTests[F[_], A, B] <strong>extends </strong>org.typelevel.discipline.Laws {<br>  <strong>def </strong>laws: CodecLaws[F, A, B]<br>  <strong>def </strong>tests(<br>    <strong>implicit<br>    </strong>arbitrary: Arbitrary[A],<br>    eqA: cats.Eq[F[A]],<br>    applicative: Applicative[F]<br>  ): RuleSet =<br>    <strong>new </strong>DefaultRuleSet(<br>      name   = name,<br>      parent = None,<br>      &quot;roundTrip&quot; -&gt; Prop.<em>forAll </em>{ a: A =&gt;<br>        laws.codecRoundTrip(a)<br>      }<br>    )<br>}</pre><pre><strong>object </strong>JsonCodecTests {<br>  <strong>def </strong>apply[A: Arbitrary](<strong>implicit </strong>format: Format[A]): CodecTests[JsResult, A, JsValue] =<br>    <strong>new </strong>CodecTests[JsResult, A, JsValue] {<br>      <strong>override def </strong>laws: CodecLaws[JsResult, A, JsValue] =<br>        <em>CodecLaws</em>[JsResult, A, JsValue](format.reads, format.writes)<br>    }<br>}</pre><p>And to define a working Person serialization test in <strong>1 line of code:</strong></p><pre><strong>import </strong>JsonCodecSpec.Person<br><strong>import </strong>play.api.libs.json._<br><strong>import </strong>org.scalacheck.ScalacheckShapeless._<br><strong>import </strong>org.scalatest.FunSuiteLike<br><strong>import </strong>org.scalatest.prop.Checkers<br><strong>import </strong>org.typelevel.discipline.scalatest.Discipline</pre><pre><strong>class </strong>JsonCodecSpec <strong>extends </strong>Checkers <strong>with </strong>FunSuiteLike <strong>with </strong>Discipline { <br>  checkAll(&quot;PersonSerdeTests&quot;, <em>Json</em>Codec<em>Tests</em>[Person].tests)<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*N3Np8qbCeN1y-6B_ui9EbA.gif" /></figure><h3>The power of abstraction</h3><p>We were able to define our tests and laws without giving away any implementation details. This means we can switch to using a different library for serialization tomorrow and all our laws and tests will still hold.</p><h4>Another example</h4><p>We can test this theory by adding support to BSON serialization using the reactive-mongo library:</p><pre><strong>import </strong>cats.Id<br><strong>import </strong>io.bigpanda.laws.serde.{ CodecLaws, CodecTests }<br><strong>import </strong>org.scalacheck.Arbitrary<br><strong>import </strong>reactivemongo.bson.{ BSONDocument, BSONReader, BSONWriter }</pre><pre><strong>object </strong>BsonCodecTests {<br>  <strong>def </strong>apply[A: Arbitrary](<br>    <strong>implicit<br>    </strong>reader: BSONReader[BSONDocument, A],<br>    writer: BSONWriter[A, BSONDocument]<br>  ): CodecTests[Id, A, BSONDocument] =<br>    <strong>new </strong>CodecTests[Id, A, BSONDocument] {<br>      <strong>override def </strong>laws: CodecLaws[Id, A, BSONDocument] =<br>        <em>CodecLaws</em>[Id, A, BSONDocument](reader.read, writer.write)</pre><pre><strong>override def </strong>name: String = &quot;BSON serde tests&quot;<br>    }<br>}</pre><p>The types here flow from</p><pre>A =&gt; BsonDocument =&gt; A</pre><p>and not F[A] as we had expected. Luckily for us, we have a solution and use the Id-type to represent just that.</p><p>And given the (very long) serializer definition:</p><pre><strong>implicit val </strong><em>personBsonFormat<br>  </em>: BSONReader[BSONDocument, Person] <strong>with </strong>BSONWriter[Person, BSONDocument] =<br>  <strong>new </strong>BSONReader[BSONDocument, Person] <strong>with </strong>BSONWriter[Person, BSONDocument] {<br>    <strong>override def </strong>read(bson: BSONDocument): Person =<br>      <em>Person</em>(bson.getAs[String](&quot;name&quot;).get, bson.getAs[Int](&quot;age&quot;).get)</pre><pre><strong>override def </strong>write(t: Person): BSONDocument =<br>      <em>BSONDocument</em>(&quot;name&quot; -&gt; t.name, &quot;age&quot; -&gt; t.age)<br>  }</pre><p>we can now define BsonCodecTests in all its <strong>1</strong> line of logic glory.</p><pre><strong>class </strong>BsonCodecSpec <strong>extends </strong>Checkers <strong>with </strong>FunSuiteLike <strong>with </strong>Discipline {<br>    checkAll(&quot;PersonSerdeTests&quot;, <em>BsonCodecTests</em>[Person].tests)<br>}</pre><h3>A (First-order) logic perspective on our code</h3><p>Our first test attempt can be described as follows:</p><pre>∃p:Person,s:OFormat that holds : s.read(s.write(p)) &lt;-&gt; p</pre><p>Meaning, for the specific person p(“John Doe”,32) and for the format <strong>s</strong>, the following statement is true: decode(encode(p)) &lt;-&gt; p.</p><p>The second attempt (using PBT) can be:</p><pre>∃s:OFormat, ∀p:Person the following should hold :  s.read(s.write(p)) &lt;-&gt; p</pre><p>Which means, <strong>for all </strong>persons p and for the specific format <strong>s</strong>, the following is true: decode(encode(p))&lt;-&gt;p.</p><p>The third (and most powerful statement thus far) using law testing:</p><pre>∀s:Encoder, ∀p:Person the the following should hold :  s.read(s.write(p)) &lt;-&gt; p</pre><p>Which means,<strong> </strong><strong>for all </strong>formats s, and <strong>for all</strong> persons p, the following is true: decode(encode(p))&lt;-&gt;p.</p><h4><strong>Summary</strong></h4><ul><li>Law testing allows you to reason about your data-types and functions in a mathematical and concise way and provides a totally new paradigm for testing your code!</li><li>Most of the type level libraries you use (like cats, circe and many more) use law testing internally to test their data-types.</li><li>Avoid writing specific test-cases for your data-types and functions and try to generalize them using property-based law tests.</li></ul><p>Thank you for reaching this far! I am super excited about finding more abstract and useful laws that I can use in my code! Please let me know about any you’ve used or can think of.</p><p>More inspiring and detailed content can be found in the <a href="https://typelevel.org/cats/typeclasses/lawtesting.html">cats-laws</a> site or <a href="https://github.com/circe/circe/blob/master/modules/testing/shared/src/main/scala/io/circe/testing/CodecTests.scala">circe</a>.</p><p>The complete code examples can be found <a href="https://gist.github.com/dorsev/0fdd8315228d7ef6914b27650f817ae6">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4243d72272f9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/free-code-camp/an-introduction-to-law-testing-in-scala-4243d72272f9">An introduction to Law Testing in Scala</a> was originally published in <a href="https://medium.com/free-code-camp">We’ve moved to freeCodeCamp.org/news</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>