<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Travis Brown on Medium]]></title>
        <description><![CDATA[Stories by Travis Brown on Medium]]></description>
        <link>https://medium.com/@travisrobertbrown?source=rss-2ae783c1ef98------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*icmxMSYkE9YtfIKj.jpg</url>
            <title>Stories by Travis Brown on Medium</title>
            <link>https://medium.com/@travisrobertbrown?source=rss-2ae783c1ef98------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:52:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@travisrobertbrown/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Community engineering report]]></title>
            <link>https://medium.com/permutive/community-engineering-report-df9ffbc53c1d?source=rss-2ae783c1ef98------2</link>
            <guid isPermaLink="false">https://medium.com/p/df9ffbc53c1d</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[dhall]]></category>
            <dc:creator><![CDATA[Travis Brown]]></dc:creator>
            <pubDate>Tue, 07 Apr 2020 10:35:41 GMT</pubDate>
            <atom:updated>2020-04-07T13:51:15.748Z</atom:updated>
            <content:encoded><![CDATA[<h3>Permutive Community Engineering, March 2020</h3><p>The highlight of March for the Community Engineering team was the release of the first Cats 2.2.0 milestone, which introduces a major change that we’ve been working towards since last September. We also published new releases of some other Typelevel and Circe libraries, moved a couple of the libraries we use a little closer to Dotty cross-building, and began work on a new project that we’re planning to launch this week.</p><h3>Cats 2.2.0-M1</h3><p>The <a href="https://github.com/typelevel/cats">Cats</a> library has always provided type class instances for Scala standard library types as “orphan” instances that require imports in order to be used. Scalaz did the same before it (at least back to 2011, when this change was introduced, for reasons nobody seems to remember or understand exactly). This approach works against the grain of the language, which allows authors of type classes to put instances for types from their dependencies (including the standard library) into “implicit scope”, which doesn’t require imports. The new <a href="https://github.com/typelevel/cats/releases/tag/v2.2.0-M1">Cats 2.2.0-M1</a> release changes the library to take advantage of this language feature (without breaking binary compatibility with previous releases).</p><p>Putting type class instances into implicit scope has a number of advantages. It simplifies the experience of using the library, since you never have to remember to add imports to bring type class instances into scope. It can also significantly improve compile times for users of the library, because it avoids cluttering up lexical scope. In previous releases of Cats, idiomatic usage involved putting over 650 implicit definitions into lexical scope, each of which then had to be considered for <em>every</em> implicit search. In Cats 2.2.0 this number is under 200 (for extension methods), with the rest of the implicit definitions living in implicit scope, where the compiler can resolve instances more efficiently. Please see <a href="https://meta.plasm.us/posts/2019/09/30/implicit-scope-and-cats/">this blog post</a> about my initial experiments in this direction for some concrete examples of compile time improvements.</p><p>In addition to the implicit scope change, the 2.2.0-M1 release introduces many new convenience methods, optimizations, and improvements, with 85 pull requests being merged since 2.1.0 in December. These changes represent a lot of work by the Permutive Community engineering team over the past couple of months, together with the 28 other human contributors (including 14 first-time contributors), and one robot:</p><pre>     77 Travis Brown<br>     19 Scala Steward<br>      9 Daniel Spiewak<br>      8 Frank S. Thomas<br>      3 Lars Hupel<br>      3 Gagandeep kalra<br>      2 Vitaly Lavrov<br>      2 mooi<br>      2 Mateusz Wójcik<br>      2 Kai(luo) Wang<br>      2 Jakub Kozłowski<br>      2 Diego E. Alonso-Blas<br>      1 Yannick Heiber<br>      1 Vasilis Nicolaou<br>      1 Shan Sikdar<br>      1 Sanjiv Sahayam<br>      1 Ross A. Baker<br>      1 P. Oscar Boykin<br>      1 Peter Perhac<br>      1 MaT1g3R<br>      1 Markus Appel<br>      1 Luka Jacobowitz<br>      1 Luis Miguel Mejía Suárez<br>      1 Luís Campos<br>      1 Leandro<br>      1 kenji yoshida<br>      1 Filippo Mariotti<br>      1 enzief<br>      1 dantb<br>      1 Brian Wignall</pre><p>Thanks to everyone who contributed to this release, and to everyone who’s helping to test it out in preparation for the upcoming 2.2.0 final release.</p><h3>Dotty cross-building</h3><p>Most of my work on Dotty compatibility in March was focused on <a href="https://http4s.org/">http4s</a>, a framework for building HTTP services that we use extensively at Permutive. The biggest challenge here was that http4s depends on a fork <a href="https://github.com/sirthias/parboiled2">of parboiled2</a>, a parser combinator library that relies heavily on Scala 2’s macro system. I’d never worked with Dotty’s new macro system before, but this seemed like a good excuse, and I ported enough of parboiled2 to make it possible to identify the other changes that were needed to get http4s building on Dotty, <a href="https://github.com/http4s/http4s/pull/3242">most of which</a> are now merged. It’s <a href="https://github.com/sirthias/parboiled2/issues/210">still not clear</a> what the future of parboiled2 will look like on Dotty, but it’s likely we’ll wrap up the port in April, if only for use in http4s’s fork.</p><h3>Other releases</h3><p>In February I <a href="https://github.com/typelevel/general/issues/100">nominated Simulacrum Scalafix</a> for Typelevel membership, and in early March this proposal was approved, so I made a couple of fixes and improvements that had been suggested and published the new <a href="https://github.com/typelevel/simulacrum-scalafix/releases/tag/v0.1.0">first release</a> in the Typelevel organization.</p><p>I also worked with a couple of external contributors to add some new features to <a href="https://github.com/circe/circe-derivation">circe-derivation</a>, including <a href="https://github.com/circe/circe-derivation/releases/tag/v0.13.0-M3">support for transforming constructor names</a> in derived encoder and decoder instances, which brings circe-derivation essentially to feature parity with circe-generic.</p><p>I also published two new <a href="https://github.com/circe/circe-golden">circe-golden</a> releases, which fix some issues that we’d run into when using the library in tests for internal services.</p><h3>Dhall for Java</h3><p>The final project that I worked on in March wasn’t a Scala library, for a change. Like many companies, Permutive uses <a href="https://kubernetes.io/">Kubernetes</a> for container management, and we have many thousands of lines of YAML configuration that make this possible. YAML’s problems as a configuration language are <a href="https://www.arp242.net/yaml-config.html">well-known</a>: it’s complicated, unintuitive, insecure, and error-prone. There are many alternatives to YAML, but one that we’re particularly interested in is <a href="https://dhall-lang.org/">Dhall</a>, which is a minimal language with a clear specification, an <a href="https://discourse.dhall-lang.org/">active community</a>, and a programming model that fits in well at Permutive, where we use functional languages like Haskell, Elm, and Scala extensively.</p><p>One of the disadvantages of Dhall for us is that while there are implementations in <a href="https://github.com/dhall-lang/dhall-haskell">Haskell</a>, <a href="https://git.sr.ht/~singpolyma/dhall-ruby">Ruby</a>, <a href="https://github.com/Nadrieril/dhall-rust">Rust</a>, <a href="https://github.com/philandstuff/dhall-golang">Go</a>, etc., the integration with the JVM ecosystem is less clear. There’s a <a href="https://github.com/f-f/dhall-clj">Clojure implementation</a>, but it’s slightly out-of-date (version 5.0.0 vs. the current 15.0.0) and missing some pieces. There’s also a <a href="https://github.com/eta-lang/dhall-eta">Java wrapper</a> for the Haskell implementation via <a href="https://eta-lang.org/">Eta</a>, but it also hasn’t been updated recently (it’s on version 4.0.0), isn’t published to Maven Central, and has a build process that’s somewhat complex.</p><p>Early last year I started working on a Java implementation, but didn’t get much beyond parsing. I chose Java instead of Scala because I wanted to support build tooling use cases where a dependency on Scala would not be convenient (and also because I thought it would be a fun challenge to implement a programming language type checker and interpreter in Java).</p><p>Last month we decided to reboot that project, and we currently have a zero-dependency, Java 7-compatible, almost complete implementation of parsing, normalization, CBOR encoding, and type checking, all in a 260K jar. My colleague <a href="https://medium.com/@timothywspence">Tim Spence</a> has also built a reference implementation of import resolution in Scala that makes it possible to load dependencies in your Dhall documents—including remote ones. We also have a Scala wrapper library that works a lot like <a href="https://github.com/circe/circe">Circe</a>, allowing you to map between Dhall and Scala types, but which additionally has the ability to decode Dhall expressions into Scala functions.</p><p>This is still a work in progress, but we’re now able to evaluate non-trivial programs that use functions from the Dhall standard prelude, and at the moment 1,045 of the 1,047 <a href="https://github.com/dhall-lang/dhall-lang/tree/master/tests">acceptance tests</a> we’re running are passing. We’re planning to finish up the remaining pieces, get this number to 100%, and publish the project in the next few days.</p><h3>Other open source tasks and projects</h3><ul><li>Created and published <a href="https://github.com/travisbrown/sbt-javacc">an sbt plugin</a> for the <a href="https://javacc.github.io/javacc/">JavaCC</a> parser generator.</li><li>Proposed <a href="https://github.com/typelevel/cats/pull/3354">a Cats policy change</a> to help with 2.11 backporting.</li><li><a href="https://github.com/typelevel/general/issues/101">Submitted Catbird</a> for Typelevel membership.</li><li><a href="https://github.com/scalameta/mdoc/pull/305">Minimized a bug</a> in mdoc.</li><li>Fixed <a href="https://github.com/http4s/http4s/pull/3264">a minor issue</a> with an unnecessary constraint in http4s.</li><li>Simplified <a href="https://github.com/http4s/http4s/pull/3241">http4s’s use of macros</a> a little.</li><li>Fixed <a href="https://github.com/circe/circe/pull/1416">an issue</a> with a leaking test dependency in Circe.</li><li>Fixed (partially) <a href="https://github.com/typelevel/cats/pull/3361">an issue</a> with an unsafe method in Cats.</li></ul><h3>April preview</h3><ul><li>New Cats 2.2.0 milestones and the 2.2.0 release.</li><li>First release of our Dhall implementation for Java and Scala.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*jjOsMLIg-V6FZCSHgEkifw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df9ffbc53c1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/permutive/community-engineering-report-df9ffbc53c1d">Community engineering report</a> was originally published in <a href="https://medium.com/permutive">Permutive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Community engineering report]]></title>
            <link>https://medium.com/permutive/community-engineering-report-24ec7e04dea?source=rss-2ae783c1ef98------2</link>
            <guid isPermaLink="false">https://medium.com/p/24ec7e04dea</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[dotty]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Travis Brown]]></dc:creator>
            <pubDate>Wed, 04 Mar 2020 08:11:13 GMT</pubDate>
            <atom:updated>2020-03-04T14:10:58.653Z</atom:updated>
            <content:encoded><![CDATA[<h3>Permutive Community Engineering, February 2020</h3><p>My two primary goals for the Community Engineering team in February were to prepare for a new Cats 2.2.0 release and to get Circe’s new Dotty-powered generic derivation to a state of feature parity with circe-generic. While we didn’t publish the first Cats 2.2.0 milestone, we did merge several significant improvements that will go into it, in addition to publishing a new Circe release that introduces Dotty artifacts, kicking off a new statically-sized collections library, and helping push forward the community effort to upgrade to the new Scala.js 1.0.0 release.</p><h3>Cats 2.2.0</h3><p>One issue that we’ve run into repeatedly at Permutive is inconsistency in the APIs of the collection types that Cats provides in cats.data. The Chain type for example is <a href="https://typelevel.org/cats/datatypes/chain.html">presented as a replacement</a> for List or Vector that’s likely to perform better in cases where you’re doing a lot of concatenation, but it was missing many essential collection operations like zipWithIndex or sortBy. In other cases we’ve started replacing NonEmptyList with NonEmptyVector and immediately run into the fact that NonEmptyVector was missing groupBy and several other methods that were available on NonEmptyList.</p><p>One of the things I worked on last month was <a href="https://github.com/typelevel/cats/pull/3307">an overhaul for these types</a> and their tests that eliminates most of this inconsistency (although not all of it, unfortunately, because of the constraints of our binary compatibility guarantees). These changes (and <a href="https://github.com/typelevel/cats/pull/3323">some follow-up clean-up</a>) have been merged and will be available in 2.2.0.</p><p>Another issue in Cats that I’d run into a few times was related to a change in the way operations on BigDecimal work <a href="https://github.com/scala/bug/issues/11590">between Scala 2.12 and 2.13</a>. This change caused Cats’s CommutativeMonoid instance for BigDecimal to be neither commutative nor associative in some rare cases on Scala 2.13, which was causing our laws-checking tests to fail occasionally. I proposed a <a href="https://github.com/typelevel/cats/pull/3303">solution</a> that went through several rounds of review and is now merged, and opened a <a href="https://github.com/typelevel/cats/issues/3317">follow-up issue</a> to track a related problem that’s still unaddressed.</p><p>I also made <a href="https://github.com/typelevel/cats/pull/3304">some significant changes</a> to the way type class instances and other definitions are brought into scope in the Cats tests, with the goal of making the tests reflect standard usage more closely. During this process I ran into <a href="https://github.com/typelevel/cats/pull/3306">some inconsistencies</a> in Cats’s syntax packages, which will be fixed in 2.2.0.</p><p>I made a few other changes in Cats in preparation for 2.2 and Dotty:</p><ul><li><a href="https://github.com/typelevel/cats/pull/3283">Benchmarked and optimized</a> sometraverse implementations.</li><li>Made Cats’s new ArraySeq instances for Scala 2.13 <a href="https://github.com/typelevel/cats/pull/3302">more consistent</a>.</li><li>Fixed <a href="https://github.com/typelevel/cats/pull/3324">an issue</a> with ArraySeq instances relying on Scala 2 behavior.</li><li>Updated <a href="https://github.com/typelevel/cats/pull/3269">my Dotty branch</a> and reported a <a href="https://github.com/lampepfl/dotty/issues/8383">new bug</a> that turned up.</li><li>Updated <a href="https://github.com/typelevel/cats/pull/3193">my branch</a> standardizing syntax for polymorphic lambdas.</li><li>Updated <a href="https://github.com/typelevel/cats/pull/3043">my branch</a> with standard library instances in implicit scope.</li></ul><p>I also spent a substantial amount of time in February on code review for Cats, and we ended up <a href="https://github.com/typelevel/cats/pulls?utf8=%E2%9C%93&amp;q=is%3Apr+merged%3A2020-02-01..2020-02-29">averaging over one pull request merged</a> per day.</p><h3>Circe 0.13.0</h3><p>The <a href="https://github.com/circe/circe/releases/tag/v0.13.0">Circe 0.13.0</a> release for Scala 2 is relatively uneventful, with most of the modules (including circe-core and circe-generic) being fully binary-compatible with their 0.12.x versions. I fixed <a href="https://github.com/circe/circe/pull/1389">one minor bug</a> in the JsonCodec annotation, and we introduced <a href="https://github.com/circe/circe/pull/1350">one new method</a>, but the focus of this release series (at least on Scala 2) is on updating to the <a href="https://github.com/typelevel/jawn/releases/tag/v1.0.0">new Jawn 1.0.0 release</a> and supporting Scala.js 1.0.0.</p><p>The most exciting thing about this release is that it introduces Dotty artifacts, which include generic derivation built on Dotty’s new derives mechanism. Circe’s generic derivation on Scala 2 is currently a minefield of decisions and trade-offs. Do you use automatic (arguably convenient but slow and confusing) or semi-automatic derivation? Do you choose circe-generic (principled and clean but takes forever to compile) or <a href="https://github.com/circe/circe-derivation">circe-derivation</a>? Do you write your own custom instances or use the configurable derivation in circe-generic-extras? Dotty simplifies all of this, allowing us to provide fast and robust derivation in circe-core via the new derives Decoder syntax.</p><p>In my initial experiments derives Codec compiles about twice as fast as circe-generic’s semi-automatic derivation for some common use cases (e.g. a couple dozen case classes with different numbers of members), and only slightly slower than circe-derivation. The Shapeless-powered derivation in circe-generic also has some runtime overhead—about 10% less throughput than manually-written instances in <a href="https://github.com/circe/circe/blob/108e90865825bc414772bd1c2f47a1b5926c8162/modules/benchmark/src/main/scala/io/circe/benchmark/GenericDerivationBenchmark.scala">our current benchmarks</a> for decoding, and a little more than that for encoding. The new derivation closes this gap, thanks to Dotty’s simplified generic representation, and in our benchmarks the derived instances actually outperform Codec.forProductN.</p><h3>Statically-sized collections for Scala 2.13 and Dotty</h3><p>At Permutive we use Shapeless’s Sized to provide extra type safety in a few places where we have collections with statically-known sizes (e.g. labels associated with a metric). Using statically-sized collections has many advantages, and turns a number of common bugs (out-of-bounds indexing, providing too many or too few values) into compile-time errors. It’s a little like working with tuples, except all of the elements are required to have the same type, and you can abstract over size in useful ways (something Scala 2 has never really supported for tuples).</p><p>Shapeless’s Sized works well in many cases, but it’s starting to show its age. It uses Shapeless’s own Nat representation of type-level positive numbers, for example, instead of Scala’s native integer singleton types, which are now supported by <a href="https://scala-lang.org/files/archive/spec/2.13/03-types.html">literal types</a> in Scala 2.13 (which we use for most of our projects at Permutive). Literal singleton types have many advantages over the Nat representation: they’re cleaner to read and write, they won’t <a href="https://www.reddit.com/r/scala/comments/f89m5e/staticallysized_collections_with_dottys_inline/fik9uzb/">cause stack overflows in the compiler</a> for large values (where “large” is a few hundred), and in Dotty they’re <a href="https://github.com/lampepfl/dotty/pull/7628">supported by new type-level operations</a>.</p><p>In addition to literal singleton types, Scala 2.13 also introduced a <a href="https://www.scala-lang.org/blog/2018/06/13/scala-213-collections.html">new collections API</a>, which will be shared by Scala 3.0. Combining these two new features to create a successor to Sized seems like an obvious win, but to my knowledge it <a href="https://gitter.im/typelevel/general?at=5e527bfa0c451412668499fd">hadn’t been done</a>, so I put together a <a href="https://github.com/travisbrown/sized">proof-of-concept implementation</a> that supports Scala 2.13 and Dotty (and <a href="https://github.com/travisbrown/sized/releases/tag/v0.1.0">published it</a> to Maven Central for both).</p><p>This implementation highlights some differences in the facilities that Scala 2 and Dotty provide for encoding and tracking information like collection sizes in types. On both Scala 2 and Dotty, for example, we want our types to track the fact that if we drop three values from a collection that’s known to have five elements, we’ll end up with a collection with two elements. On Scala 2 I’ve done this with a Pair <a href="https://github.com/travisbrown/sized/blob/938ea948300ffbb60bdba06438339083f8cfd042/src/main/scala-2.13/dev/travisbrown/sized/Pair.scala">type class</a> that carries around some information about the relationship between two numbers, and that <a href="https://github.com/travisbrown/sized/blob/938ea948300ffbb60bdba06438339083f8cfd042/src/main/scala-2.13/dev/travisbrown/sized/PairMacros.scala">requires a macro</a> to generate its instances. Our size-tracking drop method then looks like this:</p><pre>def sizedDrop[X &lt;: Int](implicit<br>  X: ValueOf[X],<br>  ev: Pair.LtEq[X, N]<br>): C[A, ev.Diff]</pre><p>ValueOf here is a type class introduced in the Scala 2.13 standard library that makes the value of a singleton type available at the value level. Pair.LtEq is our own custom type class that witnesses that X is less than or equal to N (so that we don’t drop more elements than we have), and also computes their difference (at the type level).</p><p>On Dotty the method signature is completely different:</p><pre>inline def sizedDrop[X &lt;: Int]: C[A, N - X]</pre><p>Most notably there are no type-class constraints here, and we no longer have a path-dependent type (ev.Diff) in the result type.</p><p>The implementation is also very different. While Scala 2.13’s sizedDrop implementation was a one-liner that was actually much shorter than the method signature, Dotty’s is more interesting:</p><pre>inline def sizedDrop[X &lt;: Int]: C[A, N - X] =<br>  inline constValue[X] match {<br>    case x if x &lt; 0 =&gt;<br>      error(&quot;Can&#39;t drop a negative number of elements&quot;)<br>    case x =&gt; inline constValue[N] match {<br>     case n if x &lt;= n =&gt; unsafeWrap(unsizedDrop(coll, x))<br>     case _ =&gt;<br>       error(&quot;Can&#39;t drop more elements than the collection size&quot;)<br>  }<br>}</pre><p>Instead of the ValueOf type class, we now use Dotty’s new constValue to go from the type level to the value level. While on Scala 2.13 we had to do our own type-level comparisons and arithmetic (in our materialization macro for Pair) and represent the difference with a path-dependent type, Dotty’s new metaprogramming features let us write our comparisons in an ordinary pattern matching guard (if x &lt; 0 =&gt; and if x &lt;= n =&gt;), and we can do our type-level arithmetic directly on the types (N — X).</p><p>I’m excited about what this comparison suggests about the future of metaprogramming in Scala 3. While Scala 2’s macro system gave a lot of power to library developers, this power came with heavy costs, including user-hostile compiler error messages, incomprehensible and fragile code, and long compile times. The new Dotty features I’ve used in this project—inline methods, type-level singleton operations, etc.—are strictly less powerful than Scala 2’s macros, but in my view they’re a much more promising foundation for sensible library development.</p><p>If you’re interested in reading more about this project and my impressions of metaprogramming in Dotty, I’ve written a <a href="https://www.reddit.com/r/scala/comments/f89m5e/staticallysized_collections_with_dottys_inline/fimpwan/">more detailed answer to a related question on Reddit</a>, and would be happy to follow up on any other questions there.</p><h3>Scala.js 1.0.0</h3><p>The core Scala.js artifacts were <a href="https://contributors.scala-lang.org/t/this-is-not-the-release-announcement-of-scala-js-1-0-0/4021">released in early February</a>, although the <a href="https://www.scala-js.org/news/2020/02/25/announcing-scalajs-1.0.0/">official announcement</a> didn’t happen until last week. In between the release and the announcement, Scala library maintainers were encouraged to publish their libraries for 1.0.</p><p>While we don’t use Scala.js extensively at Permutive, we know that Scala.js support is important to many of the users of libraries we help to maintain, and we wanted to help make the transition from 0.6 to 1.0 as easy as possible for the community, so we updated and published a number of projects, including <a href="https://github.com/typelevel/simulacrum">Simulacrum</a>, <a href="https://github.com/typelevel/simulacrum">discipline-scalatest</a>, and <a href="https://github.com/typelevel/cats/issues/3313">Cats</a>. My initial attempt to back-publish Cats 2.1.0 for Scala.js 1.0 <a href="https://github.com/typelevel/cats/issues/3313">failed</a>, so I took some extra time to prepare and test a <a href="https://github.com/typelevel/cats/releases/tag/v2.1.1">2.1.1 release,</a> which we published last Tuesday. I also back-published <a href="https://github.com/circe/circe/releases/tag/v0.13.0">most of the modules of Circe 0.13.0 for Scala.js 1.0</a>, and am planning to update and publish the remaining modules as our last few dependencies get updated.</p><h3>Other open source tasks and projects</h3><ul><li><a href="https://github.com/typelevel/general/issues/100">Nominated Simulacrum Scalafix</a> for Typelevel membership.</li><li><a href="https://github.com/travisbrown/cats/tree/demo/dotty-derivation">Experimented with generic derivation</a> for a few cats-kernel type classes.</li><li><a href="https://github.com/lampepfl/dotty/pull/8390">Proposed specification</a> for some current behavior of Dotty’s mirrors.</li><li>Reviewed new <a href="https://medium.com/permutive/announcing-cats-stm-0-7-0-5af9a0f72920?source=friends_link&amp;sk=6605ebddd208d79495cd3cf2e9f9b6e3">Cats STM blog post</a> by my colleague <a href="https://medium.com/@timothywspence">Tim Spence</a>.</li><li>Published the <a href="https://github.com/permutive/fs2-google-pubsub/releases/tag/v0.15.0">first non-milestone release</a> of fs2-google-pubsub 0.15.</li><li>Fixed a <a href="https://github.com/scala/compiler-benchmark/pull/101">minor Dotty bug</a> in Scala’s compiler benchmarking tool.</li><li>Reported an <a href="https://github.com/lampepfl/dotty/issues/8362">issue</a> with custom compile-time error messages in Dotty.</li><li>Investigated and reported a <a href="https://github.com/milessabin/shapeless/issues/968">Shapeless bug</a> affecting Circe users.</li></ul><h3>March preview</h3><ul><li>More Dotty cross-building (in particular <a href="https://fs2.io/">fs2</a> and <a href="https://http4s.org/">http4s</a>).</li><li>Cats 2.2.0 milestone with <a href="https://github.com/typelevel/cats/pull/3043">standard library instances</a> in implicit scope.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IIn99tKKbPPMWptWwe5HBQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24ec7e04dea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/permutive/community-engineering-report-24ec7e04dea">Community engineering report</a> was originally published in <a href="https://medium.com/permutive">Permutive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Community engineering report]]></title>
            <link>https://medium.com/permutive/community-engineering-report-4cc7c5b5ce90?source=rss-2ae783c1ef98------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cc7c5b5ce90</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[dotty]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Travis Brown]]></dc:creator>
            <pubDate>Tue, 04 Feb 2020 13:13:29 GMT</pubDate>
            <atom:updated>2020-02-04T13:17:28.327Z</atom:updated>
            <content:encoded><![CDATA[<h3>Permutive Community Engineering, January 2020</h3><p>My focus in January was on getting our core dependencies (both internal and external) cross-compiling on Dotty, while reporting all of the issues I ran into along the way. We also finally got a couple of important 1.0.0 releases out the door, and did some work on lining up changes we’d like to see in Cats 2.2.0.</p><h3>Dotty cross-building</h3><p>At Permutive we’re excited about many of the new features that are coming in Scala 3, and we’re committed to investing in the migration of the open source Scala libraries we use, starting now with Dotty. In <a href="https://medium.com/permutive/community-engineering-report-ac882fa126b9">December</a> we kicked off this effort with a new <a href="https://github.com/travisbrown/simulacrum-scalafix">Scalafix-based implementation of Simulacrum</a> and a <a href="https://github.com/lampepfl/dotty/pull/7775">pull request adding kind-projector compatibility</a> to the Dotty compiler (which was merged a couple of weeks ago and is now available in the nightly releases). We continued in January by adding cross-building support to <a href="https://github.com/typelevel/jawn/pull/214">Jawn</a> and <a href="https://github.com/typelevel/discipline/pull/131">Discipline</a> and opening several pull requests for Cats:</p><ul><li><a href="https://github.com/typelevel/cats/pull/3253">Added return types</a>, <a href="https://github.com/typelevel/cats/pull/3255">parentheses</a>, etc. required by Dotty.</li><li><a href="https://github.com/typelevel/cats/pull/3254">Changed kind-projector syntax to the subset supported in Dotty</a>.</li><li><a href="https://github.com/typelevel/cats/pull/3263">Removed syntax methods in implementations in cats-core</a>.</li></ul><p>These have all been merged, and I’m now working on <a href="https://github.com/typelevel/cats/pull/3269">a larger work-in-progress PR</a> that passes tests and is almost complete, but is still blocked by several external dependencies.</p><p>It’s worth noting that by far the hardest part of adding Dotty support to Cats was the tests—this is where we had to make the most changes, and where we ran into the most compiler bugs. This isn’t surprising, since the tests are meant to reflect usage, and we’re hoping that by making this effort now we’ll be able to smooth this migration for Cats users, both by reporting (and helping to resolve) the bugs that we’ve run into, and by recording and documenting the changes we’ve had to make.</p><p>I did a similar exercise for <a href="https://github.com/typelevel/cats-effect">cats-effect</a>, another Typelevel library we use extensively at Permutive, and made a few sets of changes that have already been merged:</p><ul><li><a href="https://github.com/typelevel/cats-effect/pull/757">Adjusted kind-projector syntax, added return types</a>, etc. for Dotty.</li><li><a href="https://github.com/typelevel/cats-effect/pull/756">Restricted ArrayStack element type</a> to avoid undefined Scala 2 behavior.</li><li><a href="https://github.com/typelevel/cats-effect/pull/764">Added some helper methods</a>, etc. for laws checking.</li></ul><p>I have <a href="https://github.com/typelevel/cats-effect/pull/762">an open PR adding Dotty</a> to the cats-effect build, but it’s currently blocked by the same issues as the one for Cats. I also started working on a cross-building branch for <a href="https://github.com/functional-streams-for-scala/fs2">fs2</a> last week, but decided to wait for a <a href="https://github.com/lampepfl/dotty/pull/8082">reworking of type aliases</a> that was in progress then (and is now merged).</p><h3>Dotty bug reports</h3><p>Last month I opened 14 Dotty bug reports for issues that I ran into while setting up cross-building for these Typelevel projects and some of our internal libraries. One of these issues turned out to be a duplicate, and three were closed as won’t-fix, and of the remaining ten, seven have already been fixed:</p><ul><li><a href="https://github.com/lampepfl/dotty/issues/7959">Type checker complains about valid Java code</a>.</li><li><a href="https://github.com/lampepfl/dotty/issues/7993">Type alias causes problems for implicit search</a>.</li><li><a href="https://github.com/lampepfl/dotty/issues/8035">ClassCastException on by-name parameter in enrichment method</a>.</li><li><a href="https://github.com/lampepfl/dotty/issues/8049">Implicit conversion isn’t applied for Id type alias</a>.</li><li><a href="https://github.com/lampepfl/dotty/issues/8064">VerifyError for generic class with private constructor and self type</a>.</li><li><a href="https://github.com/lampepfl/dotty/issues/8067">Type alias results in confusing type mismatch</a>.</li><li><a href="https://github.com/lampepfl/dotty/issues/8105">Derivation crashes for case object</a>.</li></ul><p>Of the remaining three issues, only one (the one related to implicit resolution) is inconvenient to work around:</p><ul><li><a href="https://github.com/lampepfl/dotty/issues/7999">Spurious implicit argument ambiguity</a>.</li><li><a href="https://github.com/lampepfl/dotty/issues/7926">Illegal access error when constructor parameter more specific type</a>.</li><li><a href="https://github.com/lampepfl/dotty/issues/8033">Serializing val that refers to another val captures enclosing class</a>.</li></ul><p>One of our goals for this effort was to evaluate the risk of runtime errors involved in migrating to Dotty. Our speculation was that while there would likely be costs related to language changes and compile-time issues, there would be relatively little chance of runtime failures because of compiler bugs. This hypothesis wasn’t exactly supported by our experiments: half of the issues above only turned up at runtime. It’s still early days, though, with the first Scala 3 release candidate <a href="https://www.scala-lang.org/2019/12/18/road-to-scala-3.html">scheduled for the end of this year</a>, and the fact that we’re still running into runtime issues probably shouldn’t be surprising.</p><h3>Circe and new Dotty features</h3><p>Our work on most of the projects above was focused on cross-building the same code for Scala 2 and Dotty, and didn’t really explore many of the new features that Dotty offers. The one project where we tried to go beyond cross-building was <a href="https://github.com/circe/circe">Circe</a>, where I’ve added Dotty cross-building but have also put together an <a href="https://github.com/circe/circe/pull/1375">initial implementation of generic derivation</a> built on Dotty’s new type class derivation mechanism.</p><p>This implementation is still lacking some features that are available in circe-generic and <a href="https://github.com/circe/circe-derivation">circe-derivation</a>, but as an early experiment it’s quite promising—it lets us avoid both the horrible ad-hoc macros of circe-derivation and a lot of the runtime overhead of circe-generic. In February I’ll be working on extending this initial experiment to add three things:</p><ul><li>Support for the same cases as circe-generic (recursive case classes, etc.).</li><li>Source compatibility with circe-generic (both semi-automatic and auto).</li><li>Experimental configurability (snake-case field names, etc.).</li></ul><p>I’m also hoping to be in a position to benchmark compilation times, which has historically been a major pain point for circe-generic users.</p><h3>Jawn 1.0 and Circe 0.13</h3><p><a href="https://github.com/typelevel/jawn">Jawn</a> is a JSON parsing library for Scala that’s used by Circe and many other Scala libraries. It was created by <a href="https://twitter.com/d6">Erik Osheim</a> in September 2012, and finally, after almost seven and a half years, we published <a href="https://github.com/typelevel/jawn/releases/tag/v1.0.0">Jawn 1.0.0</a> last month. This release included <a href="https://github.com/typelevel/jawn/pull/219">one last major API change</a> that I wanted to get in before committing to long-term binary compatibility, a <a href="https://github.com/typelevel/jawn/pull/221">fix for a potential denial-of-service attack vulnerability</a>, a <a href="https://github.com/typelevel/jawn/pull/213">few new convenience methods</a>, and <a href="https://github.com/typelevel/jawn/pull/214">some small API changes</a> related to Dotty compatibility.</p><p>I’ve also published a <a href="https://github.com/circe/circe/releases/tag/v0.13.0-RC1">Circe 0.13.0-RC1 release</a> that’s binary-compatible with Circe 0.12.x for all of the modules that don’t depend on Jawn (e.g. circe-core and circe-generic), and that updates all of the modules that do depend on Jawn to 1.0.0. There are also <a href="https://twitter.com/http4s/status/1219778550032732160">two new http4s 0.21.0 release candidates</a> that depend on Circe 0.13.0-RC1 and Jawn 1.0.0, and I’m planning to publish the Circe 0.13.0 release in the next few days.</p><h3>Testing</h3><p>We also published a <a href="https://github.com/typelevel/discipline-scalatest/releases/tag/v1.0.0">1.0.0 release</a> for <a href="https://github.com/typelevel/discipline-scalatest">discipline-scalatest</a>, a library that’s designed to make type-class law checking as convenient as possible for ScalaTest users. I’d personally been avoiding discipline-scalatest (and reimplementing its functionality) because it depended on a ScalaTest module that was only available in pre-releases. This module (which provides integration with ScalaCheck) now has a general-availability release, so I put some time this month into working through a <a href="https://github.com/typelevel/discipline-scalatest/pull/43">few changes</a> that I’d need to use discipline-scalatest in my own projects (specifically support for ScalaTest’s FlatSpec and FunSpec testing styles).</p><p>Over the course of the month I published three <a href="https://github.com/typelevel/discipline-scalatest/releases/tag/v1.0.0-RC3">1.0.0 release candidates</a> with incremental versions of these changes, and tried them out in a range of projects, including Circe, Cats, <a href="https://github.com/bkirwi/decline/pull/134">cats-retry</a>, <a href="https://github.com/typelevel/paiges/pull/225">Paiges</a>, <a href="https://github.com/bkirwi/decline/pull/134">Decline</a>, and many others.</p><p>Testing turned out to be a central theme of my work in January, and I worked on a number of other testing-related efforts, including <a href="https://github.com/circe/circe/pull/1367">improving test consistency for Circe</a>, updating Dotty support for <a href="https://github.com/scalatest/scalatest/pull/1763">ScalaTest</a> and <a href="https://github.com/scalatest/scalatestplus-scalacheck/pull/24">ScalaTest’s ScalaCheck compatibility module</a>, and publishing <a href="https://repo1.maven.org/maven2/dev/travisbrown/">custom artifacts of the entire ScalaTest and Typelevel stack</a> for recent versions of Dotty.</p><h3>Preparing for Scala.js 1.0</h3><p>Scala.js 1.0.0 is <a href="https://www.scala-js.org/news/2019/12/13/announcing-scalajs-1.0.0-RC2/">coming soon</a>, and some Cats and Circe users have been asking for support for the 1.0.0 release candidates. Last month I added cross-building support to <a href="https://github.com/typelevel/cats/pull/3246">Cats</a>, <a href="https://github.com/typelevel/discipline-scalatest/pull/40">discipline-scalatest</a>, and <a href="https://github.com/circe/circe/pull/1355">Circe</a>, and published Scala.js 1.0.0-RC2 releases for all of these projects (excluding a few Circe modules that have dependencies that don’t support Scala.js 1.0).</p><h3>Other open source tasks and projects</h3><ul><li>Fixed a Dotty compatibility issue in <a href="https://github.com/tkawachi/sbt-doctest/pull/133">sbt-doctest</a>.</li><li>Fixed a macro hygiene issue in <a href="https://github.com/ChristopherDavenport/log4cats/pull/273">log4cats</a>.</li><li>Proposed <a href="https://github.com/typelevel/cats/pull/3264">making the Id type alias in Cats covariant</a>.</li><li>Published a new <a href="https://github.com/travisbrown/catbird/releases/tag/v20.1.0">Catbird release</a>.</li><li>Published two new milestone releases of Permutive’s <a href="https://github.com/permutive/fs2-google-pubsub/releases/tag/v0.15.0-M4">fs2-google-pubsub</a>.</li><li>Introduced Scalafmt in several projects, including <a href="https://github.com/typelevel/jawn/pull/210">Jawn</a> and <a href="https://github.com/typelevel/claimant/pull/45">Claimant</a>.</li><li>Fixed a publishing issue in the <a href="https://github.com/sangria-graphql-org/sangria/pull/27">Sangria build</a> and released <a href="https://github.com/sangria-graphql-org/sangria/releases/tag/v2.0.0-M2">2.0.0-M2</a>.</li><li>Cleaned up some minor issues in the Cats <a href="https://github.com/typelevel/cats/pull/3243">build</a>, <a href="https://github.com/typelevel/cats/pull/3261">tests</a>, and <a href="https://github.com/typelevel/cats/pull/3265">licensing</a>.</li></ul><h3>February preview</h3><ul><li>More Dotty cross-building (in particular fs2 and <a href="https://http4s.org/">http4s</a>).</li><li>More experiments with generic derivation for Cats and Circe on Dotty.</li><li>Cats 2.2.0 milestone with <a href="https://github.com/typelevel/cats/pull/3043">standard library instances</a> in implicit scope.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IIn99tKKbPPMWptWwe5HBQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cc7c5b5ce90" width="1" height="1" alt=""><hr><p><a href="https://medium.com/permutive/community-engineering-report-4cc7c5b5ce90">Community engineering report</a> was originally published in <a href="https://medium.com/permutive">Permutive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Community engineering report]]></title>
            <link>https://medium.com/permutive/community-engineering-report-ac882fa126b9?source=rss-2ae783c1ef98------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac882fa126b9</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[dotty]]></category>
            <dc:creator><![CDATA[Travis Brown]]></dc:creator>
            <pubDate>Mon, 06 Jan 2020 13:50:35 GMT</pubDate>
            <atom:updated>2020-01-29T16:38:48.946Z</atom:updated>
            <content:encoded><![CDATA[<h3>Permutive Community Engineering, December 2019</h3><p>December was another short month for the Community Engineering team at Permutive, but we were able to finish up a couple of important projects, and to get several steps closer to cross-building Scala libraries like Cats on Dotty, which is now <a href="https://dotty.epfl.ch/blog/2019/12/20/21th-dotty-milestone-release.html">feature-complete</a> and <a href="https://www.scala-lang.org/2019/12/18/road-to-scala-3.html">scheduled to become Scala 3</a> sooner than expected.</p><h3>Cats 2.1.0</h3><p>My primary goal for December was to get Cats 2.1.0 out the door. We decided to publish a <a href="https://github.com/typelevel/cats/releases/tag/v2.1.0-RC3">third release candidate</a> that included one final (we hope) binary-compatibility fix and several changes focused on smoothing the way to cross-building on Dotty.</p><p>We gave ourselves a week to identify any remaining issues in this release candidate, and when nothing turned up we published the general-availability <a href="https://github.com/typelevel/cats/releases/tag/v2.1.0">Cats 2.1.0</a> release, which includes contributions by over forty authors, seventeen of whom were first-time contributors.</p><h3>Dotty and Simulacrum</h3><p><a href="https://medium.com/permutive/community-engineering-report-d790a45b9111">Last month’s report</a> describes our initial experiments with reworking <a href="https://github.com/typelevel/simulacrum">Simulacrum 1</a>’s macro annotations in a Dotty-compatible form. I spent some time at the beginning of December cleaning up and documenting these experiments, resulting in a <a href="https://github.com/travisbrown/simulacrum-scalafix">proof-of-concept implementation</a> as a set of Scalafix rules and a <a href="https://github.com/typelevel/cats/issues/3192">proposal</a> discussing the costs and benefits of using this approach in Cats.</p><h3>Dotty and kind-projector</h3><p>One of the biggest obstacles to cross-building Cats on Scala 2 and Dotty is Cats’s extensive use of <a href="https://github.com/typelevel/kind-projector">kind-projector,</a> a Scala 2 compiler plugin that provides some syntactic improvements for type lambdas. Type lambdas now have their own <a href="https://dotty.epfl.ch/docs/reference/new-types/type-lambdas-spec.html">first-class language-level support</a> in Dotty, but in vanilla Scala 2 they were only supported via <a href="https://meta.plasm.us/posts/2015/07/11/roll-your-own-scala/">a hack that worked more or less by accident</a>. The kind-projector plugin provides a nicer interface to this hack, allowing user-facing code to use a syntax for type lambdas that’s similar to Scala’s syntax for value-level lambdas.</p><p>Unfortunately the syntax supported by kind-projector and the syntax chosen in Dotty aren’t the same, and the way type lambdas are used in Cats means that version-specific code isn’t a viable solution (you’d end up having to duplicate many dozens of files). I implemented <a href="https://github.com/travisbrown/simulacrum-scalafix#cross-building-cats-core-on-dotty">several Scalafix rules</a> to rewrite all of the kind-projector syntax used by Cats into ordinary Scala code that can be compiled by both Scala 2 and Dotty, but that was only to unblock some of my experiments — the expanded code is <a href="https://github.com/travisbrown/cats/commit/ef1c4824564a5660e0faa54b8a05e934136d84ac">extremely verbose and unreadable</a>, and isn’t something we’d want to merge in Cats.</p><p>While the current design of Dotty’s plugin architecture doesn’t support a direct port of kind-projector, Dotty does provide a -Ykind-projector compiler flag that’s intended to support a subset of kind-projector syntax. This support was unimplemented and its exact scope was undetermined, so I spent some time identifying which kind-projector features we actually needed in Cats.</p><p>In some cases, Cats used kind-projector syntax that could be replaced by ordinary Scala code in a way that didn’t harm readability, or that even improved it. For example, we replaced all instances of kind-projector’s <a href="https://github.com/typelevel/cats/pull/3207">variance annotations</a> for type lambda arguments, as well as its syntax for <a href="https://github.com/typelevel/cats/pull/3186">type-constructor arguments</a> for type lambdas.</p><p>I then put together <a href="https://github.com/lampepfl/dotty/pull/7775">an implementation</a> of -Ykind-projector that supported the remaining subset of kind-projector functionality. This implementation is out for review, and we’re hoping it’ll be available in Dotty’s nightly builds soon. Once that’s done, we’ll only be a couple of steps from being able to merge Dotty cross-building in Cats.</p><h3>Other open source tasks and projects</h3><ul><li>Added support for Jackson 2.10 to <a href="https://github.com/circe/circe-jackson/releases/tag/v0.12.1">circe-jackson</a> 0.11 and 0.12.</li><li>Migrated Circe’s documentation <a href="https://github.com/circe/circe/pull/1345">from Tut to mdoc</a>.</li><li>Proposed a parSequence for <a href="https://github.com/typelevel/kittens/pull/215">Kittens</a>.</li><li>Helped to migrate Permutive’s fs2-google-pubsub <a href="https://github.com/permutive/fs2-google-pubsub/pull/32">from Mill to sbt</a>.</li><li>Debugging <a href="https://github.com/http4s/http4s/issues/3027">http4s client issues</a>.</li><li>Minimized a <a href="https://github.com/lampepfl/dotty/issues/7876">Dotty bug</a> we were running into in Cats.</li><li>Published a new release of <a href="https://github.com/circe/circe-bson/releases/tag/v0.4.0">circe-bson</a>.</li><li>Published a new release of <a href="https://github.com/travisbrown/catbird/releases/tag/v19.12.0">Catbird</a>.</li></ul><h3>January preview</h3><ul><li>Dotty cross-building in Cats master.</li><li>Cats 2.2.0 milestone with <a href="https://github.com/typelevel/cats/pull/3043">standard library instances</a> in implicit scope.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QSML-nLyPQzr8unWjQUZzg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac882fa126b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/permutive/community-engineering-report-ac882fa126b9">Community engineering report</a> was originally published in <a href="https://medium.com/permutive">Permutive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Community engineering report]]></title>
            <link>https://medium.com/permutive/community-engineering-report-d790a45b9111?source=rss-2ae783c1ef98------2</link>
            <guid isPermaLink="false">https://medium.com/p/d790a45b9111</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[dotty]]></category>
            <dc:creator><![CDATA[Travis Brown]]></dc:creator>
            <pubDate>Fri, 06 Dec 2019 13:33:32 GMT</pubDate>
            <atom:updated>2019-12-06T13:39:18.544Z</atom:updated>
            <content:encoded><![CDATA[<h3>Permutive Community Engineering, November 2019</h3><p>We’d planned a few big releases for November (Cats 2.1.0, Jawn 1.0.0, and Sangria 2.0.0), and while none of these actually happened, Cats 2.1.0 in particular is shaping up to be a more significant release than we had originally planned, and we’ve made a lot of progress on an experiment that we’re hoping will allow us to cross-build Cats and some related libraries on Dotty in the near future (see the third section below for details). We’re also now running most of the internal services that support Permutive’s API on Scala 2.13, which is something that we’d been working toward for the past couple of months.</p><h3>Cats 2.1.0-RC1</h3><p>We dropped support for Scala 2.11 on the master branch of Cats after the 2.0.0 release, unblocking a large number of issues and pull requests that we had put on hold because they would break binary compatibility on 2.11. I spent much of the first couple of weeks of November helping to get these changes merged, and our <a href="https://github.com/typelevel/cats/releases/tag/v2.1.0-RC1">2.1.0-RC1 release candidate</a> was the biggest release since 1.0, with 78 pull requests by 35 developers, almost half of whom were first-time contributors to the project.</p><p>I also worked on several post-2.11 clean-up projects of my own in 2.1.0-RC1, including the <a href="https://github.com/typelevel/cats/pull/3152">rearrangement of type class operations</a> that could previously only be added as syntax methods, the <a href="https://github.com/typelevel/cats/pull/3149">removal</a> of some unnecessary Either right-biasing syntax, and <a href="https://github.com/typelevel/cats/pull/3129">improved coverage for binary-compatibility checking</a>.</p><p>Although we had dropped Scala 2.11 on the master branch in Cats, we had planned to continue publishing releases for 2.11 from a <a href="https://github.com/typelevel/cats/issues/3143">separate branch where contributors could backport new features</a>. We’re still interested in working with community members who want to help with this effort, but right now it seems unlikely that we’ll be able to publish Cats 2.1.0 for Scala 2.11.</p><h3>Cats 2.1.0-RC2</h3><p>We began updating some of our services at Permutive to Cats 2.1.0-RC1 a few days after the release, and immediately ran into an instance of <a href="https://github.com/typelevel/cats/pull/3162">broken binary compatibility</a> that hadn’t been caught by any of the tests or API checking in Cats itself. I tracked this breakage down to a <a href="https://github.com/lightbend/mima/issues/426">bug</a> in Lightbend’s <a href="https://github.com/lightbend/mima">MiMa tool</a>, and put together a fix that we were able to use to find a <a href="https://github.com/typelevel/cats/pull/3163">few other instances</a> of broken compatibility that we had accidentally introduced in RC1.</p><p>We released a <a href="https://github.com/typelevel/cats/releases/tag/v2.1.0-RC2">second release candidate</a> with these fixes (and a few other minor changes) a couple of weeks ago, pushing back the 2.1.0 release, which we’d initially planned for late November. We’ve been using 2.1.0-RC2 in production at Permutive since last week with no issues, and I’m hoping that we can publish 2.1.0 by mid-December.</p><h3>Cats and Dotty</h3><p>The other major project I worked on in November was an experiment focused on getting Cats building on <a href="https://dotty.epfl.ch/">Dotty</a>, a new research language and compiler that’s intended to become Scala 3. I’d spent some time this summer experimenting with a <a href="https://github.com/travisbrown/dotty-experiments">Dotty port of Cats and Circe</a>, primarily as a way to explore what some of the new language features will mean for projects like these. Now Dotty is <a href="https://contributors.scala-lang.org/t/preparing-for-feature-freeze/3780">approaching a feature freeze</a>, and I wanted to have a better sense of what work needs to be done in order to get these libraries cross-building on Dotty — not in some hypothetical future version, but in the next few months or even weeks.</p><p>One major blocker is Cats’s use of <a href="https://github.com/typelevel/simulacrum">Simulacrum</a>, a tool for generating boilerplate related to the encoding of type classes in Scala. Simulacrum 1 is built on Scala 2’s macro annotations, which up until Scala 2.13 were only supported via a <a href="https://docs.scala-lang.org/overviews/macros/paradise.html">compiler plugin</a>, and which are not supported at all in Dotty. We’ve had this issue on our radar for <a href="https://github.com/typelevel/cats/issues/2553">over a year</a>, and had discussed a couple of possible approaches for addressing it. There seemed to be agreement that the most promising approach would be to rewrite Simulacrum as a set of <a href="https://github.com/scalacenter/scalafix">Scalafix</a> rules, and I began working on a proof-of-concept implementation in mid-November.</p><p>This implementation is <a href="https://github.com/travisbrown/simulacrum-scalafix">now available</a>, together with some related tools that make it possible to cross-build cats-core and several other Cats modules on Dotty. As part of this effort I’ve also opened <a href="https://github.com/typelevel/cats/issues/2553#issuecomment-561076000">ten pull requests</a> that are designed to make the current Cats codebase more Dotty-friendly in cases where this can be done without also making it less idiomatic or readable. Altogether it’s possible to cross-build most of Cats on Dotty with only a few dozen lines of manual changes—the rest can be done by our Scalafix rules.</p><p>I’m currently working on a few follow-up threads: getting the remaining Cats modules building on Dotty, <a href="https://github.com/typelevel/cats/issues/3192">gathering community feedback</a>, and implementing Dotty’s support for <a href="https://github.com/typelevel/kind-projector">kind-projector</a>-compatible syntax for type lambdas.</p><p>At Permutive we’re excited about Dotty and Scala 3, and we’re hoping that investing in experiments and efforts like these will help to make the idea of migrating from Scala 2 a little less scary.</p><h3>Other tasks and projects</h3><ul><li>Published <a href="https://github.com/circe/circe-yaml/releases/tag/v0.12.0">circe-yaml 0.12.0</a> with several number-related improvements.</li><li>Published the <a href="https://github.com/circe/circe-droste/releases/tag/v0.1.0">first </a><a href="https://github.com/circe/circe-droste/releases/tag/v0.1.0">circe-droste release</a> following <a href="https://twitter.com/andygscott/status/1195473464263528448">Droste 0.8.0</a>.</li><li>Published a <a href="https://github.com/travisbrown/catbird/releases/tag/v19.11.0">new Catbird release</a>.</li><li>Continued <a href="https://github.com/sangria-graphql/sangria/issues/463">helping to coordinate work</a> toward Sangria 2.0.0.</li><li><a href="https://github.com/milessabin/shapeless/issues/936">Reported</a> and <a href="https://github.com/milessabin/shapeless/pull/937">fixed</a> a Shapeless bug affecting Circe’s generic derivation.</li><li>Reported and <a href="https://github.com/scalacenter/scalafix/pull/994">fixed</a> a minor issue with imports in Scalafix.</li><li>Reported and <a href="https://github.com/scalameta/scalameta/pull/1913">fixed</a> a bug in the Tokens API in Scalameta.</li></ul><h3>December preview</h3><ul><li>Scalafix-based code generation for Circe codecs.</li><li>An implementation for -Ykind-projector in Dotty.</li><li>Cats 2.1.0, probably Jawn 1.0.0, maybe Sangria 2.0.0.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d790a45b9111" width="1" height="1" alt=""><hr><p><a href="https://medium.com/permutive/community-engineering-report-d790a45b9111">Community engineering report</a> was originally published in <a href="https://medium.com/permutive">Permutive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Community engineering report]]></title>
            <link>https://medium.com/permutive/permutive-community-engineering-october-2019-51de37bc7791?source=rss-2ae783c1ef98------2</link>
            <guid isPermaLink="false">https://medium.com/p/51de37bc7791</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Travis Brown]]></dc:creator>
            <pubDate>Mon, 04 Nov 2019 13:57:09 GMT</pubDate>
            <atom:updated>2019-12-06T13:11:53.479Z</atom:updated>
            <content:encoded><![CDATA[<h3>Permutive Community Engineering, October 2019</h3><p>The community engineering team was at 75% capacity in October (I was out for a week of vacation), but we were able to wrap up several projects and kick off a few new things.</p><h3>Sangria</h3><p>A couple of weeks ago we were finally able to <a href="https://github.com/sangria-graphql/sangria/issues/446#issuecomment-546281588">recover publishing rights</a> for the Sangria project (see the <a href="https://medium.com/permutive/community-engineering-report-ff89378e7614">August edition of this report</a> for context). I had met earlier that week with developers from Twitter, the New York Times, and <a href="https://commercetools.com/">Commercetools</a> to discuss the future of the project. Once we had recovered publishing rights, <a href="https://twitter.com/simon_yann">Yann Simon</a> and I created a <a href="https://github.com/sangria-graphql-org">temporary new GitHub organization</a>, forked the 18 repositories containing Sangria-related libraries, and started updating and publishing them with Scala 2.13 support. By the end of the day we’d published new releases for about half of Sangria’s modules, as well as a <a href="https://github.com/sangria-graphql-org/sangria/releases/tag/v2.0.0-M1">2.0.0-M1 milestone release</a> of Sangria itself.</p><p>Since then we’ve <a href="https://github.com/sangria-graphql/sangria/issues/440#issuecomment-546398600">published releases</a> for all but one of the remaining Sangria modules, and have <a href="https://github.com/sangria-graphql/sangria/issues/463">started planning</a> for the final 2.0.0 release, which we’re hoping to have ready in the second half of November. At Permutive we’re in the process of updating our internal GraphQL code to use the new Sangria 2.0.0 milestone, and we’re <a href="https://www.reddit.com/r/scala/comments/dq3jyc/please_help_us_prepare_for_sangria_200/">reaching out</a> to other adopters to encourage them to do the same.</p><h3>JSON Schema validation for Circe</h3><p>We launched three new Circe libraries in October, including circe-json-schema, which I mentioned in <a href="https://medium.com/permutive/community-engineering-report-5da212ad4912">last month’s team report</a>. We use the <a href="https://github.com/everit-org/json-schema">Everit library</a> internally at Permutive together with Circe for <a href="https://json-schema.org/">JSON Schema</a> validation, and circe-json-schema is a small project that simplifies interoperability between the two libraries and provides a minimal validation API that is consistent with other Circe modules.</p><h3>Circe and recursion schemes</h3><p>Our 0.12.0 release in September introduced <a href="https://github.com/circe/circe-rs">circe-rs</a>, an experimental module that provides a pattern functor for Circe’s JSON representation, which is designed for use with Scala recursion scheme libraries like <a href="https://github.com/slamdata/matryoshka">Matryoshka</a> or <a href="https://github.com/higherkindness/droste">Droste</a>. I’ve split this module out into its own repository on GitHub, and have added a new circe-droste module, which additionally provides the mapping Droste needs between the pattern functor and Circe’s JSON abstract syntax tree. This new circe-droste module isn’t published yet, because I decided to wait for a new Droste release that will include Scala 2.13 support, but we’re <a href="https://github.com/higherkindness/droste/pull/134">working on that</a>.</p><h3>Golden testing for Circe</h3><p>The most significant of this month’s three new Circe libraries is <a href="https://github.com/circe/circe-golden">circe-golden</a>, which provides tools for writing “<a href="https://ro-che.info/articles/2017-12-04-golden-tests">golden tests</a>” for Circe’s encoders and decoders. This style of testing helps to address some common complaints about type class instance derivation (see the project description for details), and is something we’re planning to use extensively at Permutive to reduce boilerplate and give us more confidence in evolving our data models.</p><h3>Bug fixes and new features for Circe 0.12</h3><p>I also published a couple of new Circe patch releases (<a href="https://github.com/circe/circe/releases/tag/v0.12.2">0.12.2</a> and <a href="https://github.com/circe/circe/releases/tag/v0.12.3">0.12.3</a>) that include three bug fixes, some optimizations, and several useful API additions. While the bugs that have been fixed are minor and shouldn’t affect most users, we’re encouraging all Circe adopters to update to 0.12.3.</p><h3>Scala 2.13 at Permutive</h3><p>Most of our internal libraries are now cross-published for Scala 2.12 and 2.13, and many of our internal services are now running on Scala 2.13. Our new Sangria release unblocks one of the services that’s still on 2.12, and we’re hoping to finish that migration in early November.</p><p>We do have one service that is still running on Scala 2.11 because it uses the <a href="http://spray.io/">Spray</a> framework, which is no longer maintained, and which has never been published for Scala 2.12. I spent some time this month migrating that service to <a href="https://doc.akka.io/docs/akka-http/current/index.html">Akka HTTP</a> (Spray’s replacement), which will allow us to move directly to Scala 2.13.</p><h3>Other tasks and projects</h3><ul><li>Began <a href="https://gitter.im/typelevel/cats-dev?at=5db71073e886fb5aa20f56dc">planning</a> for the upcoming <a href="https://github.com/typelevel/cats/milestone/11">Cats 2.1.0</a> release.</li><li>Fixed some <a href="https://github.com/typelevel/cats/pull/3122">method name and signature issues</a> in Cats.</li><li>Fixed a <a href="https://github.com/typelevel/cats/pull/3102">minor issue with publishing metadata</a> in Cats.</li><li>Minimized and reported a possible <a href="https://github.com/scala/bug/issues/11787">Scala compiler bug</a>.</li><li>Made <a href="https://github.com/typelevel/jawn/pulls?utf8=%E2%9C%93&amp;q=is%3Apr+is%3Aclosed+author%3Atravisbrown+created%3A2019-10-01..2019-11-01">several change</a>s to Jawn in preparation for a 1.0.0 release.</li><li>Published a <a href="https://github.com/travisbrown/catbird/releases/tag/v19.10.0">new Catbird release</a>.</li></ul><h3>November preview</h3><ul><li>Several big releases (Sangria 2.0.0, Cats 2.1.0, Jawn 1.0.0)</li><li>Ongoing work toward Circe 1.0.0</li><li>Wrapping up Permutive’s Scala 2.13 migration</li><li>An overhaul for these team reports</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51de37bc7791" width="1" height="1" alt=""><hr><p><a href="https://medium.com/permutive/permutive-community-engineering-october-2019-51de37bc7791">Community engineering report</a> was originally published in <a href="https://medium.com/permutive">Permutive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Community engineering report]]></title>
            <link>https://medium.com/permutive/community-engineering-report-5da212ad4912?source=rss-2ae783c1ef98------2</link>
            <guid isPermaLink="false">https://medium.com/p/5da212ad4912</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Travis Brown]]></dc:creator>
            <pubDate>Tue, 01 Oct 2019 08:38:58 GMT</pubDate>
            <atom:updated>2019-10-01T08:38:58.221Z</atom:updated>
            <content:encoded><![CDATA[<h3>Permutive Community Engineering, September 2019</h3><p>This article gives a summary of projects that I worked on in my second month on the Community Engineering team at Permutive. For more context about these reports, please see <a href="https://medium.com/permutive/community-engineering-report-ff89378e7614">last month’s edition</a>.</p><h3>Cats 2.0.0</h3><p>One of the projects that I had focused on in August was preparation for the Cats 2.0.0 release, which <a href="https://github.com/typelevel/cats/releases/tag/v2.0.0">finally happened</a> this month. We ended up publishing <a href="https://github.com/typelevel/cats/releases/tag/v2.0.0-RC3">one last release candidate</a> with a couple of changes that broke source compatibility with previous releases (but that we thought were important enough to delay the release for a few days), and I made <a href="https://github.com/typelevel/cats/pull/3037">one minor change</a> (and reviewed a <a href="https://github.com/typelevel/cats/pull/3042">couple others</a>) after that release candidate.</p><p>I spent much of the rest of the month on follow-up tasks related to the 2.0.0 release, including three blog posts: one on <a href="https://meta.plasm.us/posts/2019/09/06/fuck-yeah-type-erasure/">how we managed to fix the</a><a href="https://meta.plasm.us/posts/2019/09/06/fuck-yeah-type-erasure/">Parallel type class</a>, one on <a href="https://meta.plasm.us/posts/2019/09/11/why-parallel/">why Cats users should care about </a><a href="https://meta.plasm.us/posts/2019/09/11/why-parallel/">Parallel</a> to start with, and one more general <a href="https://meta.plasm.us/posts/2019/09/10/cats-2.0-migration/">Cats 2.0 migration guide</a>. I also updated and published new versions of most of the open source libraries I maintain that depend on Cats, including <a href="https://github.com/travisbrown/catbird/releases/tag/v19.9.0">Catbird</a>, which provides type class instances for Twitter’s Util and <a href="https://finagle.github.io/blog/">Finagle</a> libraries, <a href="https://github.com/travisbrown/iteratee/releases/tag/v0.19.0">iteratee.io</a>, and Circe, together with most of the other libraries in the Circe organization (see the section below for details).</p><p>I also opened <a href="https://github.com/typelevel/cats/issues?utf8=%E2%9C%93&amp;q=is%3Aopen+is%3Aissue+author%3Atravisbrown+created%3A2019-09-01..2019-09-30">several Cats issues</a> to track problems that I’d noticed during the preparation for 2.0.0 but wasn’t able to fix.</p><h3>Cats and implicit scope</h3><p>In <a href="https://medium.com/permutive/community-engineering-report-ff89378e7614">last month’s report</a> I mentioned an experiment that involves moving the standard library type class instances in Cats into what Scala calls “implicit scope”. I worked through several iterations of this project in September, starting with a <a href="https://github.com/typelevel/cats/pull/3043">pull request</a> and including a <a href="https://meta.plasm.us/posts/2019/09/30/implicit-scope-and-cats/">blog post</a> that goes into more detail about the proposal’s motivations and benefits. I’ve also <a href="http://central.maven.org/maven2/dev/travisbrown/cats-core_2.13/2.0.0-CATS_3043-1/">published the experiment to Maven Central</a>, so that other developers can try it out more easily in their own projects.</p><p>We’re still months (or more) from this change possibly becoming a reality in Cats, but so far the improvements to usability and compile times are promising, and we’re in a good position to be able to get feedback from a wider set of users.</p><h3>Circe 0.12</h3><p><a href="https://github.com/circe/circe">Circe</a> depends on Cats, and our latest release series (0.12.x) had been waiting on the final Cats 2.0.0 release for almost five months (we ended up with four milestones and four release candidates during that time). I published <a href="https://github.com/circe/circe/releases/tag/v0.12.0">0.12.0</a> the day after Cats 2.0.0 came out, and we almost immediately noticed that I’d introduced a pretty serious bug since the last release candidate. Fortunately the bug only affected one deprecated method, but we still fixed it in an <a href="https://github.com/circe/circe/releases/tag/v0.12.1">0.12.1</a> release on the same day.</p><p>We published new 0.12 releases for most of the other libraries in the Circe organisation, including <a href="https://github.com/circe/circe-fs2/releases/tag/v0.12.0">circe-fs2</a>, <a href="https://github.com/circe/circe-optics/releases/tag/v0.12.0">circe-optics</a>, <a href="https://github.com/circe/circe-jackson/releases/tag/v0.12.0">circe-jackson</a>, and <a href="https://github.com/circe/circe-derivation/releases/tag/v0.12.0-M7">circe-derivation</a>, with the circe-derivation release in particular featuring some improvements over the previous milestone. I also moved the <a href="https://github.com/circe/circe-generic-extras">circe-generic-extras module</a> out of the core Circe repository so that it can be separately <a href="https://www.reddit.com/r/scala/comments/d3ny36/seeking_maintainers_for_circegenericextras/">developed</a> and versioned, and I created and published a fork of <a href="https://github.com/sangria-graphql/sangria-circe">sangria-circe</a> named <a href="https://github.com/circe/circe-sangria">circe-sangria</a>, since it makes more sense for this integration module to live in the Circe organisation, and since <a href="https://github.com/sangria-graphql/sangria/issues/446">we’re still working</a> on getting access to the Sangria repositories.</p><p>I’ve also been working on several changes for the first Circe 1.0.0 milestone release, and wrote a <a href="https://meta.plasm.us/posts/2019/09/23/scala-and-the-visitor-pattern/">blog post</a> about one of these threads. I was hoping to publish this milestone in September, but I‘m revising that to “maybe October” 😃.</p><h3>Scala 2.13 migration</h3><p>Last month I wrote that “we’re not likely to be in a position to use 2.13 in production for at least a couple more weeks”, and we actually hit this target, although only for a canary deployment of the main Permutive API service. We’re currently still waiting on non-milestone releases of <a href="https://http4s.org/">http4s</a> and <a href="https://sangria-graphql.org/">Sangria</a> before moving over to 2.13 more widely, although we’re cross-building many of our internal libraries now that Cats 2.0.0 is out, and we’re fairly confident that the rest of the transition will be straightforward.</p><h3>Other tasks and projects</h3><ul><li>Updated <a href="https://github.com/sangria-graphql/sangria/pull/453">all of my Sangria 2.13 pull requests</a>, adding <a href="https://github.com/lightbend/mima">MiMa checking</a> for binary compatibility so that we can get the versions right once we’re able to publish releases.</li><li>Published new releases of Permutive’s own <a href="https://github.com/permutive/fs2-google-pubsub/releases/tag/v0.14.0">fs2-google-pubsub</a> and <a href="https://github.com/permutive/sbt-liquibase-plugin/releases/tag/1.2.0">sbt-liquibase-plugin</a>.</li><li>Wrote up a series of Circe questions and answers on Stack Overflow (<a href="https://stackoverflow.com/q/57919641/334519">1</a>, <a href="https://stackoverflow.com/q/58025149/334519">2</a>, <a href="https://stackoverflow.com/q/58026172/334519">3</a>, <a href="https://stackoverflow.com/a/57919949/334519">4</a>).</li></ul><h3>Other notes</h3><p>Last month I introduced some categories for estimating how I’d allocated my time, and the distribution this month looks similar:</p><ul><li>20% purely internal work (code review, planning, coding challenge review, etc.)</li><li>20% open source work that’s directly motivated by internal needs</li><li>20% open source work that will have benefits for internal projects</li><li>40% open source work with little or no internal motivation</li></ul><p>I’m putting the five blog posts I published this month into the last category, which is the main reason it’s a little higher than last month, but I think there’s an argument that they fit better in the third (this might be an indication that the framework needs some reworking).</p><h3>October preview</h3><ul><li>Circe 0.12.2 release with a few useful additions to 0.12.1</li><li>More blog posts</li><li>First Circe 1.0.0 milestone</li><li>Scala 2.13 in production at Permutive (but more this time)</li><li>New circe-json-schema library</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5da212ad4912" width="1" height="1" alt=""><hr><p><a href="https://medium.com/permutive/community-engineering-report-5da212ad4912">Community engineering report</a> was originally published in <a href="https://medium.com/permutive">Permutive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Community engineering report]]></title>
            <link>https://medium.com/permutive/community-engineering-report-ff89378e7614?source=rss-2ae783c1ef98------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff89378e7614</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Travis Brown]]></dc:creator>
            <pubDate>Tue, 03 Sep 2019 13:19:49 GMT</pubDate>
            <atom:updated>2019-09-03T13:40:57.201Z</atom:updated>
            <content:encoded><![CDATA[<h3>Permutive Community Engineering, August 2019</h3><p>This article summarizes some of the projects that I worked on during my first month on Permutive’s new Community Engineering team. We’ll be publishing a follow-up post that goes into more detail about how we’re imagining my role, and how we’re balancing internal and external priorities; this report focuses on the open source side of the work I’ve been doing.</p><p>We’re planning to publish these reports at the beginning of every month, although the format is likely to change as the team grows and as we refine our goals and metrics. If you have questions or feedback, please get in touch in the comments below.</p><h3>Cats 2.0.0-RC2</h3><p><a href="https://github.com/typelevel/cats">Cats</a> is a widely-used open source Scala library that provides fundamental abstractions for functional programming. Its core module is downloaded from Maven Central over half a million times per month, and Maven Central includes over a thousand modules that depend on it. Permutive uses Cats (and libraries that are built on Cats) extensively in almost all of our web services.</p><p>Cats 1.0 was released in December 2017, and this summer we’ve finally been preparing for a 2.0 release. When I started as a community engineer at Permutive, one of the efforts that I wanted to prioritize in my first month was finalizing this preparation and publishing a second 2.0.0 release candidate. This ended up taking the form of three projects:</p><ol><li>Reworking the Parallel type class. Because of a design mistake in Cats 1.x, this component was essentially unusable without third-party workarounds, and was a frequent source of confusion, especially for new users. Along with the other Cats maintainers, I&#39;d thought that we would have to wait for Cats 3.0 to fix the mistake, because of binary compatibility guarantees we&#39;d made for Cats 2.0. A couple of weeks ago I figured out that it could be fixed without breaking these guarantees, and <a href="https://github.com/typelevel/cats/pull/3012">this fix</a> is now available in 2.0.0-RC2.</li><li>Resolving some issues in the handling of Scala 2.13 deprecations. The latest release of the Scala language (2.13) includes some major changes to the standard collections library, and in the initial Cats 2.0.0 milestone releases we made some decisions about how to address these changes that turned out not to work very well in practice. Over the course of a couple of weeks I <a href="https://github.com/typelevel/cats/issues/2982">outlined these problems</a>, worked with the other maintainers toward <a href="https://github.com/typelevel/cats/issues/2991">consensus on the best way forward</a>, and then worked through <a href="https://github.com/typelevel/cats/pull/2983#issuecomment-522979711">over a dozen iterations of an implementation</a> (which is also available in 2.0.0-RC2).</li><li>Miscellaneous clean-up tasks. Over the last year and a half of development on Cats 1.x, the project had accumulated a large number of minor inconsistencies and mistakes that generally didn’t affect usability, but did make maintaining and understanding the code more difficult. Because these issues will be harder to fix after 2.0.0, this work was happening under a deadline, and I wasn’t able to get consensus on all of the corrections I wanted to make. The most important ones are now in place, though, including a <a href="https://github.com/typelevel/cats/pull/3001">much more limited public API</a>, more consistent naming, and the <a href="https://github.com/typelevel/cats/pull/3001">addition of some missing type class instances</a>.</li></ol><p>I ran the Cats 2.0.0-RC2 release last week, and have been working with maintainers of other open source libraries that depend on Cats to update and publish for the new release (including <a href="https://github.com/typelevel/cats-effect/releases/tag/v2.0.0-RC2">cats-effect</a>, which we also use extensively at Permutive). I’ve also been working with adopters to verify that these changes don’t break their code (or in the few cases where breakage is expected, that the migration path is clear). The <a href="https://twitter.com/travisbrown/status/1166080419151978497">reception so far has been positive</a>, and we’re planning to publish the 2.0.0 final release on 9 September (assuming no major issues turn up before then).</p><h3>Circe</h3><p><a href="https://github.com/circe/circe">Circe</a> is a library that I maintain that’s also used pretty widely at Permutive, and at many other organizations, including Stripe, Twilio, Spotify, the Guardian, the BBC, etc. It depends on Cats, and I’m coordinating its 0.12 release with Cats 2.0. I’ve published four release candidates over the course of the month, which have included one <a href="https://github.com/circe/circe/releases/tag/v0.12.0-RC2">fairly significant bug fix</a>, a lot of last-minute small improvements (some guided by what I’ve learned about our use cases at Permutive), and some external contributions.</p><p>Internally, I replaced a module called circe-generic in one of our core repositories with an alternative that I’d originally created for Stripe (<a href="https://github.com/circe/circe-derivation">circe-derivation</a>), which cut clean compile times for the project by 25–30%. I’ve also made several changes to <a href="https://github.com/circe/circe-derivation/releases/tag/v0.11.0-M3">circe-derivation</a> to better support our use cases, and in the last of this month’s four circe-derivation releases I’ve <a href="https://github.com/circe/circe-derivation/releases/tag/v0.12.0-M6">incorporated and extended a large external contribution</a> that adds many new features to the library.</p><p>I also did some early investigation toward creating a new <a href="https://json-schema.org/">JSON Schema</a> validation library for Circe, which would allow us to replace some inefficient and inelegant internal code, but haven’t had time to flesh out the idea yet.</p><h3>Scala 2.13</h3><p>The <a href="https://www.scala-lang.org/news/2.13.0">latest release</a> of the Scala language includes many improvements that will be beneficial to us, such as a faster, more robust compiler, features like literal types that simplify some of the patterns we use, and a significantly better collections library.</p><p>While we’re not likely to be in a position to use 2.13 in production for at least a couple more weeks, we do have 2.13 update branches prepared and tests passing for most of our Scala repositories. One of the things we’re waiting on is the Cats ecosystem: we can’t update to Scala 2.13 until we’re on Cats 2.x, and while I’m personally fairly confident in the 2.0.0-RC2 milestone, the final release is only about a week away, and I think it makes sense to wait.</p><p>Another issue involves an open source GraphQL library we use called <a href="https://github.com/sangria-graphql/sangria">Sangria</a>. Sadly the owner and primary maintainer of the project <a href="https://github.com/sangria-graphql/sangria/issues/445">passed away in May of this year</a>, and there have been difficulties in working with GitHub and Sonatype to transfer project ownership. I had never used or contributed to Sangria before this month, but I took up some work-in-progress by a couple of the maintainers and was able to prepare 2.13 branches for the <a href="https://github.com/sangria-graphql/sangria/pull/453">core modules</a> (and all the additional pieces we need). I’m currently publishing these changes locally for testing, and we’re hoping to reach a resolution to the ownership issues soon.</p><p>In the meantime I’ve made a number of changes in our internal Scala repos that are compatible with 2.12 (our current version for most projects), but that will make the transition to 2.13 easier.</p><h3>Other tasks and projects</h3><ul><li><a href="https://twitter.com/AlicePermutive">Alice</a> published the video of my talk from Scala Central last month, and we’ve gotten some good feedback (see <a href="https://www.reddit.com/r/scala/comments/cwlrpf/experimenting_with_porting_cats_and_circe_to_dotty/">the Reddit thread</a>, or <a href="https://twitter.com/search?q=https%3A%2F%2Fvimeo.com%2F353579169&amp;src=typed_query">Twitter</a>).</li><li>Published a new release of <a href="https://github.com/permutive/fs2-google-pubsub/releases/tag/v0.14.0">fs2-google-pubsub</a>, one of Permutive’s own open source projects.</li><li>Fixed a <a href="https://github.com/twitter/util/pull/252">stack-safety issue</a> in the 19.8.0 release of Twitter Util.</li><li>Minimized and <a href="https://github.com/scala/bug/issues/11662">reported a regression</a> in the Scala 2.13.0 compiler.</li><li>Wrote up a Circe FAQ as a <a href="https://stackoverflow.com/a/57708249/334519">Stack Overflow Q&amp;A</a>.</li><li>Did some initial experiments with running a <a href="https://github.com/fthomas/scala-steward">Scala Steward</a> instance on our internal Scala repositories. Scala Steward is a maintenance bot that can automatically open pull requests updating dependencies for Scala projects, which can save developer time and help us avoid issues with dependency mismatches across our internal projects and libraries.</li></ul><h3>Other notes</h3><p>I’m still working out how to track the distribution of my time effectively, but my rough estimate for this month looks like this:</p><ul><li>30% purely internal work (onboarding, code review, planning, admin, etc.)</li><li>20% open source work that’s directly motivated by internal needs</li><li>30% open source work that will have benefits for internal projects</li><li>20% open source work with little or no internal motivation</li></ul><p>I’m hoping over the next couple of months to have a clearer sense of what these categories should look like and how I can track them most accurately. I’m also hoping to develop a more concrete set of metrics to include in these reports.</p><h3>September preview</h3><ul><li>First Circe 1.0.0 milestone</li><li>Scala 2.13 in production at Permutive</li><li>New circe-json-schema library</li><li>Proposal and proof of concept: Cats standard library instances in implicit scope</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff89378e7614" width="1" height="1" alt=""><hr><p><a href="https://medium.com/permutive/community-engineering-report-ff89378e7614">Community engineering report</a> was originally published in <a href="https://medium.com/permutive">Permutive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>