<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Krzysztof Grajek on Medium]]></title>
        <description><![CDATA[Stories by Krzysztof Grajek on Medium]]></description>
        <link>https://medium.com/@krzysztof.grajek?source=rss-5a68e9b7a9b1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_0p7MuajgsR3sDf7I_ku0g.jpeg</url>
            <title>Stories by Krzysztof Grajek on Medium</title>
            <link>https://medium.com/@krzysztof.grajek?source=rss-5a68e9b7a9b1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:50:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@krzysztof.grajek/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scala Cats Invariant Functor]]></title>
            <link>https://blog.softwaremill.com/scala-cats-invariant-functor-be57d2e2fa91?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/be57d2e2fa91</guid>
            <category><![CDATA[cats]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Mon, 01 Mar 2021 11:53:52 GMT</pubDate>
            <atom:updated>2021-03-01T11:53:52.571Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w9oi6g29U3mbTl36wKO-AA.jpeg" /><figcaption>Dragan @ Flickr.com</figcaption></figure><p>Today, I have another kind of Functor for you — the Invariant Functor! There were some posts already touching on Covariant Functors (simply called “Functor”) or Contravariant Functors like the one <a href="https://medium.com/p/2af1141c5956">here</a>. If you understand the concept of Covariant and Contravariant types of functors, then the Invariant will be easy to grasp as it combines functionality from both the functors mentioned above.</p><p>As you probably remember, with Functors we can map one type into another with a given function f:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1839f433b16b37cedfefcf148829743a/href">https://medium.com/media/1839f433b16b37cedfefcf148829743a/href</a></iframe><p>With Contravariant Functors, we do the same but with types in f switched:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bcdca3fa38c06382809aa18c81d50c18/href">https://medium.com/media/bcdca3fa38c06382809aa18c81d50c18/href</a></iframe><p>which gets useful when we want to provide new implicit implementations of some type class by reusing implementations already available.</p><p>In short, Functor’s map is used when we append operations in a chain and Contravariant Functor’s contramap is used when we want to prepend them. There is a third option that allows us to go in both directions with imap and this is specific to the Invariant Functor type.</p><p>Invariant functors define a method called imap similarly to map and contramap in the previously mentioned functor types. Imap is all about generating new type classes via a pair of bidirectional transformations.</p><p>The most common and easiest example of Invariant Functors include codecs and parsers, where we need to have the ability to transform something in both directions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/563be60143dd4b5a2a09893af6f21146/href">https://medium.com/media/563be60143dd4b5a2a09893af6f21146/href</a></iframe><p>In Cats library, the definition for imap in Invariant functor looks similar:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5556d186ad264d681bf0e4cfbdc271d/href">https://medium.com/media/c5556d186ad264d681bf0e4cfbdc271d/href</a></iframe><p>we can implement our imap in terms of available encode and decode methods:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0975dab3a5dfb782d7a00ecbcc90ad1/href">https://medium.com/media/e0975dab3a5dfb782d7a00ecbcc90ad1/href</a></iframe><p>This way, having the encode/decode method implementations, we can provide imap which will allow us to create new CustomParser instances for other types easily, for example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f3e237bc0b68811585437ba7ef5bfe5/href">https://medium.com/media/2f3e237bc0b68811585437ba7ef5bfe5/href</a></iframe><p>Let’s say we have an implementation of our CustomParser for Long values in our implicit scope and we want to get another parser but for java.util.Date type. We know how to convert Date to and from a Long value, therefore using the Invariant Functor idea, we can provide a new parser very easily:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3abf57c6282f4571940d05eba593332b/href">https://medium.com/media/3abf57c6282f4571940d05eba593332b/href</a></iframe><p>This is pretty similar to the already discussed Covariant and Contravariant Functors. One thing to note here is that Covariant and Contravariant Functors are actually descendants of the Invariant Functor, in other words the imap function can be implemented for them using their map or contramap respectively.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3ae14a6f8dc7a612f8d745706c1c700/href">https://medium.com/media/a3ae14a6f8dc7a612f8d745706c1c700/href</a></iframe><p>From Cats Invariant documentation:</p><blockquote>Every covariant (as well as <a href="https://typelevel.org/cats/typeclasses/contravariant.html">contravariant</a>) functor gives rise to an invariant functor, by ignoring the g (or in case of contravariance, f) function.</blockquote><p>where f and g correspond to our dec and enc functions.</p><p>Cats documentation mentions another good example of creating new instances of Invariant using Semigroup type class.</p><p>Having a Semigroup[Long] already provided by Cats, we can easily add new Semigroup instances for new types when we know how to convert from one type to another and back. The example is using Long -&gt; Date and Date -&gt; Long transformations we have used earlier:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a6e16d32ec41d480bc00d85bcc8c4d1/href">https://medium.com/media/7a6e16d32ec41d480bc00d85bcc8c4d1/href</a></iframe><p>Hope that makes things even clearer. Watch the <a href="http://blog.softwaremill.com/">SoftwareMill blog</a> space for more entries on Scala, Cats, and Functional Programming in general.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be57d2e2fa91" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/scala-cats-invariant-functor-be57d2e2fa91">Scala Cats Invariant Functor</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scala Cats Contravariant Functors]]></title>
            <link>https://blog.softwaremill.com/scala-cats-contravariant-functors-2af1141c5956?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2af1141c5956</guid>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[cats]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Mon, 08 Feb 2021 10:25:39 GMT</pubDate>
            <atom:updated>2021-02-08T10:25:39.594Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*JpbttZjZm1yfYmR_hFV12A.jpeg" /><figcaption>TH @ flickr.com</figcaption></figure><p>In this post, we’ll talk (again) about Functors — the ones from Scala Cats library, but not the classic Functors like we all know and love, we’ll cover a bit different kind of Functors called Contravariant Functors — depicted in Cats itself as a type class named just that: Contravariant.</p><p>Some of you might not know this, but the classic Functor with map operation we use every day in our Scala projects seasoned with Cats is in fact a Covariant Functor. Bear in mind that the term ‘Variance’ as applied to Functor types has nothing to do with different kinds of variance we know when talking about types and generic polymorphism.</p><p>A Typical Functor, in Scala FP terms, represents a type class operating on higher-kinded types, which proves pretty useful when we want to abstract and generalize our APIs.</p><p>Just for the sake of completeness, as we won’t talk about classic Functors here, I will provide a simple example below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8bfc826c574613663afe0a10c9dfa55/href">https://medium.com/media/a8bfc826c574613663afe0a10c9dfa55/href</a></iframe><p>The generalized version of reverse is not concerned with wrapper type we want to use as long as it can find a Functor instance for it. With Cats, this usually happens with an import, eg. for Option:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b5e8adb11fdabafe9e291fd4b98d399/href">https://medium.com/media/1b5e8adb11fdabafe9e291fd4b98d399/href</a></iframe><p>Ok, so I’m pretty sure you all knew that already. Functors are everywhere and we use them a lot, Applicatives are Functors, Monads are Functors, even a simple one param function (Function1) is a Functor too. Functor is all about a map method transforming a wrapped value of type A into type B and preserving the wrapper itself.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f54e88184bbb2fa2306ed31d2865b5b9/href">https://medium.com/media/f54e88184bbb2fa2306ed31d2865b5b9/href</a></iframe><p>But…, believe it or not, there are times where we want to reverse types in the Functor function f so it will take type B and return type A but the function itself will still return wrapper of type B — confused? If so, read on as this post is for you :).</p><p>Imagine a simple type class responsible for the transformation of some type T into a Boolean — popular filtering. Let’s define it as a simple Filter trait for now:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b0cf365fbcdee422038d11e7901853b/href">https://medium.com/media/5b0cf365fbcdee422038d11e7901853b/href</a></iframe><p>To work with our type class, we will create a very simple instance and an interface so we can use it to prove the point:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cd7413956cbcd95407ce18514a5b41d/href">https://medium.com/media/4cd7413956cbcd95407ce18514a5b41d/href</a></iframe><p>The implementation is not important here, so excuse me for not inventing anything more advanced :). Of course, the first println outputs false and the second one outputs the value true.</p><p>Now imagine that you need a Functor functionality for our Filter type class to convert it from Filter[String] to Filter[Int] using a map method like we usually do:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d1cb1719621b72e42d89bfb53d7932d/href">https://medium.com/media/7d1cb1719621b72e42d89bfb53d7932d/href</a></iframe><p>Do you see the problem we have here? We cannot pass value to our f mapping function as we cannot use type B in this case as an input to the function f. What we need here is an A type.</p><p>In other words, we want to compose A =&gt; Boolean with B =&gt; Boolean having a function A =&gt; B.</p><p>So, how do we do that? We need to use a Contravariant Functor instead of a Covariant one. The type class in Cats for doing just that is named simply Contravariant.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a0251e66cd41efc054fe7169adc1c24/href">https://medium.com/media/8a0251e66cd41efc054fe7169adc1c24/href</a></iframe><p>As you have probably noticed, the input type parameter and the output types are switched compared to our old and well known map operation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e1ed37af73dd6aa74f41d129da4c47a/href">https://medium.com/media/8e1ed37af73dd6aa74f41d129da4c47a/href</a></iframe><p>Composing with Contravariant is as easy as with typical Covariant Functor:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74235290286fda21759084c952fbfb5c/href">https://medium.com/media/74235290286fda21759084c952fbfb5c/href</a></iframe><p>Similarly, we can use Contravariant to work with wrapped values in a context like Option, typical example with Cats Show[_]:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5cdb2c80ffd70a278d1edb54da80e33/href">https://medium.com/media/f5cdb2c80ffd70a278d1edb54da80e33/href</a></iframe><p>or even a shorter version:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c681c7255dedf3b69ae507bbbfaa2c3/href">https://medium.com/media/1c681c7255dedf3b69ae507bbbfaa2c3/href</a></iframe><p>In the same manner, we can use any types wrapped in Option for our filters:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a32c35840a90f9bf53a5c845b7e7854a/href">https://medium.com/media/a32c35840a90f9bf53a5c845b7e7854a/href</a></iframe><p>Or even better, getting rid of _.get :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7dec78dfa03804bd98beaa357f8e706/href">https://medium.com/media/e7dec78dfa03804bd98beaa357f8e706/href</a></iframe><p>Ok, so that’s all when it comes to Contravariant Functors. I hope that I have made it simpler and after reading this, you can add one more type class to your programming toolbox. There are, of course, a number of other interesting type classes in our beloved Cats library, even one more interesting Functor type :) but I will leave that for later.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2af1141c5956" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/scala-cats-contravariant-functors-2af1141c5956">Scala Cats Contravariant Functors</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cats Concurrency basics with Ref and Deferred]]></title>
            <link>https://blog.softwaremill.com/cats-concurrency-basics-with-ref-and-deferred-399e0335630?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/399e0335630</guid>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[cats]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Thu, 25 Jun 2020 06:11:58 GMT</pubDate>
            <atom:updated>2020-12-21T07:39:31.147Z</atom:updated>
            <content:encoded><![CDATA[<h4>Concurrent access and referential transparency</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*PmNlowZcYHohTBDiy3JjXw.jpeg" /><figcaption>Ray Dumas @ Flickr.com</figcaption></figure><p><strong>Ref</strong> and <strong>Deferred</strong> are the basic building blocks of structures in FP used in concurrent fashion. Especially when used with tagless final abstraction those two, when building the business logic, can get us both: <strong>concurrent access and referential transparency</strong> and we can use them to construct more advanced structures like counters and state machines.</p><p>Before we dive deep into Ref and Deferred it is useful to know that concurrency in Cats is based around Java AtomicReference and we will start our journey here.</p><h4>Atomic Reference</h4><p>AtomicReference is one of the elements from java.util.concurrent.atomic package and in <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/package-summary.html">Oracle docs</a> we can read that java.util.concurrent.atomic is:</p><blockquote>A small toolkit of classes that support lock-free thread-safe programming on single variables. In essence, the classes in this package extend the notion of volatile values, fields, and array elements to those that also provide an atomic conditional update operation..</blockquote><blockquote>…</blockquote><blockquote>Instances of classes <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicBoolean.html">AtomicBoolean</a>, <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicInteger.html">AtomicInteger</a>, <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicLong.html">AtomicLong</a>, and <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicReference.html">AtomicReference</a> each provide access and updates to a single variable of the corresponding type.</blockquote><p>AtomicReference is with us since Java 1.5 and it’s used to get <strong>better performance than synchronisation</strong> (which usually but not always is the case).</p><p>When you have to share some data between threads you have to protect access to that piece of data. The simplest example would be to increment some int: i = i + 1. Our example consist of actually 3 operations, first we would read the value of i then we add 1 to that value, and at the end we assign newly calculated value to the i again. With multithreaded applications we can have a situation where each thread will execute those 3 steps between other thread’s steps and the final value of i will not be possible to predict.</p><p>Normally the word synchronised or lock class mechanism would appear in your head, but with atomic.* you no longer need to worry about explicit synchronisation, and you can relay on provided utility atomic types where the check if the operation is done in one step is automatically included.</p><p>Let’s take the AtomicInteger.incrementAndGet as an example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c3e5ac51e9b17d4a1a3e46a92ce9cb6/href">https://medium.com/media/2c3e5ac51e9b17d4a1a3e46a92ce9cb6/href</a></iframe><p>With compareAndSet operation we either update our data or fail but we never make the thread wait. This way if the compareAndSet in incrementAndGet fails we just try to redo the whole operation again, fetching the current value of our data with get() at the beginning. On the other hand with synchronized-like mechanisms there is no limit on how many statements you want to execute while the lock is acquired but that block will never fail and may make the calling thread wait — providing the possibility to deadlock or decreasing performance.</p><p>Now, with the basics in our head let’s move on to our first concurrency mega-star.</p><h3>Ref</h3><p>Cats Ref is very similar to the above mentioned Java atomic reference, the main differences are that Ref is used with tagless final abstraction F , it always contains a value and the value contained in Ref — of A type is always immutable.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e62b1b88bafd0f412cf94325dda5931/href">https://medium.com/media/8e62b1b88bafd0f412cf94325dda5931/href</a></iframe><p>Ref[F[_], A] is<strong> purely functional mutable reference:</strong></p><ul><li>concurrent</li><li>lock free</li><li>and always contains a value</li></ul><p>It is created by giving an initial value and every operation on is wrapped in F eg. cats.effect.IO.</p><p>If we look closely into companion object for Cats Ref we can see that our F needs to fulfil a requirement of being a Sync.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/754c061397aa3072595677dda8dfba92/href">https://medium.com/media/754c061397aa3072595677dda8dfba92/href</a></iframe><p>The above method is just an example of many operations available on our Ref, it is used to construct Ref with an initial value. Sync gives us the ability to suspend any side effects with its delay method for every operation on Ref.</p><p>Ref is a pretty simple construct, we can focus mainly on its get, set and of methods to understand how it works.</p><p><strong>get and </strong><strong>set approach</strong></p><p>Let’s say that we have an object (for the purposes of this blog post we will name it Shared) which needs to be updated by multiple threads and we use our get and set methods to do that, constructing utility method to help us on the way:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad0ca4a8e263e1f0d0dac4e4d618cd17/href">https://medium.com/media/ad0ca4a8e263e1f0d0dac4e4d618cd17/href</a></iframe><p>Our Shared object can be constructed by applying its previous state and new value to build a new instance — the Shared can be actually anything we want — a simple List, Map or whatever we want to access concurrently in a safe manner. I have just invented Shared(prev: Shared, msg: String) for the purposes of this tutorial.</p><p>F in our example above was replaced by concrete IO from Cats effect but bear in mind that Ref is polymorphic in F and can be used with other libraries.</p><p>With our monadic IO we flatMap over each step and set the value stored in our Ref to a desired value — or… wait, maybe we don’t.</p><p>With this approach, when modifyShared will be called concurrently <strong>we can lose updates! </strong>This is because, we can have a situation when eg. two threads can read the value with get and each one of them will execute set concurrently. Methods get and set are not called atomically together.</p><p><strong>Atomic </strong><strong>update</strong></p><p>Of course we can improve on the above example and use other available methods from Ref. To execute get and set together we can use update.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f86d8145316bb07b57d6d640e659091c/href">https://medium.com/media/f86d8145316bb07b57d6d640e659091c/href</a></iframe><p>This will solve our issue with updating the value, but simple update has its drawbacks. If we want to read the value just right after updating it, in a similar fashion we were using get and set, we can end up with stale data being read, let’s say our Ref will hold a reference to a simple Int:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3e09b18db5dfb8d1f4919b9c1ef17a4/href">https://medium.com/media/b3e09b18db5dfb8d1f4919b9c1ef17a4/href</a></iframe><p><strong>modify to the rescue</strong></p><p>We can improve slightly on that by using modify which will do the same as update, but will return the updated value back to us for later use.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b28c47e2448e1fe264a389821f49c354/href">https://medium.com/media/b28c47e2448e1fe264a389821f49c354/href</a></iframe><p>As you can see, this is almost exactly the same implementation as in AtomicInteger.incrementAndGet example I have shown at the beginning, only in Scala. You can clearly see that Ref is based on AtomicReference to do its work, too.</p><h4>Ref limitations</h4><p>You have probably already noticed that in case if updating the value fails the function passed to update/ modify needs to be run nondeterministically and maybe run multiple times. The good news is though, that this solution turns out in general to be <strong>much faster than standard locking and synchronisation mechanism</strong> and it’s much safer as this solution cannot deadlock.</p><p>Once we know how simple Ref works we can move onto another Cats Concurrent class: Deferred.</p><h3>Deferred</h3><p>In contrast to Ref, Deferred is:</p><ul><li>created empty</li><li>can be completed once</li><li>and once set it cannot be modified or become empty again.</li></ul><p>Those properties make the Deferred simple and pretty interesting at the same time.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8993cad393ec736bf50f7d2416089b25/href">https://medium.com/media/8993cad393ec736bf50f7d2416089b25/href</a></iframe><p>Deferred is for <strong>purely functional synchronisation</strong>, when we call get on empty Deferred we block until the value is available. As per <a href="https://github.com/typelevel/cats-effect/blob/master/core/shared/src/main/scala/cats/effect/concurrent/Deferred.scala">documentation from the class itself</a> the blocking:</p><blockquote>blocking mentioned is semantic only, no actual threads are blocked by the implementation</blockquote><p>The same call of get on non-empty Deferred will return immediately the value stored.</p><p>The other method - complete — will fill up the value if the instance is empty and when called on non-empty Deferred it will return failure (failed IO in case of IO).</p><p>The important thing to note here is that Deferred requires F to be Concurrent which means that it can be cancellable.</p><p>Good example of using Deferred is when one part of your application needs to wait for another. Example below is taken from the great talk by Fabio Labella given at Scala Italy 2019 — <a href="https://vimeo.com/366191463">Composable Concurrency with Ref + Deferred available at Vimeo</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce68167a3e0d017f907d6a9f065e2186/href">https://medium.com/media/ce68167a3e0d017f907d6a9f065e2186/href</a></iframe><p>In the above example we have a producer and consumer and we want the producer to wait for the consumer setup to finish before writing the messages, otherwise whatever we would write in producer will be lost. To overcome this problem we can use shared Deferred instance and block on get until the done Deferred instance will be filled up with value on the consumer side (the value in this case is simple Unit () ).</p><p>Of course the above solution is not without its issues, when the consumer setup never terminates we are stuck on waiting and the producer cannot send any messages. To overcome this, we can timeout on get as well as use Either[Throwable, Unit] or some other construct instead of simple Unit inside our Deferred object.</p><p>Deferred is pretty simple but with conjunction with Ref it can be used to build more complex data structures like eg. semaphores.</p><p>For more information I encourage you to visit the Cats documentation itself where you can find out more about <a href="https://typelevel.org/cats-effect/concurrency/semaphore.html">Cats concurrency</a> and the data structures it provides.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=399e0335630" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/cats-concurrency-basics-with-ref-and-deferred-399e0335630">Cats Concurrency basics with Ref and Deferred</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Towards more functional Play — Part 2]]></title>
            <link>https://blog.softwaremill.com/towards-more-functional-play-part-2-ac20d02bb2f5?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac20d02bb2f5</guid>
            <category><![CDATA[akka]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[play]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Fri, 10 Jan 2020 08:30:06 GMT</pubDate>
            <atom:updated>2020-01-10T08:30:06.624Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/a97bc4d92c76/edit?source=your_stories_page---------------------------">Towards more functional Play — Part </a>2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*UGWVjOVfyKko2hfxBeZDZw.jpeg" /><figcaption>anaa yoo @ flickr.com</figcaption></figure><p>In <a href="https://blog.softwaremill.com/towards-more-functional-play-part-1-a97bc4d92c76"><strong>the previous part of this series</strong></a> I talked about how to start moving your existing Play application towards more functional solution. We started simply, introducing IO from the famous Cats library.</p><p>The change was pretty simple, yet very powerful, allowing us to use the power of referential transparency versus Future based mess we often encounter, especially in existing code bases.</p><p>Today we will move one step further and start using <a href="https://blog.softwaremill.com/final-tagless-seen-alive-79a8d884691d">Tagless Final</a> pattern to further abstract our interfaces and prepare our application for easier and more extensive testing. The main issue when using Tagless Final in existing Play application is the problem with built-in dependency injection — of course we will fix it but more on that later.</p><p>We are going to add some new packages like repositories and services and possibly more, to further divide our source files into a bit more meaningful groups.</p><p>Let’s start using the tagless final pattern and create PersonRepository interface with all the method declarations we need:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c2b77e1db9563808e31a22837b1a8be9/href">https://medium.com/media/c2b77e1db9563808e31a22837b1a8be9/href</a></iframe><p>As you can see, we are no longer bound to a specific effectful type like IO but to a higher kinded type F[_]. We can create concrete implementations for our interface when needed, eg: InMemoryPersonRepository for test purposes or SlickPersonRepository for database Slick library usage (more on that later).</p><p>As we are expecting some business processing, we won’t be using our repository directly in the controller but we will proxy it through the PersonService.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e895c2d1a2892751d9741bde269035b/href">https://medium.com/media/3e895c2d1a2892751d9741bde269035b/href</a></iframe><p>As you can see we are not using any guice or javax dependency injection annotations here, we would need to wire things up manually. Before wiring though, we need a concrete implementation for our repository.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b53f70dbd004e45d4662797a45a6c240/href">https://medium.com/media/b53f70dbd004e45d4662797a45a6c240/href</a></iframe><p>Note that we are still not bound to a specific effect, the only requirement we have, is that the effect we will use later implements the MonadError (we didn’t even have to do that but it’s much easier to deal with errors when we have that in place).</p><p>If you compare the above snippet to the first version of the repository discussed in Part 1 (todo: add link), you will see that we are no longer bound to the IO and we are using F[Person], F[Option[Person] etc. On top of that, I have added one more method to get a Person from the database using the name argument — this will be used to show some useless business logic in the service layer.</p><p>Because we have resigned from using the built-in dependency injection mechanism available in Play, we need to configure our application loader by hand. This is much easier than it sounds and can be done directly with new or using some wring library like <a href="https://github.com/softwaremill/macwire">macwire</a> (which I highly recommend as it’s created by one of our SoftwareMill’ers — <a href="https://twitter.com/adamwarski?lang=en">Adam Warski</a>).</p><p>I have created two separate component groups and load them with custom AppLoader:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/244687c70eaeb9e79d93af136aaedc34/href">https://medium.com/media/244687c70eaeb9e79d93af136aaedc34/href</a></iframe><p>The database specific dependencies are available in DatabaseComponents and used in more general AppComponents.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2412e8ce431e03c4286ffef7a7834dc4/href">https://medium.com/media/2412e8ce431e03c4286ffef7a7834dc4/href</a></iframe><p>Please refer to the source code inside ‘wiring’ package for more information.</p><p>At this point we have abstracted our repository and service layer and we can decide which effectfull concrete implementation we will use when we wire the whole application up (in this case we picked IO). As a bonus to our solution, we have defined that our effect should implement the algebra of MonadError which will help us further to deal with errors in our call stack.</p><p>For the full implementation of the presented solutions in example Play application please refer to the Github repository <a href="https://github.com/softwaremill/play-scala-slick-example-part2">here</a>.</p><p>Watch this space for more stuff regarding the Play framework and how to make it more FP.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac20d02bb2f5" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/towards-more-functional-play-part-2-ac20d02bb2f5">Towards more functional Play — Part 2</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Towards more functional Play — Part 1]]></title>
            <link>https://blog.softwaremill.com/towards-more-functional-play-part-1-a97bc4d92c76?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a97bc4d92c76</guid>
            <category><![CDATA[play]]></category>
            <category><![CDATA[akka]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[functional-programming]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Thu, 09 Jan 2020 08:34:39 GMT</pubDate>
            <atom:updated>2020-01-09T08:34:39.600Z</atom:updated>
            <content:encoded><![CDATA[<h3>Towards more functional Play — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*WWnhhM9_6znG9LqIPZQfRQ.jpeg" /><figcaption><a href="https://flickr.com/photos/pagedooley/">Kevin Dooley</a> @ Flickr.com (CC 2.0)</figcaption></figure><p>From time to time, with strange turbulence in space-time dimensions, it happens that I need to work with the Play framework. This of course is nothing bad and I got used to it, I know its quirks and I can quickly build a microservice following the Play’s convention over configuration principles. Although Play Framework is based on Akka Streams and working with Akka is a breeze, I miss some of the common Functional Programming patterns when encountering a typical Play-based project.</p><p>In this post I will try to show you how to make your Play application to be more FP, how this can help you reason about your code, to get rid of Future referential transparency problems and more.</p><p>I thought that the best way to show how we can write a Play application differently is to take the existing Play starter application with Slick and step-by-step convert it to a more functional creature.</p><h4>The Old — Good, Bad and Ugly</h4><p>You can find a lot of examples on how to use Play framework in the Play Samples <a href="https://github.com/playframework/play-samples">repository</a>. We will focus mainly on play-scala-slick-example project which can be found <a href="https://github.com/playframework/play-samples/tree/2.7.x/play-scala-slick-example">here</a>.</p><p>The bad thing about the play-scala-slick-example project is that it has no service layer which normally exists in a microservice Play-based project doing the business logic, but it’s a minor one. I guess the authors of the example projects didn’t care about that much, as it’s not needed to show the core of the functionality they try to present. Anyway, we will try to fix that with some examples later, as it will help us to show the advantages of more functional solution.</p><p>The way things work in our original example is that we have basically a model, repository and controller and you are able to modify the state using the view. My favourite thing about Play is that it has a common JSON handling mechanism, therefore if you choose Play to be your solution to write multiple microservices talking to each other, you will very rarely encounter JSON serialization and deserialization problems (which, by the way, takes usually 1/3rd of technical discussions in every other project ;) ). There are many other nice things about Play but let’s not focus on that now and move to more serious matter.</p><p>The fact that Play is using Akka (which I like) causes that we end up with Future wrapped values everywhere (which I don’t like). This has a number of serious consequences:</p><ul><li>Futures are not referentially transparent and they memoize their values (this generates a bunch of other serious consequences)</li><li>Because they memoize their value, calling the same future reference multiple times changes nothing (but causes confusion in the code)</li><li>map, flatMap etc. on Future requires ExecutionContext to be in scope</li><li>you cannot cancel a Future once it is started</li><li>when using Future you don’t know if you are dealing with side effects or not</li><li>they are difficult to test</li><li>they are difficult to refactor</li><li>they are difficult to reason about</li></ul><p>And there are more. You can read about it in the excellent blog post <strong><em>‘Are Scala Futures the past?’</em></strong> <a href="https://medium.com/@sderosiaux/are-scala-futures-the-past-69bd62b9c001">here</a>.</p><p>I assume the reader has an existing application, or maybe a number of microservices written in Play framework and would like to step by step move towards more functional approach, without doing a huge revolution but with small evolutions applied one by one, where all the people in the team can follow and learn. We start with very simple but powerful change.</p><h4>Stage 1</h4><p>The simplest thing you can do to make your Play application a bit better is to use some data structure which performs effects on evaluation, for example <strong>Cats IO</strong>.</p><p>With IO the values are pure and immutable and because of that, they preserve referential transparency. IO represents just a description of a side effectful computation. It’s much easier to test and to reason about, you just use it as any other value.</p><p>In our example project we have two methods in the repository returning Future</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43ecb844b5e7d2af4e1678f10f3c315a/href">https://medium.com/media/43ecb844b5e7d2af4e1678f10f3c315a/href</a></iframe><p>We can easily convert them to return IO instead and deal with IO wrapped values elsewhere in the project. This gives us one major advantage — referential transparency. Once you have your side effects wrapped in the IO monad you can forget about famous problems often encountered with Futures:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbc68097459c9a4571422c9fb7aa53c7/href">https://medium.com/media/cbc68097459c9a4571422c9fb7aa53c7/href</a></iframe><p>The code above will print &quot;hey&quot; value once, whereas with IO we can do the same and have better understanding of what’s happening:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e3d31efaefb9dabd9d87ac6cafd1ded/href">https://medium.com/media/6e3d31efaefb9dabd9d87ac6cafd1ded/href</a></iframe><p>Of course, when wrapping the Future you still need to remember about its memoization feature:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4ef83c2e0b309705b5eb5ea3d50664d/href">https://medium.com/media/a4ef83c2e0b309705b5eb5ea3d50664d/href</a></iframe><p>will print &quot;hey fut!&quot; only once! When wrapping the Future remember to wrap the block of code which creates the reference and not the reference after creation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c511a84f39f5d0c5b0ac0f6e38133592/href">https://medium.com/media/c511a84f39f5d0c5b0ac0f6e38133592/href</a></iframe><p>Example for our use case in play-scala-slick-example project would be:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc7d5fec03d3b0df9ac8bbacbe6c1686/href">https://medium.com/media/bc7d5fec03d3b0df9ac8bbacbe6c1686/href</a></iframe><p>This above code could work but we would need to wrap all our db calls into double wrapped IO setup. We can do better of course. Executing our slick db.run(..) methods can be wrapped with specially created util method:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e64963b6cde9acd9ec858889c984b53/href">https://medium.com/media/7e64963b6cde9acd9ec858889c984b53/href</a></iframe><p>The code above will solve your problem with repeating IO wraps and will convert our Future to IO using IO.async for best slick integration.</p><p>For all this to work you would need to add implicit ContextShift for our IO operations, you can either add an implicit parameter to the method or just add it in the class scope initialized from implicit ExecutionContext available in Play Framework.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f073a9d365ddd406292d68bcccce920/href">https://medium.com/media/9f073a9d365ddd406292d68bcccce920/href</a></iframe><p>That is a pretty easy change giving us a lot of benefits. Now, when we have our side effectfull operations wrapped in IO, we need to use it somehow.</p><p>The simplest approach would be to execute unsafeToFuture() in all the places which we consider our ‘end of the world’ — mostly Controller methods.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9266b53cf60bcfdfe935530397685b2a/href">https://medium.com/media/9266b53cf60bcfdfe935530397685b2a/href</a></iframe><p>This solves our problem but it’s cluttering our controllers with many unsafeToFuture() calls. We can do a bit better than that. With a simple type class mechanism we can extend our existing Play’s ActionBuilder and use IO directly in our controllers:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/318766b6144a2d7d5769a85fbf6e4113/href">https://medium.com/media/318766b6144a2d7d5769a85fbf6e4113/href</a></iframe><p>The ActionBuilder example was actually borrowed from the YouTube presentation “<a href="https://www.youtube.com/watch?v=phTq8XBh81U&amp;t=985s">Alexandre Delegue — Refactorer une application play en utilisant cats et cats IO!</a>” — too bad it’s in French ;)</p><p>Having the above imported, we can simply do:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b6589ab6fdd57b25b8d17502929ed43a/href">https://medium.com/media/b6589ab6fdd57b25b8d17502929ed43a/href</a></iframe><p>Wherever you are returning Future.successfull(..) calls, you can safely get rid of the Future itself and use IO.apply or IO.pure to do the same, for example in our controller addPerson method we can do (when encountering errors in the form submission):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23df3513379c7d49a05e869f0be6218c/href">https://medium.com/media/23df3513379c7d49a05e869f0be6218c/href</a></iframe><p>For the differences between IO(..), IO.pure(..) and others, please see the excellent Cats documentation <a href="https://typelevel.org/cats-effect/datatypes/io.html#pure-values--iopure--iounit">here</a>.</p><p>The simple change presented in the first part of this series, gives us the ability to use our IO everywhere in the Play-based project. This greatly simplifies reasoning about the code, refactoring and in the long run testing. To modify your existing project successfully to use IO in the way presented you would need to add the cats-effect dependency of course and enable the higherKinds compiler option in your build.sbt. You can find the source code for this example in the <a href="https://github.com/softwaremill/play-scala-slick-example-part1">GitHub repository</a>.</p><p>Watch this space for more tips and tricks on how to make your existing or new Play application in a more functional manner.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a97bc4d92c76" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/towards-more-functional-play-part-1-a97bc4d92c76">Towards more functional Play — Part 1</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monoid in the Category of Endofunctors]]></title>
            <link>https://blog.softwaremill.com/monoid-in-the-category-of-endofunctors-b85bab43587b?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b85bab43587b</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[cats]]></category>
            <category><![CDATA[category-theory]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Tue, 03 Dec 2019 07:50:47 GMT</pubDate>
            <atom:updated>2019-12-03T07:50:47.118Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*e6ShVQpQaNSURpohcve-Qw.jpeg" /><figcaption>Phil Reed @ Flickr CC 2.0</figcaption></figure><p>Today we are going to deep dive into the Category theory again to find out what the famous sentence, being the title of this blog posts, means.</p><p>The sentence originally appeared in the book “<a href="https://books.google.pl/books?id=MXboNPdTv7QC&amp;pg=PA138&amp;lpg=PA138&amp;dq=%22monoid+in+the+category+of+endofunctors%22+mac+lane&amp;source=bl&amp;ots=feQWTkH2Uw&amp;sig=tv-1JwaMOygKGmFE2vM2FhJVS9o&amp;hl=en&amp;ei=5iWsTJCkBIPSsAPQwJ36Aw&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;redir_esc=y#v=onepage&amp;q&amp;f=false">Categories for the Working Mathematician</a>” by Sounders Mac Lane and it looks like this:</p><blockquote>All told, a monad in X is just a monoid in the category of endofunctors of X, with product × replaced by composition of endofunctors and unit set by the identity endofunctor.</blockquote><p>To understand the quote fully we would need to touch on Monoids, Monoidal Categories, Applicatives, Endofunctors and more. Hopefully, at the end of this post this sentence will no longer be a bunch of magical words glued up together.</p><h4>Monoid</h4><p>Let’s start with the Monoid. To make things simple I’ll try to use Scala as the language for the snippets wherever possible.</p><pre>trait Monoid[A] { <br>  // an identity element <br>  def id: A <br>  // an associative operation <br>  def op(x: A, y: A): A <br>}</pre><p>A Monoid is any type that has two functions: an operation taking two arguments of a type and returning a value of that type, and that also has a function which given a value returns the value of the same type without changing anything — the identity operation.</p><p>In Cats, you may be familiar with Semigroup which has a method combine:</p><pre>trait Semigroup[A] { <br>  def combine(x: A, y: A): A <br>}</pre><p>and Monoid is basically a Semigroup with additional method empty — our identity function.</p><pre>trait Monoid[A] extends Semigroup[A] {   <br>  def empty: A <br>}</pre><p>Please note that we need to follow the Category Theory rules here so the combine operation has to be associative and empty value should be an identity for our combine operation:</p><pre>combine(x, empty) = combine(empty, x) = x</pre><p>Basically if something is a Monoid we know we can combine it together to form bigger Monoids or write more generalized functions which will take something which can be composable instead of some concrete types.</p><p>Ok, so we know what a Monoid is and, simplifying things a bit, we can say that a Monoid is a set of objects and a method of combining them. The simplest examples would be:</p><ul><li>a String where the identity function return an empty String (“”) and our combine function would concat the two instances into one</li><li>an Int where the combine function would be + and identity would be 0</li><li>a List where the identity function would be List.empty or Nil and the combine function would concat two lists together (++)</li></ul><p>and so on.</p><p>For examples on this, I encourage you to see the Cats Monoid <a href="https://typelevel.org/cats/typeclasses/monoid.html">documentation</a> and great Monoid <a href="https://www.reddit.com/r/scala/comments/45gqpd/whats_a_monoid/">discussion on Reddit</a>.</p><p>From Category Theory point of view, Monoid is the simplest Category of one object with a structure. In other words it’s a Category with one object and an (endo-)functor going from that single object back to itself.</p><p>From Wikipedia:</p><blockquote>In <a href="https://en.wikipedia.org/wiki/Category_theory">category theory</a>, a <strong>monoid</strong> (or <strong>monoid object</strong>) (<em>M</em>, μ, η) in a <a href="https://en.wikipedia.org/wiki/Monoidal_category">monoidal category</a> (<strong>C</strong>, ⊗, <em>I</em>) is an object <em>M</em> together with two <a href="https://en.wikipedia.org/wiki/Morphism">morphisms</a></blockquote><blockquote>μ: <em>M</em> ⊗ <em>M</em> → <em>M</em> called <em>multiplication</em>,</blockquote><blockquote>η: <em>I</em> → <em>M</em> called <em>unit</em></blockquote><p>So, this is actually our def combine(x: A, y: A): A with def empty: A from the previous examples.</p><h4>Monoidal Category</h4><p>To explain what a Monad is, it’s usefull to know what a Monoidal Category is. If you look into Wikipedia definition for Monoid, you will see a strange (⊗) symbol called ‘multiplication’. The Monoidal Category is a Category in which you can “multiply” objects using some kind of a tensor product (⊗) and which also has a special object Ithat is the unit of multiplication.</p><p>In Monoidal Category, for any objects a and b there is a mapping a ⊗ b and this mapping is functorial in both arguments, which means that you can also “multiply” morphisms.</p><p>In other words, Monoidal Category is like our Monoid but applied to a Category itself.</p><p>Example of a Monoidal Category is a Category of Sets, Set is Monoidal Category with cartesian product x serving as a tensor product, and the singleton set 1 as the unit.</p><p>In a Set Category, the objects are Sets, morphisms are functions and where any particular one-element set serves as the unit.</p><p>So, similary to Monoid, Monoidal Category can be defined with:</p><ul><li><strong><em>•: S × S → S</em></strong></li><li><strong><em>e : 1 → S</em></strong></li></ul><p>where:</p><ul><li><strong><em>(a • b) • c = a • (b • c), for all a, b and c in S</em></strong></li><li><strong><em>e • a = a • e = a, for all a in S</em></strong></li></ul><h4>Functors in Monoidal Category</h4><p>If you follow along with me on a journey to better understand the Functional Programming concepts you may be already familiar with the idea of a Functor (if not you can read about it <a href="https://blog.softwaremill.com/functors-from-category-theory-into-cats-6aa1bdb641f2">here</a>). Once again, Functor is a mapping of objects and morphisms from one Category to another, that preserves composition and identity.</p><p>With our Monoidal Category example, we can imagine that we have a bunch of Functors mapping objects from Category C to Category of Sets. Our Functors would also form a Category, this category will consist of Functors as objects and morphisms between any two functors as natural transformations.</p><p>With our [C, Set] Functors Category our Functor should map the tensor product in C to the cartesian product in Set and also the unit I from C to the singleton set 1.</p><p>If we take Functors which preserve monoidal structure when mapping C to Set and we care only about one way transformation (from C to Set and not the opposite) we will end up with something called <strong>Lax Monoidal Functor </strong>— known as <strong>Applicative Functor</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/0*_e7sAW9dIx5vujd4" /><figcaption><a href="https://bartoszmilewski.com/2018/05/16/free-monoidal-functors-categorically/">https://bartoszmilewski.com/2018/05/16/free-monoidal-functors-categorically/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/77/0*zCGdGaqd6B6wbuTH" /><figcaption><a href="https://bartoszmilewski.com/2018/05/16/free-monoidal-functors-categorically/">https://bartoszmilewski.com/2018/05/16/free-monoidal-functors-categorically/</a></figcaption></figure><p>In Cats, our Lax Monoidal Functor looks like the following (<a href="https://typelevel.org/cats/typeclasses/applicative.html">example formulation via product</a>)</p><pre>trait Applicative[F[_]] extends Functor[F] { <br>  def product[A, B](fa: F[A], fb: F[B]): F[(A, B)] <br>  def pure[A](a: A): F[A] <br>}</pre><p>As you can see, the above definition is like 1 to 1 equivalent of what we get with Category Theory natural transformation definitions for Lax Monoidal Functor.</p><h4>Note on Endofunctors</h4><p>We have talked about Category of Functors and we know what is Monoidal Category. Now, you may ask what an Endofunctor is? Endofunctor is a mapping of objects and morphisms from one Category back to the same Category. Actually, all the Functors we are dealing with in functional programming are Endofunctors, this is because we are dealing with only one Category — Category of types.</p><h4>Monad</h4><p>If you have used Cats a bit, you may be aware at this point that Monad in Cats extends Applicative — in other words we are really close :) but we will get back to it later.</p><p>From Category Theory point of view Monad is an Endofunctor equipped with two natural transformations.</p><p>Functor T : X → X and:</p><ul><li><em>natural transformation</em><strong><em>: μ : T × T → T</em></strong></li><li><em>natural transformation</em>: <strong><em>η : I → T</em></strong></li></ul><p>where <strong><em>×</em></strong> means functor composition and<strong><em>I</em> </strong>is the identity endofunctor on <em>X.</em></p><p>Our <strong><em>μ </em></strong>and<strong><em> η </em></strong>are equivalent to flatMap and unit in Scala Cats.</p><p>Monad is an Applicative so its also a Functor. The distinctive feature of a Monad over the simple Functor concept is that we can build new structures from the pieces of original structure without nesting. With simple Functor with a map (fmap) we would simply nest each Functor (<em>F[F[A]]) </em>whereas with Monad we can flatten the resulting structure so we won’t end up with a structure reflecting the whole flow used to build it.</p><p>Similarly to Functor, we can think of a Monad as a container but a container for “side effects”, this means that with a Monad the side effects can be grouped into one structure where the order of effects is maintained. We will also have a “do nothing” side effect where when grouped with another side effect will be the same as the single side effect. Our flatMap (or Haskell join) is a grouping operation, where two side effects are grouped into one. And that’s basically all there is to a Monad.</p><p>To get the feeling for the sentence/blog title we are trying to decipher let’s move back a bit to our Monoid example, a Monoid is an object eg. M with two morphisms (as described earlier):</p><ul><li>μ: <em>M</em> ⊗ <em>M</em> → <em>M (multiplication, product)</em></li><li><em>I</em> → <em>M (unit)</em></li></ul><p>If we apply these laws to a Category of Endofunctors we will have product ⊗ replaced by composition of Endofunctors and unit being an identity Endofunctor.</p><p>Any Monad is by definition an Endofunctor, which also means it’s an object in the category of Endofunctors, where the monadic <strong><em>μ(flatMap) </em></strong>and<strong><em> η(unit) </em></strong>operators satisfy the definition of a Monoid in that particular Monoidal Category.</p><p>From the programming perspective Monoids are very good at combining things together. If you look into the examples with Ints, Strings, Lists etc. this becomes very clear. So with Monoid you can always compose two integers to get an integer but this does not always apply to functions, you cannot always compose two functions and get a function of the same type at the end (especially functions with higher kinded types). This is where Monad comes to the rescue.</p><p>In Cats, the Monad documentation states:</p><blockquote><em>Monad</em> extends the <a href="https://typelevel.org/cats/typeclasses/applicative.html"><em>Applicative</em></a> type class with a new function <em>flatten</em>. Flatten takes a value in a nested context (eg. <em>F[F[A]]</em> where F is the context) and “joins” the contexts together so that we have a single context (ie. <em>F[A]</em>).</blockquote><p>So our definition of Monad in programming world would be:</p><ul><li>def unit: A → F[A]</li><li>def map: F[A] → (A → B) → F[B]</li><li>def flatten: F[F[A]] → F[A]</li></ul><p>In Scala the Monad definition could look like the following:</p><pre>trait Monad[M[_]] {<br>  def unit[A](a: A): M[A]<br>  def flatMap[A, B](ma: M[A])(f: A =&gt; M[B]): M[B]<br>}</pre><p>So, like all Applicatives, we have a unit/pure function which takes a value and wraps it, as well as the map, which takes a function and applies it to an element returning a wrapped object. Now, in case of Monads, our special Functors, we have one additional method — flatten — which prevents us from nesting the side effects.</p><h4>Summary</h4><p>Simplifying the things greatly, we know that Functor is something that we can map with, we also know that in Functional Programming world we deal only with one Category, which is Category of Types and therefore we basically only deal with Endofunctors. Having a Category of Endofunctors we need to find a fixed Endofunctor which has some additional properties like the <strong><em>μ(flatMap) </em></strong>and<strong><em> η(unit) </em></strong>morphisms<strong><em>.</em></strong></p><p>Monad will be an Endofunctor which is able to turn two Endofunctors into one with μ: <strong>F ⊗ F→ F </strong>eg:<em> </em>F[F[T]] into F[T] and wrap itself up into an Endofunctor context with<strong> <em>I</em> → <em>M (unit)</em></strong><em>. </em>In other words we have a Monoid in a Category of Endofunctors.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b85bab43587b" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/monoid-in-the-category-of-endofunctors-b85bab43587b">Monoid in the Category of Endofunctors</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Functors — From Category Theory into Cats]]></title>
            <link>https://blog.softwaremill.com/functors-from-category-theory-into-cats-6aa1bdb641f2?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aa1bdb641f2</guid>
            <category><![CDATA[category-theory]]></category>
            <category><![CDATA[cats-library]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[functional-programming]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Tue, 19 Nov 2019 07:52:04 GMT</pubDate>
            <atom:updated>2019-11-19T07:52:04.573Z</atom:updated>
            <content:encoded><![CDATA[<h3>Functors — From Category Theory into Cats</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*uLQBSCuUnjcK6Mo_rAZWfg.jpeg" /><figcaption><a href="https://flickr.com/photos/skullyfm/">Franck Mahon</a> (modified) @ Flickr.com CC 2.0</figcaption></figure><p>Today we are going to disassemble Functor as another concept in Category Theory widely used in Functional Programming. The idea of a Functor seems trivial and many people when asked what it is, quickly respond that it’s ‘something that can be mapped over’ — this is true but it’s worth to know the full picture, what is actually mapped and how. This is where Category Theory (and of course Bartosz Milewski) comes to the rescue.</p><p>Once again I highly recommend Bartosz Milewski’s video series about <a href="https://www.youtube.com/watch?v=I8LbkfSSR58&amp;list=PLbgaMIhjbmEnaH_LTkxLI7FMa2HsnawM_">Category Theory for Programmers on YouTube</a> as well as his <a href="https://bartoszmilewski.com/2015/01/20/functors/">blog post</a> series about those concepts.</p><h4>What is a Functor</h4><p>Mathematically speaking, Functor is a very simple idea — it is mapping of objects from one Category to another Category. This sentence closely resembles the ‘something that can be mapped over’ idea repeated over and over in a programming world. But the story doesn’t end here.</p><p>We know that Category is objects and morphisms (arrows), we also know that Category is about composition and identity and Category’s structure is defined through composition. Functor allows us to map Categories and preserve the structure. To preserve the structure, we need to map not only the objects but the morphisms too. In other words, we can think of a Functor as a set of functions mapping a set of arrows from one Category to the set of arrows in another.</p><p>A Functor is kind of mapping of objects and morphisms that preserves composition and identity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*4GGHf1ezPoocOEWD-T-E2Q.png" /></figure><p>We have two Categories: A and B. In Category A we have two objects a and b with morphism f. Our Functor is a mapping of objects a and b to Fa and Fb and mapping of morphisms, in this case single morphism: f to Ff .</p><p>Continuing on a structure of Category, if you can compose two morphisms in one Category you can compose them in another when using a Functor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*4O4-LpTFpbvHmueArgBTHg.png" /></figure><p>We have a morphism in Category A between a and c (f · g) which is a composition of morphisms f and g, and in our Category B we have mapped the objects and morphisms correspondingly with a Functor so that</p><p>F(g · f) = Fg · Ff</p><p>One more requirement for our Functor is to preserve identity. In Category A we know that id a = a so our mapped version should look like the following:</p><p>Fid a = Fa or actually F(id(a)) but we can shorten it to Fa</p><p>Ok, so that’s the theory, now let’s see some examples.</p><h4>Example</h4><p>One of the simplest cases for a Functor explanation is something like Scala’s Option. Let’s say we want to map object of type A to an object of type Option[A]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*rRvc_lTULy3KMLhFY4opgw.png" /></figure><p>Staying for a little bit longer in Category Theory mode but with the Option example, let’s say that we have some objects and morphism of objects in one Category and we want to map them to another category with a Functor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*dCnJVBtduL1kxOZ6c-kjlg.png" /></figure><p>In the example above we are mapping object a to Option[a] and object b to an Option[b], we are also mapping a morphism f with a function fmap. The fmap name is used here on purpose as we all know that real Scala’s Option has no fmap function available (unless used with cats <em>implicit</em> Functor type class but we get back to this in a short while)</p><p>When mapping a concrete object withOption we can have two possible values, Some or None. So our Functor would need to execute the following:</p><p>Mapping None:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*WzqGPrgX02YxWhuXl4-NiQ.png" /></figure><p>Mapping Some(..):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*ufD4jV_m9AQazKvyX0qdVg.png" /></figure><p>Mapping morphism f:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*4jayN6HYSsgAJiB1iKBhxQ.png" /></figure><p>Preserving composition:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*bUUEU0Ojeqjyfn-fIZ0J9w.png" /></figure><p>Preserving identity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*rwMROV0fQ3FNFMMx0bGdDQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*CF_LrrX1C0ii5uDv4J_FIQ.png" /></figure><p>As you can see, our Functor is a set of functions mapping objects and morphism from one Category to another. The case where we map the whole morphism f between objects a and b is called <em>lifting</em> as you will see shortly in the Cats Functor section below.</p><p>In Scala, our fmap from above definitions would like a function of two arguments, the first one being the function we want to apply and the second one being a Functor instance (Option, List etc.). This of course doesn’t work like that in Scala. In Scala, by convention, map() is invoked upon an instance of the class and takes only one argument — the function we want to apply, the principles stay the same.</p><p>If we think about Functor examples, we know that Functor is something that can act on types, like our before-mentioned Option[A] or List[A], so except for a Functor being a `set of functions` we can also think about Functor as a type constructor that supports fmap operations, or, in addition, we can also think about Functor as a container.</p><p><em>Important note: do not confuse word </em><em>fmap used here with Scala’s </em><em>flatMap. </em><em>fmap name is borrowed here from Haskell and it’s about mapping, like </em><em>map in Scala is, and not about mapping and flattening the result like </em><em>flatMap.</em></p><h4>Cats Functor</h4><p>Among many type classes available in Cats for different concepts borrowed from Category Theory we have a Functor too.</p><blockquote>Functor is a type class that abstracts over type constructors that can be map‘ed over. Examples of such type constructors are List, Option, and Future.</blockquote><p>As you can see below, Functor map looks like the classic map function and lift will lift up a function call to its Functor equivalent (mapping morphisms mentioned earlier):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b72eab14dd53888ceb656250b5ccfba7/href">https://medium.com/media/b72eab14dd53888ceb656250b5ccfba7/href</a></iframe><p>For all the Category Theory lovers there is also an fmap implementation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45b957b13076528a78cb5bffee899ac9/href">https://medium.com/media/45b957b13076528a78cb5bffee899ac9/href</a></iframe><p>as you can see it has the same signature as our old’n good map function.</p><h4>Usage Examples</h4><p>Possibly most of us are used to Functor’s map function to work with effects. For example we can map on Future to apply some pure function to a value wrapped in our effect (Future) without needing to unwrap and wrap the value back into the effectful wrapper. In other words, we are working on a value without leaving the effect context. This is very useful but we can get even more from a simple Functor concept.</p><p><strong>Composing with nested data types</strong></p><p>The most common example for the Functor you will possibly see is about mapping on nested data types. With regular Scala approach, you would need to use map inside other map calls to execute some pure function on the value nested deep inside such a structure, for example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15f28e68e069179991217cf484d5bb96/href">https://medium.com/media/15f28e68e069179991217cf484d5bb96/href</a></iframe><p>Whereas with Functor available for Option and List with cats.implicits._ you can do the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ddae8d4e19c9f929b01adf62abb5db2/href">https://medium.com/media/2ddae8d4e19c9f929b01adf62abb5db2/href</a></iframe><p>or with Nested Functor wrapper:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97f7e2912c9d7bb7a2f10cced3ff2e21/href">https://medium.com/media/97f7e2912c9d7bb7a2f10cced3ff2e21/href</a></iframe><p>It becomes even more useful when the structure we want to use is more complex:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d521b55879591a2e46ec21b33514af5/href">https://medium.com/media/0d521b55879591a2e46ec21b33514af5/href</a></iframe><p>With a chain of compose calls and Functor implicits provided for List and Option we can execute functions on deep nested structures.</p><p>If the Functor implicit is not available for our structure we can create our own:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/112b98b002c5eed80fed310b7c57c4cb/href">https://medium.com/media/112b98b002c5eed80fed310b7c57c4cb/href</a></iframe><p>There is no Functor available for Map so with a simple type trick we can create one for ourselves and compose it to have a Functor which we can use for our complexContainer.</p><p>As you can see we can use Functors made available for you through cats.instances.*, create Functors for the existing container types in Scala as well as compose them together to work with more complex structures.</p><p>Watch this space for more Category Theory learning materials and Cats implementation examples.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aa1bdb641f2" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/functors-from-category-theory-into-cats-6aa1bdb641f2">Functors — From Category Theory into Cats</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCP Goodies Part 10— Measuring happiness with Google Cloud Vision API]]></title>
            <link>https://blog.softwaremill.com/gcp-goodies-part-10-measuring-happiness-with-google-cloud-vision-api-d70ac00de712?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/d70ac00de712</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[google-data-studio]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Fri, 08 Nov 2019 08:27:45 GMT</pubDate>
            <atom:updated>2019-11-08T08:27:45.994Z</atom:updated>
            <content:encoded><![CDATA[<h4>An attempt to measure happiness in a fully-remote company using Google Cloud Platform.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BVUvMEgkINDcG8BeiKfbuw.jpeg" /><figcaption>Mike Hoff @ Flickr.com CC 2.0</figcaption></figure><h4>The story</h4><p><a href="http://www.softwaremill.com/">Softwaremill.com</a> is a fully remote company. We do work from our homes and see each other live from time to time in many different places around Poland. Every day though, we see each other and talk for a short while on a video conference using our day-to-day tools.</p><p>This is the last part of the “GCP Goodies” series and to make it a bit more fun for myself and to utilize some more services Google Cloud Platform offers, I have created a simple solution as an attempt to measure happiness at my workplace :).</p><p>You can read more about our company at excellent blog post series: “<a href="https://blog.softwaremill.com/how-we-built-a-fully-remote-self-organizing-company-before-it-was-cool-290bac1b0b98">How we built a fully remote, self-organizing company before it was cool</a>”</p><h4>The tools we use</h4><p>For our remote work, communication with the clients, as well as between ourselves, we mostly use Slack (for instant messaging and one-to-one video calls), BigBlueButton for daily video meetings and TeamSpeak for voice communication. As mentioned earlier, we meet almost every day for a short company video meetings where we use BigBlueButton for video only and TeamSpeak for voice. The meeting takes place at the same time every day and lasts, more or less, 15 minutes.</p><p>Besides standard chit-chat talk about current mood, weather etc., across the week we have a special question prepared for each day everyone may answer. The questions are usually non-work related and their purpose is to know each other better, mostly they are about people and their personal interest, things we do outside of work etc.</p><h4>Happiness measurement attempt</h4><p>Please, bear in mind that this blog post is still more about Google Cloud Platform than about any serious attempt to happiness measurement using some computing resources, but still, the results posted at the end of this article look promising ;).</p><p>The idea for this blog post has been simple. Every day, at specific time, we are going to launch a virtual machine on GCP and log into our video conference software (BigBlueButton) as some virtual user. Once our virtual user ‘sees’ all the people on a meeting, it will take a screenshot of all the faces every few minutes and save it to the Google Storage Service. Once we collect the screenshots for the whole week we will run a Python script to analyze how many people were smiling each day at our meetings.</p><h4>Setup</h4><p>Because BigBlueButton software uses flash on its website the handling of the web application with automated scripts was a bit more challenging than usual. At the end, I have decided to run the Chrome web browser on a Windows platform and use Python with selenium to do all the operations needed to log in into the meeting, create the screenshot, process it a bit and upload to Google Storage service.</p><p>Let’s start with creating Windows based virtual machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*06TskbRWiOhxDpq55dCSjQ.png" /></figure><p>You can set windows password after the instance creation by clicking on an instance name and navigating to its details page.</p><p>To enable remote desktop connection to our VM we need to open the port 5901 for a TCP traffic. Navigate to VPC Network -&gt; Firewall rules and add the following details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*8jEw8ek4_7VxPcWhUwtohg.png" /></figure><p>The next step is to create a service account which will be used to communicate with other GCP services from the Windows VM (like sending screenshots to the bucket for further processing).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*W--jXdxSCc2uIRErRXF5Hg.png" /></figure><p>Click Create and save the generated JSON file somewhere for later use.</p><h4><strong>Virtual Video Conference User</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*h_ATTB5a6AvalIvW3XUIlA.png" /></figure><p>Some preparations need to be done on the VM machine so the Selenium Python script will be able to enable flash and log in into our conference room. I have used Chrome RDP for Google Cloud Platform plugin to log into the server and install the needed components.</p><p>I highly recommend setting up static IP address and assign it to the virtual machine you will use. This way you will always connect to the same IP address and will have one less thing to remember when accessing the VM.</p><p>Setting up static IP address can be done under Networking -&gt; VPC Networks -&gt; Static IP Addresses</p><p>To install software needed for this tutorial on Windows machine, I highly recommend using choco.</p><p>We need to install Python and Chrome browser. Start by installing choco and then install Python in version 3.7:</p><pre>@&quot;%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe&quot; -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command &quot;iex ((New-Object System.Net.WebClient).DownloadString(&#39;https://chocolatey.org/install.ps1&#39;))&quot; &amp;&amp; SET &quot;PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin&quot;</pre><p>and then install Python (please note that you need to install version 3.7 or less, otherwise you will have problems installing Pillow library later on):</p><pre>choco install python --version 3.7<br>refreshenv</pre><p>If you want to do that from the powershell mode, simply enter powershell in the command line and you will switch to PS mode. Therefreshenv command refreshes your console environment so you should be able to use your newly installed software straight away.</p><p>choco installed with the above command was in version 0.10.15.</p><p>When in powershell mode, execute the command found <a href="https://www.snel.com/support/install-chrome-in-windows-server/">here</a> to install Chrome on windows server.</p><pre>$LocalTempDir = $env:TEMP; $ChromeInstaller = &quot;ChromeInstaller.exe&quot;; (new-object    System.Net.WebClient).DownloadFile(&#39;http://dl.google.com/chrome/install/375.126/chrome_installer.exe&#39;, &quot;$LocalTempDir\$ChromeInstaller&quot;); &amp; &quot;$LocalTempDir\$ChromeInstaller&quot; /silent /install; $Process2Monitor =  &quot;ChromeInstaller&quot;; Do { $ProcessesFound = Get-Process | ?{$Process2Monitor -contains $_.Name} | Select-Object -ExpandProperty Name; If ($ProcessesFound) { &quot;Still running: $($ProcessesFound -join &#39;, &#39;)&quot; | Write-Host; Start-Sleep -Seconds 2 } else { rm &quot;$LocalTempDir\$ChromeInstaller&quot; -ErrorAction SilentlyContinue -Verbose } } Until (!$ProcessesFound)</pre><p>Once we have the software we need, we have to install python libraries used by our script, after choco Python installation, use pip or pip3 to install the libraries:</p><pre>pip install selenium<br>pip install pynput<br>pip install google-api-python-client<br>pip install google-cloud-storage<br>pip install Pillow</pre><p>Now we can test our script. I have created a windows shell script which simply navigates first to the correct directory and then executes our python script itself:</p><pre>cd C:\Users\krzysztof_grajek<br>python screen.py</pre><p>krzysztof_grajek directory is a home directory created by GCP when building the windows VM for us.</p><p>Thescreen.py script is self explanatory, the important bits are the key presses needed to enable flash in a browser for the web page we want to use:</p><pre>actions = actions.send_keys(Keys.TAB * 21)</pre><p>Similar one needs to be executed to log in once the flash takes over the control of the web page. In such cases, tab navigation and hitting enter button with selenium is your friend :) — dirty hack but works. The code is pasted below, you can find the sources for all the scripts in the repository for part-10 of this tutorial series <a href="https://github.com/softwaremill/gcp-goodies">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/721d89afee48c7e4b70fc7381adcd085/href">https://medium.com/media/721d89afee48c7e4b70fc7381adcd085/href</a></iframe><p>The last part which needs to be done on the windows side of things, is the scheduled job which will run every couple of minutes and start up our ps1 script. To do that, enter the following in the command line console:</p><pre>schtasks /create /tn HappyTask /tr &quot;powershell -file C:\Users\krzysztof_grajek\runhappy.ps1&quot; /sc minute /mo 2 /ru System</pre><p>That basically means, create a task executed every 2 minutes which will run the runhappy.ps1 shell script as a System user.</p><p>The ps1 script will start up the process handled by Python — we will start the Chrome browser, enable flash for our web page, navigate there, log in and take the screenshot. Once the screenshot is ready, we resize it by 100% in each direction and upload it to Google Storage service.</p><p>Please note that you need to have the credentials json file available on windows machine to send the data to the Google Storage bucket.</p><h4>Scheduling Server work</h4><p>Anytime the server VM is running the scheduled task created there will launch the script and send the screenshots to our bucket. There is only one small problem, the Windows machine is not cheap and we don’t need it to be run all day and send screenshots of the meeting when nobody is in there. What we want is to launch the VM when the meeting starts and stop it when the meeting ends. As the meetings at SoftwareMill are always at the same time of the day that job is easy.</p><p>The part for Scheduling Server Work is almost the same as the tutorial on Google Cloud website itself:</p><p><a href="https://cloud.google.com/scheduler/docs/start-and-stop-compute-engine-instances-on-a-schedule">Scheduling compute instances with Cloud Scheduler</a></p><p>We will create a pair of Cloud Functions to start and stop our VM and then execute those functions with Cloud Scheduler service.</p><p>First, we need to distinguish which instance we want to start/stop. For this purpose we will label our instance so we won’t accidentally stop or start other instances we have.</p><p>Go to VM details and add label to our instance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*a7RkQ0cwfyDNoLklrTNWsw.png" /></figure><p>Navigate to Compute -&gt; Cloud Functions and add new functions for starting and stopping our VM instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*CT63F6xckIoeisuE-1XjKA.png" /></figure><p>index.js content for starting the instance:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cac05dc56ea4013bf8c58c101e383abd/href">https://medium.com/media/cac05dc56ea4013bf8c58c101e383abd/href</a></iframe><p>package.json content for starting the instance:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef7fc44234c71a9d8151f740bfa3c5db/href">https://medium.com/media/ef7fc44234c71a9d8151f740bfa3c5db/href</a></iframe><p>Similarly, do the same for stopping the instance function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*Fy4qU_uV3Dpyg8RSwXoc7w.png" /></figure><p>index.js content:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/505fc86aa2499694db4ec28e8e8296e7/href">https://medium.com/media/505fc86aa2499694db4ec28e8e8296e7/href</a></iframe><p>package.json content:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf09cc489ab9d307b8069d669dfd8f13/href">https://medium.com/media/cf09cc489ab9d307b8069d669dfd8f13/href</a></iframe><p>Once we have the functions ready we can test them by sending base64 encoded json with the data we want.</p><p>Prepare payload to test the function:</p><pre>echo -n ‘{“zone”:”us-central1-a”, “label”:”env=dev”}’ | base64</pre><pre>{“data”:”eyJ6b25lIjoidXMtY2VudHJhbDEtYSIsICJsYWJlbCI6ImVudj1kZXYifQ==”}</pre><p>Testing start/stop functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iJqJVWqX7ijWqhTsRuqhFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*M00Bw_pYitBG0VP93BcDSA.png" /></figure><p>You should see on the Compute VM section your instance being stopped/started accordingly.</p><p>Once we have the start/stop functions working and we have tested that they indeed stop and start our instance, we can create a scheduler job to do that for us, let’s say — every day from Monday to Friday at specific hours.</p><p>Set up Cloud Scheduler job to start/stop an instance automatically:</p><p>Navigate to Tools -&gt; Cloud Scheduler to create new job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*KlAOqw5t48eX3n6rt9tFcQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*LwJNpun9zB4LVRrGg2i8lA.png" /></figure><p>That will call our Cloud function every day from Monday to Friday at 10 am.</p><p>shutdown job:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*CX21LAxDJ64IOsg6yylAug.png" /></figure><p>Will do the same as the previous job but for the stop function, half an hour later.</p><p>You can test it with RUN NOW button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OgJbjLxqf75Hr1WRvl8-rw.png" /></figure><h4>Processing Results</h4><p>At this point we should have everything set up and working to gather the data for us. The data are simple screenshots taken every few minutes during our video conference meetings throughout the week.</p><p>We will utilize one more Google Cloud service we haven’t used before on the GCP Goodies series — that will be Google Cloud Vision API.</p><p>With this simple service we will send every screenshot we have collected during the experiment, and for each face detected on the screenshot we will determine how happy the person detected was at that time.</p><p>The happiness levels are described by Google using values ranging from unknown to very likely: &#39;UNKNOWN&#39;, &#39;VERY_UNLIKELY&#39;, &#39;UNLIKELY&#39;, &#39;POSSIBLE&#39;, &#39;LIKELY&#39;, &#39;VERY_LIKELY&#39;.</p><p>Once we gather all the results we will flatten them out and store them in a single CSV file, which we will process further in Google Data Reporting service.</p><p>First we get all the files stored in our bucket:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ed1fe18e5ce64d00d5af68e8da92c35/href">https://medium.com/media/3ed1fe18e5ce64d00d5af68e8da92c35/href</a></iframe><p>and for each file we get the likelihoods of joy and group them by day:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/581c4a9c54db79c91f668ff8117abe13/href">https://medium.com/media/581c4a9c54db79c91f668ff8117abe13/href</a></iframe><p>Detecting faces and returning the joy likelihoods is just a single call to the Google APIs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c29c82c3f88d4c8017ffc992e6db32d/href">https://medium.com/media/2c29c82c3f88d4c8017ffc992e6db32d/href</a></iframe><p>We don’t need to download the files and send them again to another Google service (like Google Vision), we just need a path which we can use to point Google Vision API where the image we want to process is stored.</p><p>Please note, before you run the script, you need to pass the credentials somehow so that Python Google libraries we use will know you are the authenticated user and have access to the resources. For the purpose of this tutorial, just execute the following before running the script on your local machine:</p><pre>export GOOGLE_APPLICATION_CREDENTIALS=”/path/to/your/happy.json</pre><p>assuming that happy.json is the service account credentials file you had created earlier.</p><h4>Results</h4><p>After running the script you should be able to find a newly created .csv file in the same directory. The script contains the number of occurrences for specific joy level and day, e.g:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9bed123ff060e3bd3d4a6cc583b56c7/href">https://medium.com/media/a9bed123ff060e3bd3d4a6cc583b56c7/href</a></iframe><p>In other words, on the 22nd of October there was 314 detected faces for the whole meeting with level 1 (UKNOWN) and 11 occurrences of level 5 — which is joy_likelihood VERY_LIKELY .</p><p>Open up <a href="https://datastudio.google.com/u/0/navigation/reporting">Google Data Studio Reporting</a> and select Create and then Data source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*Bj6HdCIgyvKNaWe2NhudKA.png" /></figure><p>From the list of available options select File Upload</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*LvJSH4sZw5B6oSKTbZg5DQ.png" /></figure><p>Select the file we have just created and wait for the processing to finish:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hMvxSmxxfttyMj8VFI5ynA.png" /></figure><p>Click Connect in the top-right corner:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*e6eTGpZmO0Wfgo2Bvc2HVA.png" /></figure><p>and then, Create Report:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*_a5PxxfVKpArpguxBVsctQ.png" /></figure><p>Delete everything that has been added automatically to our report and add a new Bar chart</p><p>Modify the chart properties:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*D4B4EwD-_YsTsnhBh846yQ.png" /></figure><p>Set as a Breakdown Dimension our level to see the data grouped by Day and Level values. Then change the default Sort to use Day dimension in ascending way.</p><p>After playing around a bit you should end up with something similar to the graph below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1mIxNXCOfDAdngDYBrSXQ.png" /></figure><p>The levels 3,4 and 5 are &#39;POSSIBLE&#39;, &#39;LIKELY&#39;, &#39;VERY_LIKELY&#39; correspondingly, therefore sum of those will show us the most happy day for the total of 6 days of this experiment.</p><p>And the winner is.. <strong>Thursday, 24th of October 2019.</strong></p><p>To be frank, I was expecting that the clear winner will be <strong>Friday 25th</strong> but unfortunately it scored less, the reason for it, I believe, was the unfortunate random question we all can answer during each of those meetings. The one for Friday 25th of October was “<strong><em>What irritates you the most?</em></strong>”, whereas the Thursday one was “<strong><em>Are you fulfilled in life?</em></strong>” As you can see, the questions themselves and the topics we discuss everyday vary a lot :)</p><p>As a bonus I will post a couple of the screenshots created by our Google-based Happiness Monitoring Solution for you to see how happy we are :D</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YBJnGjzZDmylJc4mTQpS1g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAdxjSBvSDJ6gevyaexxaQ.png" /></figure><p>If you find our company interesting and would like to see yourself on one of our ‘happy meetings` ;), please do not hesitate to check our currently open positions at: <a href="https://softwaremill.com/join-us/">https://softwaremill.com/join-us/</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d70ac00de712" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/gcp-goodies-part-10-measuring-happiness-with-google-cloud-vision-api-d70ac00de712">GCP Goodies Part 10— Measuring happiness with Google Cloud Vision API</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kleisli Category — From theory to Cats]]></title>
            <link>https://blog.softwaremill.com/kleisli-category-from-theory-to-cats-fbd140bf396e?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/fbd140bf396e</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[cats-library]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Mon, 04 Nov 2019 08:31:57 GMT</pubDate>
            <atom:updated>2019-11-04T08:31:57.455Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kleisli Category — From theory to Cats</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*HcIVnhWSvWVXmGFPf5-ckg.jpeg" /><figcaption><a href="https://flickr.com/photos/skullyfm/">Franck Mahon</a> @ Flickr.com CC 2.0</figcaption></figure><h3>Kleisli Category — the journey</h3><p>From <a href="https://en.wikipedia.org/wiki/Kleisli_category">wikipedia.org</a></p><blockquote>In <a href="https://en.wikipedia.org/wiki/Category_theory">category theory</a>, a <strong>Kleisli category</strong> is a <a href="https://en.wikipedia.org/wiki/Category_(mathematics)">category</a> naturally associated to any <a href="https://en.wikipedia.org/wiki/Monad_(category_theory)">monad</a> <em>T</em></blockquote><p>So far so good, but let’s break it down into pieces with a few examples on the way so we can go back to this sentence at the end of this post and understand exactly what it means.</p><p>Let’s suppose we want to handle the logs for our method executions so we can handle them as one single list of logs (by the way, the log example leading us to a category is borrowed from great video <a href="https://www.youtube.com/watch?v=I8LbkfSSR58&amp;list=PLbgaMIhjbmEnaH_LTkxLI7FMa2HsnawM_">lecture series</a> by Bartosz Milewski).</p><p>All the code samples are available in GitHub <a href="https://github.com/softwaremill/kleisli-example">repository</a>.</p><h4>Solution 1</h4><p>The simplest solution would be to store the logs in a global variable — I know, you have just shrugged but no worries, it’s just for demonstration purposes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2fd796a73f0fd0d44a6356fb0f24037d/href">https://medium.com/media/2fd796a73f0fd0d44a6356fb0f24037d/href</a></iframe><p>With logAcc as our log accumulator we can execute simple_1 and simple_2 functions and keep the logs nicely organized. But.. we all know it’s terrible and we all know that if you are reading this you want to see some functional way of solving things :)</p><h4>Solution 2</h4><p>The second approach — we pass the logs from one function execution to the next as a parameter:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/217b373418193304804ab31fc5d61bb0/href">https://medium.com/media/217b373418193304804ab31fc5d61bb0/href</a></iframe><p>This is better, the functions are pure and, at least, we know from the signature that the negating function does something else too. We can pass the second element of resulting tuple to the next function and have our concatenated log at the end. This of course is bad, the function uses some logic to handle logs inside which breaks single responsibility principle, and the function is difficult to memoize which should switch on a red light when thinking about pure functions. Let’s try something else..</p><h4>Solution 3</h4><p>We can decide to return a tuple like we did in the second example but we take just a Boolean parameter. We defer the log concatenation to the point where we call our functions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ef08db0ca9b2ec5227f568249e2bdc2/href">https://medium.com/media/7ef08db0ca9b2ec5227f568249e2bdc2/href</a></iframe><p>Now, we have pure functions and some log ‘context’ we pass around and handle at the end. All is good, except the way we handle the resulting log, we need to keep the results from each execution, concatenate the logs etc., this can get difficult. In other words, we need a better way of composing those functions.</p><h4>Solution 4</h4><p>With function composition, we take two composable functions and create a single function out of it. If you are interested in Category Theory, you have probably already seen the example of composing function A -&gt; B with function B -&gt; C which results in function A -&gt; C. Composition is fundamental to functional programming. Having three types A, B and C we can create our compose function like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51043d877c7367022cce0cea6f273dbc/href">https://medium.com/media/51043d877c7367022cce0cea6f273dbc/href</a></iframe><p>The function above takes two functions as arguments and returns another function of type A to C.</p><p>With our Tuple example and ‘log context’ we can create our own composing function. The function will take two functions taking Boolean and returning (Boolean ,String ) and it will convert those to a single function which takes a Boolean and returns (Boolean ,String ). For this example I have generalized Boolean to a generic type A:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd0cbe2090a65602776887e3dead246d/href">https://medium.com/media/bd0cbe2090a65602776887e3dead246d/href</a></iframe><p>Please note that string concatenation operation is happening within our composition — composeT function. We know, by looking at our function that it’s a simple function composition and it’s associative, and having some knowledge about String, we know that composeT should be associative as String concatenation is associative.</p><p>If we have associativity, let’s check if we can create an identity. To create an identity function we need to have a function which will do nothing to our log. In other words we need a function which will take a Boolean and return the same Boolean value with an empty String as a tuple:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9051e725d0503a4fbbbd6d7a03665bf6/href">https://medium.com/media/9051e725d0503a4fbbbd6d7a03665bf6/href</a></iframe><p>Easy!</p><h4>Associativity? Identity? Does it ring any bells?</h4><p>At this point we know we have associativity and identity — in other words we have a <strong>Category</strong>. In our Category the objects are Scala types and the arrows are A -&gt; (B, String) (instead of simple A -&gt; B arrows). The fact that the arrows are not simple transformation from A -&gt; B makes them so called <strong>Kleisli arrows</strong>. Kleisli arrows can work with many different types, not only a tuple of some type B and String, they are defined for the types we impose as few conditions as possible, in other words for monadic types. This is where <a href="https://typelevel.org/cats/datatypes/kleisli.html">Cats</a> definition of Kleisli comes in:</p><blockquote>Kleisli enables composition of functions that return a monadic value, for instance an Option[Int] or a Either[String, List[Double]], without having functions take an Option or Either as a parameter, which can be strange and unwieldy.</blockquote><h4>So, how this is usefull?</h4><p>Kleisli is all about composing. Let’s say we have a 3 functions we want to compose:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb517e9c4e12d14d236b4161f73efc42/href">https://medium.com/media/eb517e9c4e12d14d236b4161f73efc42/href</a></iframe><p>The simplest approach would be to call it one by one and pass the result from one to the other like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/363c5dc57b351de685f071bbb7f80ff2/href">https://medium.com/media/363c5dc57b351de685f071bbb7f80ff2/href</a></iframe><p>Or, you could just inline those calls like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1dcfcf782c55705c4fbcc4e9c3e28653/href">https://medium.com/media/1dcfcf782c55705c4fbcc4e9c3e28653/href</a></iframe><p>In Scala we have functions like compose and andThen for doing just that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5617c503c53849cdab78d93d64dac70e/href">https://medium.com/media/5617c503c53849cdab78d93d64dac70e/href</a></iframe><p>The problem with all the above examples is that we need to match the inputs of one function with the outputs of another to make this all work. If some of the outputs will be a wrapper around some type (e.g. Future or Cats IO) we will get into trouble quickly.</p><p>Let’s wrap some values into something useful so we can see how you would typically handle that too:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98efe3c5eaed020eb86ba73b4d1bf73e/href">https://medium.com/media/98efe3c5eaed020eb86ba73b4d1bf73e/href</a></iframe><p>Now it’s clear that we cannot just pass return value from generate to process etc. But we know that IO is a Monad and we can easily flatMap on it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfb9426c53e241379e2917dbe03a4437/href">https://medium.com/media/cfb9426c53e241379e2917dbe03a4437/href</a></iframe><p>or with a cleaner approach — for comprehension:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11c193c66953004bf76cb5aeaab97295/href">https://medium.com/media/11c193c66953004bf76cb5aeaab97295/href</a></iframe><p>With Kleisli we can do something similar but in many cases in more clear and readable way. Let’s start with the most verbose version:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7224ae4fdc522101f0328215b9e52b48/href">https://medium.com/media/7224ae4fdc522101f0328215b9e52b48/href</a></iframe><p>Here, we wrap each function in Kleisli(..) type and then use the same andThen mechanism we have used with functions matching inputs and outputs (<em>Side note: Cats library actually overrides Scala </em><em>andThen and Scalaz uses </em><em>andThenK for this purpose</em>).</p><p>So far so good but we can do better :). There is an operator in Cats for working with Kleisli which you can use to get much cleaner code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25073920ba17a3fd9dff19dc1f77eed4/href">https://medium.com/media/25073920ba17a3fd9dff19dc1f77eed4/href</a></iframe><p>In addition, if you are using andThen approach, you don’t need to wrap in Kleisli all of the functions you want to use. The first one is enough, e.g:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30e9264cb8c44eb9fcea7b2e16b3f6b9/href">https://medium.com/media/30e9264cb8c44eb9fcea7b2e16b3f6b9/href</a></iframe><p>I hope that throughout this tutorial you could see why Kleisli Category is a category naturally associated to any Monad T :). We went through a simple example with a tuple to get into Category and Kleisli arrows, and from that to the Cats library definition and usage.</p><p>Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fbd140bf396e" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/kleisli-category-from-theory-to-cats-fbd140bf396e">Kleisli Category — From theory to Cats</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCP Goodies Part 9 — Time series data handling and visualization]]></title>
            <link>https://blog.softwaremill.com/gcp-goodies-part-9-time-series-data-handling-and-visualization-a7f504de9e88?source=rss-5a68e9b7a9b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7f504de9e88</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-data-studio]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Krzysztof Grajek]]></dc:creator>
            <pubDate>Thu, 24 Oct 2019 08:12:54 GMT</pubDate>
            <atom:updated>2019-10-24T08:12:54.424Z</atom:updated>
            <content:encoded><![CDATA[<h3>GCP Goodies Part 9 — Time series data handling and visualization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wOWwrOxg9khNN-QDEmkIUA.png" /></figure><p>This time we are going to utilize a couple of GCP services to display a report. The whole point of this tutorial is to use BigQuery with DataStudio but to make things a little bit more interesting and challenging at the same time we are going to push the data continuously to the database using simple web application.</p><h4>Setup Overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*wDfq23Ip5KcGhdXBrXDfdA.png" /><figcaption>Created with draw.io</figcaption></figure><p>The report itself will be build with Google Data Studio but the data for the report will travel from the CSV file read at random positions by the web application on Kubernetes cluster. The web application will push the data to MySQL database and that data will be available to BigQuery. Data read by BigQuery will become the source of truth for the report. Throughout this tutorial you will be able to see how easy it is to create this flow and how powerful DataStudio with BigQuery can become.</p><h4>Creating GCP Infrastructure</h4><p>As usual, we are going to create a Kubernetes cluster where we deploy our web application. The sources for the application which you can build and deploy are available as part of the whole GCP Goodies series <a href="https://github.com/softwaremill/gcp-goodies">here</a>.</p><p>Open up a console and navigate to part-9 directory. We need to set some variables first, like the name of the cluster to be created or the zone where we want to create it.</p><p>Important note: Select your zone wisely, as for the cluster itself it’s not of much importance the zone in which our SQL Instance will operate; it’s very important as for some of them its not possible to use BigQuery federated queries. In other words, creating SQL Instance in wrong zone can result in inability to use BigQuery with it.</p><pre>NAME=stackdriver-test-9<br>ZONE=us-west2-a <br>gcloud auth login<br>gcloud config set compute/zone $ZONE<br>gcloud config set project softwaremill-playground-2</pre><p>Create a 2-node cluster:</p><pre>gcloud container clusters create $NAME --num-nodes=2 --scopes=<a href="https://www.googleapis.com/auth/cloud_debugger">https://www.googleapis.com/auth/cloud</a>-platform</pre><p>Now it’s time to create our SQL instance. Navigate to Storage -&gt; SQL and select Create Instance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*GnZjwRVz40NFP5kukEr_sQ.png" /></figure><p>Set up the details like name, instance type, region and leave the rest with defaults. Make sure that the region you choose is allowed in BigQuery CloudSQL Federated Query mechanism as mentioned earlier. Cloud SQL is supported in all BigQuery regions: <a href="https://cloud.google.com/bigquery/docs/locations">https://cloud.google.com/bigquery/docs/locations</a>.</p><p>It would be best if you choose the same region/zone as you have chosen for the cluster in the previous step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*mRV1mXahB-YEMS8hc5wp0A.png" /></figure><p>Once the instance gets created, click on its name and navigate to the Users tab to create a new user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*dOGugW9UgiXOnprsdxXARw.png" /></figure><p>Write down the Username and Password, we will need that for later when creating secrets on Kubernetes cluster.</p><p>The next step would be to create the database itself, navigate to Databases tab and click Create Database button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*8vbdUihZH2WYzVTEH27kvw.png" /></figure><p>Now we need to create a service account which we will use to connect from our pod on GKE to SQL Instance.</p><p>Navigate to APIs &amp; Services -&gt; Credentials and click on Create Credentials -&gt; Service Account Key</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*AvxwYMjfkzK2WAGfPm2cDw.png" /></figure><p>Select New Service Account, add name and select role to be Cloud SQL Client or Cloud SQL Admin. Click Create to download the credentials json file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*Soep_q7UTWotd7BrG04QFw.png" /></figure><h4>Data set</h4><p>To make things a little bit easier on the web application side we are going to use already available time series data set. This will allow us to build graphs from data which makes some sense and it’s not completely randomly generated. Our data set is called Occupancy Detection Dataset and it describes measurements of a room. Used with machine learning models the objective is to predict whether or not the room is occupied.</p><p>There are 20,560 one-minute observations taken over the period of a few weeks. There are attributes like temperature, light, co2, humidity etc.</p><p>The source for the data is credited to Luis Candanedo from UMONS.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2e47f395a38b2439f57c4c171de6172/href">https://medium.com/media/d2e47f395a38b2439f57c4c171de6172/href</a></iframe><p>In our web application, the scheduler pings an Akka actor to push a randomly picked row from this data set to the database:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/984a5f137676b5aecbc21d7730e190fe/href">https://medium.com/media/984a5f137676b5aecbc21d7730e190fe/href</a></iframe><p>This way, when creating the report we can use Refresh functionality to see how the data changes over time while building the report.</p><p>Download the <a href="http://archive.ics.uci.edu/ml/datasets/Occupancy+Detection+">data set</a> and extract it to some location on your local machine. We will need to push it later to our Kubernetes cluster for use by the web application.</p><h4>Releasing the application</h4><p>At this point you should have Kubernetes cluster and Cloud SQL Instance running, database and database user created, google credentials file for Cloud SQL and data set downloaded. It’s time to play with kubectl command.</p><pre>gcloud compute instances list<br>gcloud container clusters get-credentials $NAME<br>kubectl get po</pre><p>For our Kubernetes pod to be able to connect to Cloud SQL instance we need to create two separate secrets, one holding the credentials to the database and one storing the credentials.json file for our service account.</p><p>Create a file with the following content (changing the values to the ones you have used when creating the user and the database earlier)</p><pre>username=some_username<br>password=some_password</pre><p><em>(there is a file called </em><em>credentials.txt inside the </em><em>conf directory of the </em><em>part-9 project)</em></p><p>Create secrets on the cluster itself:</p><pre>kubectl create secret generic cloudsql-instance-credentials --from-file=credentials.json=/path/to/your/credentials.json</pre><pre>kubectl create secret generic cloudsql-db-credentials --from-env-file ./conf/credentials.txt</pre><p>Get the database connection string:</p><pre>gcloud beta sql instances describe datareporting | grep connection</pre><p>and change kubernetes-manifest.yaml under command section, e.g:</p><pre>command: [&quot;/cloud_sql_proxy&quot;,<br>          &quot;-instances=softwaremill-playground-2:us-west1:datastorage=tcp:3306&quot;,</pre><p>Create config map to store our data read by the web application every second at random position and insert it into the database:</p><pre>kubectl create configmap config --from-file /path/to/your/datatraining.txt</pre><p>Last, change the database name in application.conf to the one you have created earlier.</p><p>Build the application with sbt clean release and note the version generated. Use that version to change value in kubernetes-manifest.yaml under theimage path property.</p><p>Release the app:</p><pre>kubectl create -f conf/kubernetes-manifest.yaml<br></pre><h4>Querying data with BigQuery</h4><p>Now the fun part begins. We are going to connect BigQuery with our Cloud SQL Instance. To do that we will use Federated Query functionality of BigQuery to run queries against our sql instance.</p><p>Navigate to Big Data -&gt; Big Query and then ADD DATA and Create Connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*cJ6fHtfN8B-ZwdxHM0UrCg.png" /></figure><p>We are going to use Cloud SQL Federated Queries to play with our data set.</p><p><a href="https://cloud.google.com/bigquery/docs/cloud-sql-federated-queries">Cloud SQL federated queries | BigQuery | Google Cloud</a></p><p>Fill up the connection details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*TE0umRqSnCOqxq3cU-fiPg.png" /></figure><p>The instance id will be the same as the one used in kubernetes-manifest file.</p><p>Select your external connection, click Query Connection button and example BigQuery sql will be populated for you, click Run</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bdhd76p_T21Ik2tAvXujlQ.png" /></figure><p>You can query all the records stored so far in roomdata table with simple:</p><pre>SELECT * FROM EXTERNAL_QUERY(“softwaremill-playground-2.us-west2.datareporting”, “SELECT * FROM roomdata;”);</pre><p>You should see the records available in BigQuery query results pane:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*vkc4rv4oEjil20r3X_aLrw.png" /></figure><p>Now we can start creating our first report, click Explore with Data Studio button and start adding charts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*AUAWTyugt9PrMDnnVuE5Sw.png" /></figure><p>First chart will be simple line graph, pick it up from Add a chart drop down and set the dimensions like the ones below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*j6sJdh_FDUPel80ouYZWUw.png" /></figure><p>You should see lines for each metric being added to the graph as you follow along adding the metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V77_hO3aJl5muYUkUG1xqA.png" /></figure><p>There is a ton of functionalities coming out of the box in graphs built with data studio, you can zoom on the data range, select specific metrics you want to see and many more. Play around it to see how cool the data studio charts can be.</p><p>The next kind of an item we are going to add are Gauges, Gauge is a community widget so you select it using the drop down with boxes icon on the right of the Add a chart selection, pick one, change its size on the dashboard and set the dimensions as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*XjsCzu1nf-wN1W7EjRgDyQ.png" /></figure><p>Just to make things a little more interesting, the last item we are going to add will be table, pick one from the Add a chart drop down selection and fill the metrics and dimensions as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*v2IHfBiE_8CU7HmZlvUTag.png" /></figure><p>You can set the number of rows per page, create summary row, sort the data etc.</p><p>Final version of our Data Studio report looks like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WneSiO0bv_yIvdOLHSTRRQ.png" /></figure><p>What’s interesting is that selecting single value in chart causes the rest of the widgets to display only that value:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AIoavqehQzrDjDTPy6yUJA.png" /></figure><p>Similarly, selecting single point in our table causes single point display on our chart and gauges:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZRsYCJ99nG5XR4A60bRXg.png" /></figure><p>Once you are happy with what you’ve got on the dashboard you can create report out of Data Studio and share it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*Ymvt69z6-8RUFTK5BaCOGQ.png" /></figure><p>Creating report functionality allows you to add some additional text and other widgets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_iyyAQbz1UY55RQJ0iRnWQ.png" /></figure><p>Click View:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wOWwrOxg9khNN-QDEmkIUA.png" /></figure><p>When you finish polishing up your report you can share it. Again there are a few options to choose from, similarly to the standard Google Docs functionality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*4GEc5PLTf5xabaseeiL35g.png" /></figure><p>In this part of the GCP Goodies tutorial we went from building a simple web application which generates some data, processing that data in BigQuery up to analyzing it and creating report with Data Studio.</p><p>Flow like this could be an end-to-end process for some companies. This shows how powerful Google Computing Platform with its plethora of services could be. Having only a GCP console and less than an hour you can often create a full-blown solution — gathering, processing and visualizing your data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7f504de9e88" width="1" height="1" alt=""><hr><p><a href="https://blog.softwaremill.com/gcp-goodies-part-9-time-series-data-handling-and-visualization-a7f504de9e88">GCP Goodies Part 9 — Time series data handling and visualization</a> was originally published in <a href="https://blog.softwaremill.com/">SoftwareMill Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>