<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michał Bogacz on Medium]]></title>
        <description><![CDATA[Stories by Michał Bogacz on Medium]]></description>
        <link>https://medium.com/@michalbogacz?source=rss-b91ddd9d1583------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-X6XTbx4IyQJ1CWZB7L_Ag.png</url>
            <title>Stories by Michał Bogacz on Medium</title>
            <link>https://medium.com/@michalbogacz?source=rss-b91ddd9d1583------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:16:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@michalbogacz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using Avro to enhance cloud storage. 77% Cost reduction]]></title>
            <link>https://medium.com/virtuslab/using-avro-to-enhance-cloud-storage-77-cost-reduction-9cb022acd555?source=rss-b91ddd9d1583------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cb022acd555</guid>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[avro]]></category>
            <dc:creator><![CDATA[Michał Bogacz]]></dc:creator>
            <pubDate>Mon, 25 Sep 2023 13:53:43 GMT</pubDate>
            <atom:updated>2023-09-26T07:55:17.826Z</atom:updated>
            <content:encoded><![CDATA[<p>By moving from CosmosDB to Azure Storage and using the Avro format, we reduced storage costs by 77%. And reduced storage size by 63%.</p><p>A success story details you can find <a href="https://virtuslab.com/case-study/storage-costreduction-no-vendorlock/">here</a>.</p><p>This is the story of how we achieved this result.</p><figure><img alt="cloud storage" src="https://cdn-images-1.medium.com/max/1018/1*s9aPKKg3M8RJSbaWHNWlkQ.png" /></figure><h3>Use case</h3><p>Before I explain what we did to improve storage efficiency, I want to share some of our system requirements.</p><p>1. Scope</p><p>We build, maintain and improve data-intensive applications for a global freight forwarder.</p><p>While we deal with many dimensions of their business, this blog is about one. Shipment is the primary data entity we work with and it is a huge one consisting of thousands of rows. The Shipment data entity describes packages moving around the world. It’s not data that can be easily broken down into smaller pieces. It has many options, events and related parts that we need to work with as a whole, a bit like a document.</p><p>In normal system design, we would break the data down into smaller pieces, which allows us to easily apply reason and logic, and compress space.</p><p>To split information into smaller sections, developers must first understand it completely. Our client is striving to standardise shipment data, but the project is still ongoing. At present, they cannot offer a complete definition and segmenting the data would lead to confusion.</p><p>Due to time constraints, we opted not to divide the shipment data into smaller parts. Instead, we chose to employ more advanced logic in the code.</p><p>2. Performance</p><p>As mentioned earlier, this is a data-intensive application.</p><p>For the Shipping entity storage:<br>Read/Get: 54.68 million requests per month<br>Update/Create/Replace: 29.86 million requests per month</p><p>This number is increasing every month.</p><p>One of the most essential optimisations for performance is denormalization. This means that if you create a lot of entities and see that it is slow to retrieve them from persistent storage (and you need them to apply business logic), you can merge them.</p><p>Another way of looking at this is that forcing normalisation without understanding all the business logic (even future logic) can be detrimental to performance.</p><p>3. The Cloud</p><p>Our customer has chosen a cloud provider and they understand that the business environment is continually evolving. Relying on technology that is unavailable in other cloud providers, or cannot be configured independently, puts you at the vendor’s mercy. That’s why we were careful at the start of the project to choose technologies that are widely used and can be run on your computer, preventing dependence on a single supplier.</p><h3>Early decisions</h3><p>When we started the project, we had little time to try out different options. With a short time to production and no clarity on normalisation, we decided to play it safe and just store all the shipment data as one.</p><p>After a few discussions and proofs of concept, the decision was made to use CosmosDB with the MongoDB API. This was reasonable for the following reasons:</p><ul><li>We can store quite large entities as json (up to 4MB)</li><li>Vendor lock is limited, we can easily migrate to MongoDB (and use MongoDB for local testing)</li><li>It’s fast</li><li>We can search within all fields, which is likely to be useful in the future</li><li>Geo-replication is built in and would be easy to enable.</li></ul><p>There were other options we considered, including Azure Blob Storage. But we were concerned that unstructured data without the ability to search would greatly increase the likelihood of data corruption.</p><h3>Verified assumptions</h3><p>As time passed, we began to see problems with our storage design. Don’t get me wrong, the system worked, but the number of small problems we had to solve was growing. And the benefits we were hoping for were diminishing.</p><p>What were the issues? Here are the main ones:</p><ol><li>Serialisation — I don’t want to go into details, but serialising such large entities was not a pleasure. The official MongoDB driver doesn’t work well with large structures combined with Scala ADT. We definitely spent too much time trying to make this work.</li><li>Cost — the price of CosmosDB was much higher than we expected.</li><li>Geo-replication — our client really wanted this in the beginning, but after seeing the cost, the decision was made not to enable it.</li><li>Search — Marketing misled us. A search that works with 1GB of data may not work as expected with 500GB. We were constantly hitting timeouts and the cost of a full search query in CosmosDB was huge. After a while, we stopped using it altogether. Parts of the data we needed to search were copied to storage elsewhere (e.g. PostgreSQL), and we accessed data in CosmosDB only by partition ID.</li><li>Data corruption — Schemaless DB has no schema validation, so we can put anything in there. It’s very flexible, but you have to make sure that your changes are compatible with what you already have in storage.</li></ol><p>We were confident that data corruption would not affect us. We do a lot of testing. We check pull requests carefully. But how do we remember every change in schema-less storage? Imagine a logic you created a year ago that was never released to users, hidden under the feature flag, and then removed because of changing requirements. A piece of logic that added a field to 0.5% of entities and the storage was not “cleaned”. Then you must reintroduce it a year later but with a different field type.</p><p>Do you know what happens when your application tries to read it? Yes, it will throw an error and may block your entire processing flow. We knew there had to be a way to protect against that. And the idea came from Kafka, which we also use.</p><p>With Kafka, we were constantly changing fields, but the Confluent Cloud Schema Registry protected us from corrupting data. You have to specify schemas and validate compatibility.</p><p>It got us thinking, what if we could do the same thing with our shipping storage?</p><h3>Apache Avro</h3><p>On the advice of Confluent Cloud, we started using Kafka with Apache Avro. It took us a while to understand why it’s a really good format for data.</p><p>The basics are as follows:</p><ol><li>You can think of it like JSON, but with schema.</li><li>You always need a schema to write or read data.</li><li>Schema can be stored separately. It doesn’t need to be stored with the data. Imagine in JSON you have: <br>{ “thisIsVeryLongKey”:”value”}<br>In Avro, <em>“thisIsVeryLongKey”</em> would be in the schema, and only <em>“value”</em> would be in the data part. More than 2 times the size reduction!</li><li>Schema can evolve according to the rules you specify. A good article on why schema are important can be read here: <a href="http://radar.oreilly.com/2014/11/the-problem-of-managing-schemas.html">http://radar.oreilly.com/2014/11/the-problem-of-managing-schemas.html</a></li></ol><p>And, if you want a deeper understanding of Avro, take a look at this great description: <a href="https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html">https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html</a></p><p>From our point of view, the main advantages are:</p><ul><li>The Avro schema can evolve in a controlled way. You can create a unit test to make sure your change is compatible with ALL versions you have in storage.</li><li>You don’t store or send schema all the time, the only part you send / store is used. This greatly reduces the size.</li></ul><p>Remember that in the cloud you pay for transfer and storage. Storing and transmitting less data can be a significant cost reduction.</p><p>We chose to combine Avro with Azure Storage. This gave us storage that was really well suited to this use case. We never needed a schemaless solution, we needed controlled flexibility without affecting the whole data set.</p><p>With Avro, Azure Storage became storage for structured data with controlled evolution.</p><h3>Pricing and storage calculations</h3><p>Here’s a small calculation of how our storage and pricing changed in a typical month.</p><p>Storage <strong>reduced by 63%:</strong></p><figure><img alt="table with storage data" src="https://cdn-images-1.medium.com/max/606/1*sbbsw03nfeV2LrFm1vMl7w.png" /></figure><p>Number of operations:</p><ul><li>Find: 54.68M</li><li>Update: 29.86M</li></ul><p>Operations cost <strong>reduced by 77%</strong>:</p><figure><img alt="table with operations costs data" src="https://cdn-images-1.medium.com/max/606/1*Bfj40MfdMaQVORHuQg6paQ.png" /></figure><p>This may not be an astonishing dollar reduction, but it is for one month in one environment. And that is just for shipping warehousing. In a global business, it’s not the only huge entity we deal with :)</p><h3>Summary</h3><p>Learning about Apache Avro was really refreshing for me. At first, we didn’t want to use binary formats for fear of compatibility problems. With Avro we have compatibility and storage reduction of more than 50%. It’s safer and cheaper.</p><p>I hope this blog post will inspire others to take an interest in data formats and also create some great optimisations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cb022acd555" width="1" height="1" alt=""><hr><p><a href="https://medium.com/virtuslab/using-avro-to-enhance-cloud-storage-77-cost-reduction-9cb022acd555">Using Avro to enhance cloud storage. 77% Cost reduction</a> was originally published in <a href="https://medium.com/virtuslab">VirtusLab</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Story of importing a large dataset to Akka Cluster]]></title>
            <link>https://medium.com/virtuslab/story-of-importing-a-large-dataset-to-akka-cluster-60ed3bba8184?source=rss-b91ddd9d1583------2</link>
            <guid isPermaLink="false">https://medium.com/p/60ed3bba8184</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[akka-cluster]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[akka]]></category>
            <dc:creator><![CDATA[Michał Bogacz]]></dc:creator>
            <pubDate>Mon, 12 Jul 2021 10:24:25 GMT</pubDate>
            <atom:updated>2021-07-13T07:28:17.107Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y7EA-LiL9Z_DOKlaT3vGfA.jpeg" /></figure><p>This is a story of importing a large dataset to Akka Cluster and the pitfalls we had. It is a technical one, but I won’t explain the features of the Akka Cluster itself. It would be a really sizable blog post if I were to introduce the basics of Akka Cluster and Kafka, so I will assume you have a basic knowledge of them.</p><p>Here, I won’t be describing separate features, but a story, how it worked for our use case: importing a large <em>Internet of Things (IoT)</em> dataset.</p><p><strong>1. Requirements and architecture overview</strong></p><p>Our goal was to create a new platform to integrate with IoT device data. The requirements include:</p><p>1. IoT data is sent continuously, from around 5,000 devices</p><p>2. We need to consume not only the current data, but also data from the entire last year, which combines to be over 50 million measurements</p><p>3. IoT messages need to be available, sorted in the temporal order in which they are received</p><p>There are many reasons why we chose Akka Cluster + PostgreSQL (with <a href="https://doc.akka.io/docs/akka-persistence-jdbc/current/">akka-persistence-jdbc</a>), but from a technical point of view, these include:</p><p>- Easy scalability when more devices are added (by just adding more nodes to the cluster)</p><p>- An easy way of sending data to other systems, with a possibility to replay events (with Akka Projection)</p><p>- Enforcing an <a href="https://doc.akka.io/docs/akka/current/typed/persistence.html">ES/CQRS approach</a> (Commands/Events)</p><p>- Keeping the latest IoT device state in memory (in actors) for fast access</p><p>There was one more requirement: minimize the chance of losing data from devices. This sounds simple, but it can go wrong in many ways, from different reasons like network problems, application lifecycle issues, and business logic updates.</p><p>Therefore, we decided to use a message queue before Akka Cluster as a place where the data can be stored before consumption. One of the most well-known and battle-tested message queues is Kafka, and there’s also good integration of Kafka with Akka using <a href="https://kafka.apache.org/">Alpakka Kafka </a>connector, so we chose it for our solution.</p><p>Within our code, each IoT device in the Akka Cluster is represented as an actor. The stream from Kafka sends messages to all actors, and receives acknowledgement after each message is processed. If the message is not processed by an actor for some reason, Kafka replays it.</p><p>Kafka also gives us one more benefit: since we can buffer current messages, why not put in queue also data we need to replay from last year? Well, that sounded good and a plan emerged. We created two main topics: <em>iot_replay</em> for data from last year, and <em>iot</em> for current data (i.e., the continuous stream from devices). As shown in this image, the import process starts with <em>iot_replay, and</em> then we connect the <em>iot</em> topic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2C_EH1U_oqZ8NBHtZwVJ9Q.png" /></figure><p>It’s also worth mentioning that we didn’t create a Kafka partition for each IoT device. The data rate (in the continuous stream) is not too high, and the number of devices can change over a time. We created 10 partitions, so each partition will send messages from 500 devices.</p><p><strong>2. The problem</strong></p><p>We created a platform, and for a test import of one month of data, it worked. We connected the platform to the <em>iot</em> stream, and it worked too. It seemed that the setup was ready to start real import.</p><p>On the day of the import we created a cluster with a size of 10 nodes, and connected to “replay.” Data from the Kafka <em>iot_replay</em> was consumed at the pace of 2,000 msg/s. After a day, our system had all the data from last year!</p><p>Next, we changed the Kafka cluster consumer to point to the <em>iot</em> topic and redeployed the Akka Cluster nodes. After that we looked at Akka Telemetry, and our consumption was… less than 10msg/s, as shown in this image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*V6-V4BRVRsw_xXcW" /><figcaption>Well, this was unexpected. But such things can happen, right?</figcaption></figure><p>Our first thought was that if the cluster processed a lot of data, it might take some time for the cluster to come back up to full speed. Also, there were more business important matters to solve, so we switched to other tasks and left the cluster to recover itself.</p><p><strong>3. The culprit?</strong></p><p>As we worked on other matters, days passed, and the system still wasn’t consuming messages with its initial speed. We stopped hoping that it would get better, and started investigating.</p><p>After searching through logs and checking metrics we found out that most of our actors didn’t even start. The stream was slowed down because the actors couldn’t consume messages. And the reason was… DB query timeout.</p><p>Let’s look at metrics. Ideally, recovery failure should not happen in your Akka Cluster at all, but as shown in this image, it was indeed happening:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/0*O5Hc44iIQCJI3_oV" /></figure><p>We saw that each message spent a lot of time in stash:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*SERT5FVWnCsAWlYq" /></figure><p>But why are there any messages in stash? Because they wait there before the actor recovers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*NEoMLYnDzRX0ReUS" /></figure><p>Week before import, we had a test with 500,000 messages in a journal, and there was no query timeout — but when we imported 50M messages, it started to occur. Also, this started to happen after replay because we restarted the cluster, and Akka Cluster tried to start actors. Immediately upon seeing the data, we understood our mistake: we forgot to add a snapshotting mechanism. In theory this is optional, but it’s pretty much a requirement in a bigger system. If someone forgets it… well… a failure like ours might happen.</p><p>So two fixes were applied:</p><p>a. We added snapshot generation to the code</p><p>b. We needed to start processing again, so we increased timeouts in persistence configuration:</p><pre><em>akka.persistence.journal-plugin-fallback.circuit-breaker.call-timeout</em> — the max time of query to DB</pre><pre><em>akka.persistence.journal-plugin-fallback.recovery-event-timeout</em> — the max recovery time for getting an event (we changed this just be to sure it isn’t a problem)</pre><p><strong>4. Friend or foe?</strong></p><p>We thought this might be the end of this story: the new configuration was in place, and snapshotting was added. We stopped seeing problems with actors not getting started in metrics, and waited for cluster recovery. But unfortunately, the story doesn’t end here, because no recovery happened.</p><p>Again, we fooled ourselves with thoughts like, ”the cluster needs time”, “snapshots are not ready yet,” “let’s wait some more.” But another day had passed.</p><p>To be fair, there was some improvement: we had no recovery failures. But stream processing was still far away from the initial results. Also actor recovery still was taking a long time (from 5 to 20 seconds) and was happening all the time. We checked the database, and snapshots were there. That was off… if the cluster had already been working the whole day, why were the actors constantly recovering?</p><p>Then it stuck us… <em>passivation</em>! Enabled by default, this mechanism in Akka Cluster is used to improve memory usage. Simply speaking, it stops actors and restarts them only when a new message is received. By default, after 120 seconds with no messages coming in, an actor is removed from memory. This is possible because the whole state of the actor is already in the journal (stored as events), and/or snapshots.</p><p>So our cluster ended up in a loop. Consumption of messages was slow because actors had to be restarted to receive messages, and messages were consumed slowly, so the actors were passivated.</p><p>To make this more clear, let’s go back to the numbers. We had 10 partitions (separate streams) connected to 10 instances and 5,000 actors. So each instance had its own stream and 500 actors. Messages are sent from the stream sequentially, and need to be acknowledged by the actor. Also, messages are ordered by time, so in the worst case scenario — to get a second message for the same actor — 499 other messages must be consumed. If consumption of each message takes 5 seconds (the recovery time of the actor), there’s no way to consume 499 messages before passivation happens.</p><p>Therefore, we decided to disable passivation for the time of the import process, and we were quickly back on track, at a pace of 2,000 msg/s once again!</p><p><strong>5. The real culprit</strong></p><p>The system worked fast again, but we still had a lot of questions:</p><ul><li>Why was passivation enabled by default if the recovery time can be so long?</li><li>Was our data that huge? (At that time we had 69,000,000 events in the journal.)</li><li>Why do we still need 20 seconds to start some of the actors?</li></ul><p>We decided to do another test: how much redeployment will interrupt the working cluster?</p><p>As shown in this image, we did the first deployment before 9:55, then again at 10:10:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*QB9DugBw0wLQd-vB" /></figure><p>The actors started slowly, because of the long recovery time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/0*-xCD_oK7MUEbE42g" /></figure><p>Finally, after 30 minutes the cluster was fully operational again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/0*78g2-JPDICRH0iP2" /></figure><p>From a business perspective we could accept this redeployment time. From a technical perspective, not really. Deployment can be frequent, and in Reactive Systems it needs to be fast. Fortunately, my teammate Jacek didn‘t give up, and dug into the DB queries of the <em>akka-persistence-jdbc</em> code. And guess what? He found that the slow query could be optimized! (You can see the details of his findings in <a href="https://github.com/akka/akka-persistence-jdbc/pull/533">this pull request on Github</a>.) After that fix, the cost of the query dropped drastically, so we enabled passivation again.</p><p><strong>Summary</strong></p><p>If you’re running Akka Cluster, don’t forget about configuring snapshots, and remember about the implications of passivation. Also — as shown in the charts in this article — it’s good to observe metrics on how actors are behaving under a real load.</p><p>We also recommend making sure that you update project dependencies. In many cases they can provide significant improvements for your applications, such as improving the performance issue we discovered. (Thanks to Lightbend, the akka-persistence-jdbc fix was released really fast, and is available in version 5.0.1)</p><p>As a final note, it’s also worth mentioning that finding and fixing such issues wouldn’t be possible if Akka was not open source.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60ed3bba8184" width="1" height="1" alt=""><hr><p><a href="https://medium.com/virtuslab/story-of-importing-a-large-dataset-to-akka-cluster-60ed3bba8184">Story of importing a large dataset to Akka Cluster</a> was originally published in <a href="https://medium.com/virtuslab">VirtusLab</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming large data sets]]></title>
            <link>https://medium.com/@michalbogacz/streaming-large-data-sets-f86a53e43472?source=rss-b91ddd9d1583------2</link>
            <guid isPermaLink="false">https://medium.com/p/f86a53e43472</guid>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[akka]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Michał Bogacz]]></dc:creator>
            <pubDate>Tue, 20 Jun 2017 20:43:24 GMT</pubDate>
            <atom:updated>2017-06-20T20:43:24.542Z</atom:updated>
            <content:encoded><![CDATA[<p>Building REST microservices requires dealing with many restrictions. One of these restrictions is a limited memory. But don’t make the false assumption that you can’t pass data sets bigger than your microservice memory. With Akka HTTP and Akka Streams, it’s not only efficient but also simple. In this blog post, I will show you how to do that.</p><h3>Introduction</h3><p>Let’s start from the beginning. When you build REST microservice, one of the most important endpoints is method GET on a group of resources. Its purpose is to expose resources gathered under given path.</p><p>Example: “GET” method on endpoint “/books” — returns all books.</p><p>At first sight, there is nothing special about it. Just returning a list of resources. But what happens if this list contains many results? So many, that getting them into memory can harm your microservice stability?</p><p>Let’s consider two different solutions.</p><p>Note: Code samples for this blog post can be found on <a href="https://github.com/michalbogacz/http-streaming">github</a>.</p><h3>Solution 1: Client controls the size — Pagination</h3><p>The first approach is simple: user knows best what size the response microservice should return. How can he achieve that? By defining in the request, the number of maximum resources in the response, and the number of resources to be skipped from the beginning of the response. This way, the user is splitting resource list into many requests. And can get them one after another.</p><p>This concept is called <strong>pagination</strong>.</p><p>The benefit of this approach is clear: strict control of returned payload size. That can eliminate possible memory issues for the client and (with proper validation of pagination parameters) can eliminate memory problems in microservice too.</p><p>Now let’s see how this can be implemented in Scala.</p><p>Case class for Pagination parameters:</p><pre><strong>case class </strong>PageParams(pageNumber: Option[Int], <br>                      pageSize: Option[Int]) {<br>  <em>require</em>(pageNumber.forall(size ⇒ size &gt; 0), &quot;page number must be greater then 0&quot;)<br>  <em>require</em>(pageSize.forall(size ⇒ size &gt; 0 &amp;&amp; size &lt; 1000), &quot;page size must be greater then 0 and less then 1000&quot;)<br><br>  <strong>private val </strong><em>DefaultPageNumber </em>= 1<br>  <strong>private val </strong><em>DefaultPageSize </em>= 16<br><br>  <strong>val </strong><em>limit</em>: Int = pageSize.getOrElse(<em>DefaultPageSize</em>)<br>  <strong>val </strong><em>skip</em>: Int = (pageNumber.getOrElse(<em>DefaultPageNumber</em>) - 1) *         <em>limit<br></em>}</pre><p>This is one of possible ways to implement pagination parameters.</p><p>Code presented above defines two optional fields:</p><ul><li>pageSize — the number of elements returned in one response/page</li><li>pageNumber — the number of a page to be returned</li></ul><p>As you can see above, we are not skipping single resources, but groups of resources called “pages”. When pagination parameters are not given, default first page will be used (with 16 elements). Fields “skip” and “limit” will be used later.</p><p>To get all resources from the endpoint, the user specifies one pageSize and invokes GET requests with incrementing pageNumber until an empty response occurs.</p><p>The directive to use this case class is shown below:</p><pre><strong>trait </strong>ServiceDirectives <strong>extends </strong>Directives {<br><br>  <strong>val </strong><em>pageParams</em>: Directive1[PageParams] =       <br>           parameters((&#39;pageNumber.as[Int].?,        <br>                       &#39;pageSize.as[Int].?)).as(PageParams)<br>}</pre><p>And the routing:</p><pre><strong>val </strong>route =<br>  path(&quot;resources&quot;) {<br>    <em>pageParams </em>{ pageParams =&gt;<br>      get {<br>        complete(<em>getData</em>(coll, pageParams)<br>        .map(<em>HttpEntity</em>(ContentTypes.<em>`application/json`</em>, _)))<br>      }<br>    }<br>  }</pre><p>At last the most important function:</p><pre><strong>def </strong>getData(coll: MongoCollection[Document], pageParams:  <br>            PageParams): Future[String] =<br>Source.fromPublisher(coll.find().skip(pageParams.skip)<br>  .limit(pageParams.limit))<br>    .map(_.toJson)<br>    .intersperse(&quot;[&quot;, &quot;,&quot;, &quot;]&quot;)<br>    .runFold(&quot;&quot;)((acc, e) ⇒ acc + e)</pre><p>In the code above I’m using “mongodb-driver-reactivestreams” driver. It exposes Reactive Streams <em>Publisher</em> for many different operations.</p><p>To get <em>Publisher</em> with documents from MongoDB collection, I’m using “coll.find().skip(pageParams.skip).limit(pageParams.limit)” (now you see why I needed “skip” and “limit” fields in “PageParams”).</p><p>Constructor “Source.fromPublisher” changes <em>Publisher</em> to Akka Streams <em>Source</em>. I used “.map(_.toJson)” to simplify marshaling, as it changes documents to <em>Strings</em> (<em>Strings</em> with MongoDB Extended JSON format).</p><p>The last two steps are “intersperse(“[“, “,”, “]”)” and “.runFold(“”)((acc, e) ⇒ acc + e)”. The first step creates correct JSON List response by adding start, end and middle elements. The second one concatenates all results to one <em>String</em>.</p><p><strong>Note</strong>: In my example, I used MongoDB as a database. But this solution is not limited to MongoDB. Actually, it’s very simple to change the storage. You just need the database driver which exposes Reactive Streams Publisher. For example Slick API has a Publisher too: <a href="http://slick.lightbend.com/doc/3.2.0/dbio.html#streaming">http://slick.lightbend.com/doc/3.2.0/dbio.html#streaming</a></p><p>After running the code from example (in GitHub run object “ApiReturningList”) and making a GET request to “localhost:8080/resources” you get response with a body similar to the below example:</p><pre>[<br>  {<br>    &quot;_id&quot;: {<br>      &quot;$oid&quot;: &quot;593fc5f3964652a1ffc9836d&quot;<br>    }<br>  },<br>  {<br>    &quot;_id&quot;: {<br>      &quot;$oid&quot;: &quot;593fc5f4964652a1ffc9836e&quot;<br>    }<br>  }<br>]</pre><p>Of course, the response depends of your MongoDB collection content.</p><p>It’s pretty simple, right? But unfortunately, this solution has few drawbacks. And for one of our users, those were blockers.</p><h4>Different requirements</h4><p>As I mentioned, services that I develop with my team have many REST API users. As you can guess, different users have potentially different use cases. Pagination works well for most of them, but not for all. It turned out that for one user the service responses were slow and sometimes timeouts could occur.</p><p>My team investigated it and found that:</p><ul><li>His use case was cache synchronization and he wanted to iterate over all data</li><li>User had a lot of data, more than 200 000 JSON documents</li><li>We encountered timeouts when the database operation took too long</li></ul><p>But this still does not explain why timeouts occurred, right? Well, not exactly. MongoDB skip and limit are not suitable for pagination over large data sets. In MongoDB documentation, you can read “The cursor.skip() method is often expensive because it requires the server to walk from the beginning of the collection or index to get the offset or skip position before beginning to return results.” [<a href="https://docs.mongodb.com/manual/reference/method/cursor.skip/">link</a>]. This means that responses for the first pages were fast, but for the last pages, were so slow, that timeouts could occur. And that is definitely bad.</p><p>We needed a better solution. The solution that can pass all data through our service, without memory allocation peaks and easy for us and for our current API users.</p><p>And we found it: HTTP Chunked Encoding!</p><h3><strong>Solution 2: Chunking</strong></h3><p>The idea is simple: the data is sent in response as a series of “chunks”. Basically, elements can be sent one by one, without storing the whole response in memory and without closing connection until every chunk is passed. This way you can send unlimited size of data in just one request!</p><p>It’s not new idea, it was introduced in HTTP 1.1 specification. To inform HTTP client about chunked encoding response, the “Transfer-Encoding: chunked” header is added. And best of all — it’s really simple to use with Akka HTTP and Akka Streams!</p><p>Let’s see the examples. First, we create new getData function:</p><pre><strong>def </strong>getData(coll: MongoCollection[Document]): Source[ByteString, NotUsed] =<br>Source.<em>fromPublisher</em>(coll.find())<br>  .map(_.toJson)<br>  .map(<em>ByteString</em>(_))<br>  .intersperse(<em>ByteString</em>(&quot;[&quot;), <em>ByteString</em>(&quot;,&quot;), <em>ByteString</em>(&quot;]&quot;))</pre><p>Differences from first solution:</p><ol><li><em>String</em> is changed to <em>ByteString</em>.</li><li>Returned type is <em>Source[ByteString, NotUsed] — </em>we are not folding the results</li><li>“skip” and “limit” is removed (this is not necessary in this example, but still can be included if you need)</li></ol><p>Now let’s check the route:</p><pre><strong>val </strong>route =<br>  path(&quot;resources&quot;) {<br>    get {<br>      complete(<em>HttpEntity</em>(ContentTypes.<em>`application/json`</em>,<br>                          <em>getData</em>(coll)))<br>    }<br>  }</pre><p>In the above code sample, the response is completed with <em>HttpEntity</em> and <em>ByteString</em> <em>Source</em>. You can try it out by running “AppReturningChunking” from my GitHub examples.</p><p>The response will be the same but contains additional header “Transfer-Encoding: chunked”.</p><p>Using HTTP chunking can give you many benefits:</p><ul><li>better memory management (we are not folding results in memory, just passing them through)</li><li>the pagination is still possible! (we can add “skip” and “limit” if needed)</li><li>we are returning whole MongoDB collection with one database cursor, which is faster and more consistent than pagination</li><li>speed: depending on JSON size and page size, chunking can speed up getting all data even few times!</li></ul><p>One more advice: In your production application don’t forget to adjust your timeouts. Even if chunking is much faster than the pagination from the Solution 1, there still can be a user trying to get gigabytes of data.</p><h3>Summary</h3><p>I didn’t present all implications of using chunking, such as:</p><ul><li>Can chunking gigabytes of data result in user’s service memory overflow? (short answer: not necessarily)</li><li>In REST, the status is sent at the begging of the response, what about errors that occur after response sending started?</li><li>Is it possible to not only get but also send data to the service in a similar way?</li></ul><p>Answers, to these questions, would make this blog post too long. But if you are interested I can write more posts about chunking.</p><p>From my experience, chunking is worth to implement when you are dealing with large data sets. In my team, we added chunking even to endpoints with pagination. This really improved memory consumption in our microservices (and users didn’t even see the difference — all responses are backward compatible).</p><p>My advice for developers: explore different possibilities and features! Technologies like Akka Streams, Akka Http, and Reactive Streams clients to a database can create a powerful and elastic combination. Don’t take for granted that all you did is enough and users have to deal with all your API limitations. Sometimes it doesn’t take much work, to make your users happy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f86a53e43472" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>