<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[SmartCat.io - Medium]]></title>
        <description><![CDATA[Stories about solutions we develop using a combination of Data Science, Data Engineering and DevOps Expertise and the problems we face in our day to day work with clients at SmartCat. - Medium]]></description>
        <link>https://medium.com/smartcat-io?source=rss----948d4f9c991---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>SmartCat.io - Medium</title>
            <link>https://medium.com/smartcat-io?source=rss----948d4f9c991---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:49:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/smartcat-io" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to do Structured Streaming like a boss]]></title>
            <link>https://medium.com/smartcat-io/how-to-do-structured-streaming-like-a-boss-d7ff78c1ccde?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/d7ff78c1ccde</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[structured-streaming]]></category>
            <dc:creator><![CDATA[Milivoje Popovac]]></dc:creator>
            <pubDate>Thu, 19 Sep 2019 16:39:40 GMT</pubDate>
            <atom:updated>2019-09-19T16:39:40.246Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1 — Event time and Watermarking</h4><p>So, you want to do Real-Time data processing in a short period of time — and consequently provide almost instant output. Great! You’re in good company, since it’s one of the hottest topics in BIG DATA today. But! How do you ***do it***?</p><p>This is the part where Spark’s powerful Structured Streaming comes in.</p><p>Structured Streaming provides “fast, scalable, fault-tolerant, end-to-end exactly-once stream processing without the user having to reason about streaming.” It’s built on the top of Spark SQL engine, where the streaming computation is expressed in the same way as a batch. It’s based on 3 design points:</p><ul><li>declarative APIs — instead of you being responsible for tracking the state and dropping it after, or reacting on duplicated events, you define only what to compute — not how to compute it</li><li>Event Time — process data based on timestamps embedded in records at their creation time, as opposed to when the record is received</li><li>Continuous execution for lowest possible latency and micro-batch for high throughput</li></ul><p>This is all nice and pretty, but what if: 1) you have records that come to the system at a random order, 2) if you have to specify how late you expect to see new events in the input stream? These are all important questions because you can’t assume that the size of the state store (where the intermediate information is saved) is infinite.</p><p>Let’s take a break for a minute and look at this example, to learn some new terms. Let’s say you’ve got an input stream of Instagram hashtags, and you want to count them so you find the most frequent ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*2Ubm-mY8Uv6A3p-ZC7lDeg.png" /></figure><h3>Tumbling Window</h3><p>As you can see on the image above, there’s an aggregation being performed over a window of time — in this case 30 minutes. Note:<em> event time</em> with windowing only works with <strong>aggregations</strong>. After the time expires for one window, the calculation is triggered. The engine will operate on the data received since the last trigger. You’ll notice that there aren’t any overlaps between windows. One event can fall into only one window. This type of window is called a Tumbling Window.</p><p>There’s another type, <em>Sliding</em>. They allow each event to fall into multiple windows, like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*UW5dpYoZQ2EbIt5bD4U6ng.png" /></figure><p>Here’s a sliding window with 30 minute increments, but you get the state every 15 minutes (the blue arrows).</p><p>However, these examples have one major weakness. The state used for storing intermediate data will grow INDEFINITELY. Why? Because we never specified how late we expect to see our data. But if there’s a way to do this, we might be able to age-out data and clear state store.</p><p>The solution? Watermarks.</p><p>A <strong>watermark</strong> is “an amount of time following a given event or set of events after which we do not expect to see any more data from that time”. All stateful processing that operates on<em> event time</em> applies to this as well.</p><p>Going back to our hashtag example, we will have a timestamp when it’s produced and also define a 25 minute threshold — the max amount of time hashtags are allowed to be late.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*od0dx7OXDdO3FCtE3nnjhQ.png" /></figure><p>So, the <em>watermark</em> is defined as a maximum <em>event time</em> seen in the previous window minus threshold for late data. The watermark is calculated at the beginning of every trigger (at 01:15, 01:30, 01:45, etc.). For example, #me came at [01:36] to the processing system, but it was created at [01:08]. Given that the current value for our watermark (01:01) was below the tracked event (#me at 01:08), the engine will update the state for the window [00:45–01:15] and maintain it.</p><p>However, when the next trigger had occurred the updated value for the watermark (01:18) was greater than the end timestamp for the previous considered window. This led to the clearing of the intermediate state. Plus, all data after that, which is below the watermark, is skipped.</p><p>All of this brings us to the next question, what will be the output of this stream and how frequent will it be?</p><p>Enter Output modes.</p><p>They help us define what will be printed to the external storage. There’s 3 of them:</p><ul><li>Update — results will be printed after each trigger, where only updated values have been forwarded. E.g at 01:15, 01:30, 01:45, etc.</li><li>Append — only when the intermediate state is dropped, the final result is printed to the sink (e.g at [02:00] results from window [00:45–01:15] will be printed)</li><li>Complete — doesn’t clear the old state since this mode by definition will keep all results from all windows (it should not be used)</li></ul><p>And there you have it! A neat way to use Event Time and Watermark to track states and clear them later. But that’s just one piece of the puzzle.</p><h3>End of part 1</h3><p>What if you want more complex windows? Like if they’re based on counts and not event times? Or if you want variable window sizes? Those are some pretty cool questions and they’re hot and ready for you in PART2: Arbitrary Stateful Processing.</p><p>Coming soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7ff78c1ccde" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/how-to-do-structured-streaming-like-a-boss-d7ff78c1ccde">How to do Structured Streaming like a boss</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DevOps Anonymous — Lessons Learned from Building Cloud Infrastructure from Scratch]]></title>
            <link>https://medium.com/smartcat-io/devops-anonymous-lessons-learned-from-building-cloud-infrastructure-from-scratch-3e768ab6a161?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/3e768ab6a161</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Milan Milosevic]]></dc:creator>
            <pubDate>Fri, 10 Aug 2018 09:02:31 GMT</pubDate>
            <atom:updated>2018-08-10T09:02:30.800Z</atom:updated>
            <content:encoded><![CDATA[<p>I was sitting at yet another DevOps Anonymous (DA) meeting. I was not actually a DevOps (yes, I know that having a DevOps role misses the point of the DevOps approach, but I’m trying to tell a story here, so be quiet). I was a software engineer at that time, mostly working with Java. But I enjoyed listening to DevOps people having problems. They all loved me because they thought I also received alerts in the middle of the night; I didn’t.</p><p>It was Bob’s turn:<br>“I was pushing to automate things from the start, but nobody listened. Everyone was making ad hoc changes, solving the problem at hand. Nobody was documenting anything. Until one day we had to replicate the environment for a different customer. It took us three months to untie everything and set it up again.” — Bob’s eyes glared with tears.</p><p>I was enjoying these meetings. That was my vacation. Until recently. Two years ago I started being more involved in how/where/when my code was being deployed. And a year ago I began deploying other people’s code. So, attending DA meetings stopped being my guilty pleasure; I became a full member with my own struggles. And this is my story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/0*v3vx-evmPJkGkHIi" /><figcaption>SysAdmin making changes in cloud infrastructure (2018)</figcaption></figure><h3>My Turn</h3><p>“I’m Milan, and I’d like to share an experience from our newest endeavor. The idea for the project was to automate all the infrastructure creation <strong>right from the beginning</strong>. The AWS infrastructure was deceitfully simple: one master region with MongoDB replica set, CodePipeline for CI, CloudWatch for monitoring and alerting (and several smaller supported components and services); and three satellite regions with ECS services running NodeJS app in Docker containers. The app services in all regions communicate with each other using RabbitMQ (running in the separate containers). Quite simple, right?” — it was my first time to share something in the DA meetings after three years of attending.<br>“We estimated the work to be done in six or seven weeks… How wrong were we. But here are the lessons we learned along the way.”</p><h4>1) Even simple and well-documented services in AWS take long to automate… Or do they?</h4><p>Yes, they do.</p><p>But let me first introduce the tools we were using.</p><p><a href="https://aws.amazon.com/cloudformation/">CloudFormation</a> is a native AWS language for describing resources that need to be created. Since AWS develops it, the assumption is that it supports the majority of services and configuration options that we need. However, since we’re also using Ansible for the provisioning of the database component (we already had the roles we needed), we decided to use CloudFormation inside of Ansible wrapper; which turned out to be the right combination, especially when we had troubles making a cross-region VPC peering connection through CloudFormation for some reason (probably because some parameter for CloudFormation template was not documented well). Anyway, it worked out with Ansible’s <a href="https://docs.ansible.com/ansible/latest/modules/ec2_vpc_peer_module.html">ec2_vpc_peer</a> module.</p><p>So, CloudFormation and Ansible (with <a href="https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html">ec2 dynamic inventory</a>). Why is it so time-consuming to automate everything from the start?</p><ol><li>Creating VPC and EC2 resources in AWS takes a lot of time. It takes 5–10 minutes for AWS to create a VPC(s), bastion server and all the EC2 instances that we need. Almost the same time to delete it. Therefore, making and testing changes while developing often involves deleting the stack and recreating a new one which results in 10–20 minutes of waiting for the resources to be created. And if I make an error, like passing the wrong reference, the creation fails (sometimes after 10 minutes) and rollback starts, which, again, takes some time. All in all, experimenting can be very time-consuming.</li><li>When you’re creating AWS resources through web console UI, AWS often creates depending resources in the back. For example, if you want to create an ECS service, load balancers will be created and configured for you automatically by AWS. On the other hand, when you’re creating ECS through CloudFormation, you have to create and correctly configure all the backing resources (such as load balancers) yourself. It’s not rocket science, but it takes time to read the docs about a resource, try it out, decide on a proper configuration and integrate with the rest of the stack; then test and repeat.</li></ol><h4>2) Make it work — then automate it</h4><p>You are familiar with this picture of how Spotify builds products:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*SeZ_DbKAuUOidE5P" /></figure><p>In hindsight, the same principle applies to building and automating cloud infrastructure (and probably for everything business related). We should focus on providing value from the start. First, let’s build the infrastructure by hand, by clicking through AWS web console UI.<br>That way, we will provide real, customer-facing value from the start. Then, we automate the creation of one component at a time. There is an overhead when you have to maintain the infrastructure that is built one half by scripts and one half by hand, of course. You have to manually edit the scripts and insert the IDs of the resources created from UI. It’s not as simple as it sounds (if it sounds simple at all). But I think that getting the product/feature to the market as early as possible is worth the struggle.</p><h4>3) Optimize for debugging</h4><p>Since experimenting with cloud resources takes a lot of time (because the creation/changes/rollback takes time), Ansible/CloudFormation scripts should be optimized for debugging and experimenting. This means that, if you’re experimenting with one resource and there’s no need for the rest of the stack to be running, comment out the rest of the stacks. An astonishing observation, I know!But too many times I was thinking like this: ‘Okay, let me try out a different parameter for MongoDB instances. I’m sure that all will be well and I’ll probably leave it as it is.’</p><p>Then I spin up all the resources to avoid losing time on commenting out the unnecessary ones. And then, 4 hours later I’m still making changes and experimenting and always waiting for the whole stack to be created/updated/deleted.</p><p>The same goes for Ansible. If I’m experimenting with a module that I haven’t used before, I should comment out the rest of the playbook. Even though usually there won’t be any effect on the tasks that haven’t been changed, Ansible will have to run them all, and I have to wait for the part of the code that I’m testing to be executed. I always think: “It will probably succeed this time, I don’t want to lose more time commenting out the rest of the playbook.” But it never succeeds on the first try (unless I optimize for debugging and comment out some code, of course, then it works flawlessly, and I have to uncomment the code again).</p><h4>4) `grep` is your BFF</h4><p>Tooling in DevOps ecosystem is not as advanced as heavyweight IDEs in Java world (like Eclipse or IntelliJ).</p><p>However, doing DevOps work means that we’re very close to the Unix command line. Therefore, utilizing the command line programs should be part of our workflow. And that is awesome!<br>`grep` is your best friend forever. Let that sink in. It is your best friend because it is the fastest, most flexible way to search for a place where, for example, an Ansible variable is used and/or defined. Once you learn to grep — it is forever. `grep` was created almost half a century ago, and it’s here to stay. Can you say that for Eclipse, IntelliJ, Atom, Sublime, or other IDEs and text editors? Not so sure. Investing in learning several `grep` parameters and use cases definitely pays off.</p><p>Want to check where the “satelite_regions” variable is used and defined?<br>grep -r satellite_regionsresults with:</p><pre>./infra.yml: regions: “{{ [ master_region ] + satellite_regions }}”<br>./infra.yml: with_items: “{{ satellite_regions + [master_region] }}”<br>./infra.yml:</pre><pre>with_items: “{{ satellite_regions }}”./inventory/aws-dev/group_vars/all/vars/all.yml:satellite_regions: [‘sa-east-1’]./inventory/aws-production/group_vars/all/vars/all.yml:satellite_regions: [‘ap-southeast-1’, ‘ap-southeast-2’, ‘sa-east-1’]./inventory/aws-sandbox/group_vars/all/vars/all.yml:satellite_regions: [‘sa-east-1’]</pre><pre>./inventory/aws-staging/group_vars/all/vars/all.yml:satellite_regions: [‘sa-east-1’]</pre><p>By the way, here’s a really lovely illustration of `grep` use cases (all <a href="https://twitter.com/b0rk/status/991880504805871616">credits</a> to <a href="https://twitter.com/b0rk">Julia Evans</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*AItvpIpqn5CDVgAy" /></figure><p>Since we’re already at the command line almost all the time, you can start using Vim and embrace the best text editor ever… Actually, let’s move on.</p><h4>5) Hardcoding is fine and DRY means DO REPEAT YOURSELF</h4><p>Imagine this <a href="https://stackoverflow.com/q/21978641/600667">very common scenario</a>:<br>We need to create several (or dozens of) EC2 instances which are almost the same. They are probably different in just several parameters. In Java (or any other high-level programming language) that problem would probably be solved by creating a method with input parameters (those config options that have to be different). So, I was always tending to make this kind of ‘generic’ approach with CloudFormation/Ansible. Unfortunately, this does not apply here because CloudFormation is not a Turing complete language (it does not have a notion of loops).</p><p>Therefore, we have to repeat ourselves. This goes for hardcoding as well. We wanted too much to make everything as generic as possible, but that led to hate which led to suffering and so on. In the end, maybe my CF/Ansible scripts do not have to create SSL certificate dynamically. Why would I want to automate the creation of it? Do I need thousands of certificates? If I need only several and I can reuse them, I’ll create them manually and hardcode the ARN reference into the scripts/inventory and save a lot of time.</p><h3>Conclusion — is automating an infrastructure worth it?</h3><p><strong>Short answer</strong>: yes, but we have to be smart(er) about it.</p><p><strong>A bit longer answer</strong><br>When we automate infrastructure we get all the advantages (and disadvantages) of maintaining and running the code:</p><ul><li>ease of execution- errors (bugs) are systematically fixed- far less documentation- people are interchangeable on the project</li><li>easier to do experiments (this might be the strongest point. My next blog post will be about this.)</li></ul><p>But we have to make sure to provide business value as fast as possible. Automating everything from the start can slow us down significantly (maybe even stop us). Therefore, having a fully automated infrastructure is the ultimate goal, not something we start from. We have to be aware that a compromise between automation and providing value is necessary, as depicted on this diagram (taught at business schools):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*prN0TLMLG8XHVGn_" /></figure><p>The Conjoined Triangles of Success (Barker, Jack, 2015)</p><p>To me, it is also about the feeling that I’m creating something great with a single command. Think about it. Just a few years ago it was unthinkable to run a single command and spin up a complex, distributed infrastructure with database and application services spanning 3 or more continents with everything being highly available and fault tolerant. If there’s an outage in one region, the requests will automatically be rerouted to another AWS region.</p><p>All of that is just beautiful… And we can use this technological marvel to entertain our customers with funny cat videos.</p><p><em>Originally published at </em><a href="https://www.smartcat.io/blog/2018/devops-anonymous-lessons-learned-from-building-cloud-infrastructure-from-scratch/"><em>https://www.smartcat.io/blog/2018/devops-anonymous-lessons-learned-from-building-cloud-infrastructure-from-scratch/</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e768ab6a161" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/devops-anonymous-lessons-learned-from-building-cloud-infrastructure-from-scratch-3e768ab6a161">DevOps Anonymous — Lessons Learned from Building Cloud Infrastructure from Scratch</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can a bot learn a second language?]]></title>
            <link>https://medium.com/smartcat-io/can-a-bot-learn-a-second-language-21dba6d4b2f6?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/21dba6d4b2f6</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nina Marjanović]]></dc:creator>
            <pubDate>Thu, 21 Jun 2018 11:50:12 GMT</pubDate>
            <atom:updated>2018-06-21T11:50:11.980Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8XUTgcNrub4hcrrlV63LDQ.png" /></figure><p>Creating a chatbot that can understand and use a language other than English can be an ambitious task. Chatbots are still in their early days and even though there are many NLP libraries available nowadays, most of them support only the English language. From stopwords and POS taggers to pretrained word2vec models, it can be time-consuming to work on an NLP problem in a different language.</p><p>We at <a href="https://www.smartcat.io/">SmartCat</a> tried a bit of a different approach in creating a bot that uses the Serbian language. Using a large dataset of unlabeled sentences written in Serbian and performing ML methods, we were able to create a chatbot that resulted in a very decent performance. It was able to return an expected response 9/10 times. Interested in how we did it? Keep on reading.</p><h3>Processing</h3><p>As with any other NLP problem, we started by processing our sentences. The processing steps included converting text to lowercase, removing numbers that were irrelevant, removing punctuation and stripping extra white spaces. All these methods can be performed on a text written in any language. If we had been working with sentences written in English we would have continued by performing stemming, lemmatization, removing sparse terms and stopwords, but this got us thinking. Knowing that stopwords occur very often in sentences (an example of such words in English would be words such as: a, an, the, such, as…) we decided to perform tf-idf statistic on our dataset. That way we created a large vocabulary of words in Serbian that had tf-idf index assigned to them. Words with lower tf-idf value were considered irrelevant — either stopwords or sparse terms. Words that had a higher tf-idf value were considered important. Stemming and lemmatization were not performed for two reasons — 1. there are many suffixes, many rules and even more exceptions in the Serbian language on when a suffix can be applied to a word stem; 2. there are a few dialects and many grammar errors in our dataset which could lead to wrong conclusions. We will iron out this issue in further steps.</p><h3>Exploration</h3><p>Alright, now we have some processed sentences to work with. Usually we would continue by performing POS tagging so we can learn something about semantics. Unfortunately, there are no POS taggers available for Serbian language and creating one would take too much of our time. That is why we opted for creating a word2vec model. Even though our dataset wasn’t perfect, we were positive that it would give us some insight on word similarity and relations. If you are interested in learning more about how word2vec can do this, you can check out our <a href="https://www.smartcat.io/blog/2017/word2vec-the-world-of-word-vectors/">blog post</a>. We trained our model using around 400.000 sentences, created word embeddings and then continued by clustering retrieved embeddings using K-Means. It worked like a charm — it was clear that there was a cluster that contained greeting phrases, the one that contained personal information such as names or ID numbers, the one that contained gratitude phrases, complaints, problem reports etc. After tuning our algorithm and creating subclusters where needed, we were able to label our large dataset in no time. Now we were aware of the semantics which was what we needed to generate adequate responses.</p><h3>Creating a ChatBot and results</h3><p>We used the clustering algorithm only as a classification tool, as we believed that the classification model trained on our labeled dataset would perform better over time. In our tests, random forest resulted in the best accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/0*f3gEqt4FXCPHOHoS" /></figure><p>Translation:</p><p>User: hi</p><p>Bot: Hello! Thank you for contacting us. How can I help you?</p><p>User: why do you have frequent issues with your website</p><p>Bot: We will check immediately with a competent service so we can give you the right information.</p><p>User: ok, thank you</p><p>Bot: Thank you for contacting us. If you have any further questions, we will be pleased to answer.</p><p>Whenever a new message is received, the chatbot would process it, classify it, and return the response that was assigned to the predicted class if the prediction accuracy was higher than the threshold we set. We tested our bot on 100 previously conducted conversations. These conversations contained 210 messages from users in total. Out of 210, 23 messages could not be handled by the bot and 11 messages were not handled correctly. That means that in 84% of cases the bot was able to respond and in 95% of cases the bot was able to handle the conversations correctly.</p><p>We were able to achieve even better results using n-grams. We separated every message received into n-grams where n can take values in range 1..no_of_words. That way we were able to detect multiple classes for each message and give a better response. For example, a sentence could contain both a greeting phrase and a question. This approach does require more time to process each message, but it could be applicable to use cases where immediate response is not of great importance.</p><p>We hope we gave you an idea on how to handle your non-English speaking bot. Have you tried any other approach? Let us know in the comments, we would be happy to discuss.</p><p><em>Originally published at </em><a href="https://www.smartcat.io/blog/2018/can-a-bot-learn-a-second-language/"><em>https://www.smartcat.io/blog/2018/can-a-bot-learn-a-second-language/</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21dba6d4b2f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/can-a-bot-learn-a-second-language-21dba6d4b2f6">Can a bot learn a second language?</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bitcoin Trade Signals]]></title>
            <link>https://medium.com/smartcat-io/bitcoin-trade-signals-1d9c25786bc4?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/1d9c25786bc4</guid>
            <category><![CDATA[superset]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[trading]]></category>
            <dc:creator><![CDATA[Ljubica Vujovic]]></dc:creator>
            <pubDate>Mon, 16 Apr 2018 17:48:09 GMT</pubDate>
            <atom:updated>2018-04-16T17:48:09.540Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*Y_OcToDfZeiQMcL2MjRfBQ.jpeg" /></figure><h3>What will Bitcoin do</h3><p>When asked what the stock market will do, J.P Morgan replied “It will fluctuate.”. If we could hypothetically ask Mr Morgan another question, very popular these days, I bet his answer would be “It will fluctuate a loot”.</p><p>Of course, the question is about the most hyped thing these days after Deep Learning:</p><blockquote>What will Bitcoin do?</blockquote><p><a href="https://www.smartcat.io/">SmartCat</a> team answers that question with mathematical precision, so by the second paragraph you will start trading and by the end of the post, you will be rich. :)</p><p>And by the beginning of this sentence, you’ve probably realized I was joking.</p><p>After reading a lot about cryptos, listening to many failure scenarios and blooming future prospects, I was extremely impressed how little we know about main influence factors and how poorly we can quantify risks within investing in Bitcoin. Although the idea of decentralized currency is something I truly believe can bring benefits to the economy and reallocate global resources, before going “all in”, we definitely need to understand better what makes Bitcoin hit the ceiling.</p><p>This blog post is about our journey of transforming public emotions, big news, blockchain data into signals which can provide us with a better understanding as well as instructions for investing. In the first part, we will describe how we collect and measure public emotions and fundamental data related to blockchain. Then we will continue with prediction models and results of our exploratory analysis. In the end, we will describe how we want to use predictions of our models to empower trader strategies and conclude with further ideas for development.</p><h3>FUD and FBD ( Fear, Uncertainty and Doubt and Fundamental Bitcoin Data)</h3><p>Let’s take a good look at Twitter and assume a possible reaction to these tweets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*B3bs5Z9eDlpwtKS193C57w.png" /></figure><p><em>Oh, poor Woz, he was firstly enticed by Jobs, and now deceived by this bad Bitcoin gang. Too fraudulent for me, I’m not gonna have anything with these wanna be currencies!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*3csMWapa8cpm3zYB6Y941A.png" /></figure><p><em>Good job Germany! I knew someone will start noticing the potential and prosperity Cryptos can bring! I should invest anytime soon!</em></p><p>Although exaggerated, these two examples represent some part of public sentiment which can be an incentive to buy or sell. These actions make the market go up or down, so tracking the sentiment could be a useful price predictor. Our first approach relies exactly on this fact. We tracked many people who proved to be big shots in the crypto world a.k.a influencers on Twitter, collected their tweets and estimated the sentiment. We decided to use Twitter as a social platform for this usage because its content is in micro blog format, which is convenient for processing, but any other network or source of information could be added.</p><p>One of the possible ways to estimate the sentiment of the sentence is to train a model on labeled data. In theory, we could read every tweet and label it with a number from -1 to 1 based on our personal impression. This is followed by training, tuning, testing and voila, the estimator is ready. This process is extremely time and energy consuming so we wanted to do better.</p><p>The next idea was to use a pretrained model, but the main problem with pretrained models is that their training set is a domain-specific corpus which means that we would be using a model trained on data with a different distribution than our data.</p><p>In the end, the solution was to use a rule-based sentiment analysis tool. Most of the features were inspired by the work from trading, but some of the features were the results of our own intuition.</p><p>For example, one of the very interesting insights is brought with the feature constructed as a ratio of mean and standard deviation of positive score of tweets based on previous 30 days, which is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XgEfC9Mq_9ksF0QuZvX9Ag.png" /></figure><p>This feature had a high correlation with the Bitcoin price (0.86 for a year) and succeeded to precede to some big price leaps. Although we are aware that correlation is not causation, we can provide an interpretation of this result. If standard deviation is high, which means that the sentiment is highly volatile, even if the overall score is positive, we are not convinced that the Bitcoin market is doing well. However, if everything seems stable and if most of the people are talking good, this will be captured by a high value of this feature and this kind of situation has potential for a bullish scenario. Finally, this may be our way of quantifying FUD :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*IXzP-JOQVJWioLFexLa09Q.jpeg" /></figure><p>On the other hand, we also believe that, besides news and speculations, some very solid things could bring valuable information to our models. Blockchain data is a set of time series which represents diverse things related to Blockchain and Bitcoin. We used these time series also as features of our models.</p><h3>Prediction models</h3><p>With all prepared data, the next step is … machine learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*2TsluoCuU_zcHyPxEJHTbg.jpeg" /></figure><p>Firstly, we performed training on a daily level. This was a straightforward decision because one of the data sources was on that granularity. In the previous step we created as many features as we could, so it is very probable that some of them are not useful or are highly correlated with each other. In order to have a robust model with good generalization power, we put some effort in reducing the number of features. Feature selection methods were used to choose a set of features which bring the best metrics. In the first fifteen most important features, according to these methods, ten of them were features created with the sentiment from Twitter.</p><p>Then, we plunged these data into several models starting from SVM to ensemble models. In the end, we presented our results with interactive Superset dashboards. You can read more about Superset setup read <a href="https://www.smartcat.io/blog/2018/test-driving-apache-superset/">here</a>.</p><p>Comparing our classification accuracy with several baselines, including the one which predicts the same as previous day and the one which randomly predicts class, we beat them with overall accuracy of 72%. <a href="http://btc.labs.smartcat.io/">Dashboard</a> with results is live with user guest/guest so you can check it out any time you want and the preview of the Bitcoin Trade Signals dashboard is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2jxW7bACW8zTMbRx6KI7hw.png" /></figure><h3>Trader</h3><p>Now comes the catchy thing or the big question.</p><blockquote>How can we use outputs from our models to decide whether to buy or sell?</blockquote><p>One of the most simple answers would be to use signals from our dashboard in order to make a better informed decision. But we look for more than just reading numbers and following the gut feeling. We want an automated trader powered by our predictions. The idea was to improve some of the existing trading strategies, for example <a href="https://www.investopedia.com/university/movingaverage/movingaverages4.asp">SMACrossover</a>. SMA Crossover stands for Simple Moving Average Crossover. The main idea behind this strategy is to follow two time series: Short Moving Average series and Long Moving Average series of price. Let’s assume short MA is MA26 which is just a mean of the last twenty six samples of price and that long MA is MA100. As shown on the picture bellow, if short MA is crossing below the long MA, this is a signal for selling and vice versa, if short MA is crossing above long MA this is a signal for buying.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*4zxpAb3HEvhZur5EtgNU_Q.png" /></figure><p>We chose this strategy because it’s simple and interpretable, but our approach could be implemented with far more complex strategies. One of the possible improvements of SMA Crossover with outputs from our classification models is compared to the baseline strategy and in average on daily level, for several test periods in history, it beats the baseline with a 16.9% bigger return.</p><h3>Final remarks</h3><p>In a certain manner, we confirmed and quantified the influence of people’s sentiment to price. Although convenient, Twitter is not the only and best source for the information we want to analyze, so adding more sources promises improvement in metrics. Furthermore, developing more sophisticated sentiment estimation, trying new algorithms and adding more complex trading strategies are our future steps.</p><p>But the greatest challenge also lies in understanding what would be the most useful feature for people who want to rely on our signals. Is it just pure sentiment, predictions from our models or the net return of strategy? We will put some effort in figuring this out as well. If you have any ideas or want to express your personal opinion, feel free to share them with us. We are looking forward to that.</p><p><em>Originally published at </em><a href="https://www.smartcat.io/blog/2018/test-driving-apache-superset/"><em>www.smartcat.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d9c25786bc4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/bitcoin-trade-signals-1d9c25786bc4">Bitcoin Trade Signals</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Test-driving Apache Superset]]></title>
            <link>https://medium.com/smartcat-io/https-medium-com-dooxan-test-driving-apache-superset-58eb1b6b750?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/58eb1b6b750</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[superset]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <dc:creator><![CDATA[Reality Under Load | Dusan Randelovic]]></dc:creator>
            <pubDate>Mon, 05 Mar 2018 09:44:37 GMT</pubDate>
            <atom:updated>2018-03-27T10:57:21.665Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Curious Case of Algo-Trading Dashboard</h4><p>For one of our recent internal projects, we needed a quick and easy way to showcase some first insights, do some plotting and interactive storytelling with the data. We also wanted to build a live, working dashboard in front of a (future) product — to serve as a demo. The project in question was <a href="https://www.smartcat.io/case-studies/2018/bitcoin-trade-signals/">Bitcoin Trade signals</a> and you should soon be able to read more about it in a separate blog post. For the purpose of this test-drive I took just one component of the whole project — Trading dashboard, and on it I’ll point to some cool features of <a href="https://superset.incubator.apache.org/">Apache Superset</a> and also to some peculiarities we encountered.</p><p>Superset is Airbnb’s data exploration platform: modern, intuitive, open sourced, built on top of Flask and SQLAlchemy and therefore covering many possible data sources. One can also read about it as an enterprise-ready and scalable BI solution, but testing that particular claim is for another story and different use-case. Our algo-trading project is small-scale, going through several steps:</p><ul><li>collecting data hourly or daily from various sources into a simple SQLite database,</li><li>doing some predictive analytics (daily and/or weekly), and finally</li><li>using predictions as signals for algorithmic trading of, what else these days than — Bitcoins.</li></ul><h3>Setup</h3><p>To schedule data collection and training of models, we use another Airbnb’s tool — Apache Airflow (one could argue that we are somewhat biased toward Airbnb-powered open-source tools). In the final setup, architecture consists of 3 components: dockerized models/trading scripts, dockerized Airflow version 1.9.0, with custom config and <a href="https://airflow.apache.org/concepts.html?highlight=dag#dags">DAG</a>s and dockerized Superset version 0.22.1. Existing docker images for Superset lack full automation for admin user creation and database initialization, so we changed Dockerfile and entrypoint scripts a bit. This setup is flexible enough so feel free to use it in other cases that you find fit. The scaffold of this setup that we abstracted away from the project could be found on <a href="https://github.com/smartcat-labs/superset-trading-dashboard">github</a>.</p><p>If you follow the instructions from the repo, in just a few steps you should have containers running with Airflow and Superset GUI on ports specified in docker-compose.yml. The application flow goes like in the picture below:</p><ul><li>Airflow DAG from its own container starts some python scripts in the trader container,</li><li>SQLite DB is mounted to the trader container and populated when new data comes, existing strategies are backtested or new trades are done,</li><li>DB is also mounted to and accessed from inside the Superset container and your Dashboard is updated, live.</li></ul><p>Simple and pretty neat — I could imagine making many similar demos, in different domains, running on this same setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*hB6JqXnUSBH0Azr7." /></figure><h3>From DBs to Tables to Slices</h3><p>When you have your database in place and imported in Superset, you can start playing around. Superset comes with powerful SQL Lab to make all sorts of queries on your newly imported DB, but in order to actually do that you’ll need to edit DB settings and tick the <strong>Expose in SQL Lab</strong> option and/or <strong>Allow DML</strong> option. Next, you’ll expect to be ready to make parts of dashboards called slices, which could be charts, tables, static parts, iframes, etc. However, in order to make slices for dashboard(s) you’ll need to add specific tables on which you could work. This was the first unintuitive situation with Superset — the fact that slices worked on single tables only.</p><p>Adding the tables is easy though, and modifying settings for their columns is recommended! This way you obtain fine control of which columns you want to be groupable, filterable, which aggregations are allowed on which columns, etc. Slices are just a step away, but first let’s dig a bit deeper into this unintuitive part that I mentioned.</p><p>What if we want to visualize on some JOIN between tables? Is that supported? Well, this has actually been asked a couple of times here-and-there and is currently <a href="https://superset.incubator.apache.org/faq.html#can-i-query-join-multiple-tables-at-one-time">the first item in the FAQ</a>. The provided answer and possible solutions are not that informative, so here are 2 straightforward solutions:</p><ol><li><strong>Use table edit mode to add additional metrics.</strong><br>In table edit mode you could set additional composite metrics from your existing columns but also construct them by JOIN-ing other tables’ columns with a custom SQL query.</li><li><strong>Create a VIEW in SQL Lab and visualize on it.</strong><br>If you followed closely, you’ll remember that in the lab you could do all sorts of queries (even DML ones) so some JOINs are supported there too. When you click on visualize on the JOIN result, View gets automatically saved in the list of tables as if it were a regular table in DB. One possible downside is that exported/pickled Dashboards with this type of VIEWs could become non-portable — I have had this issue but this is not a confirmed bug.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*tKlNrXKdY79D4ZmK." /></figure><h3>(Some specific) Slices</h3><p>There is a fair amount of possibilities between types of charts, tables, static and interactive blocks and even maps with custom data that could be made with Superset. I won’t go through all of them here — just the ones that turned out challenging for Trader dashboard:</p><p><strong>Charting challenge 1: Visualize Backtesting results of (many) algo-trading strategies<br></strong>In algorithmic trading, backtesting is a technique of replaying historical situations in order to test an algorithm. We did this with pyalgotrade in an event-driven fashion for many parameters’ combinations and saved the results in a database (every backtesting run creates one table). Soon after the number of backtesting runs passed over a dozen (and later even a hundred), with 150–1000 or more records each, I knew my data model was not optimal for comparing them on charts. So I joined the tables! No, not in SQL Lab, but by changing the data model. This was necessary since simply joining the tables once wouldn’t resolve the problem when we do more and more backtesting runs. The resulting table was of course larger, but easy to query and do GROUP BY on specific backtesting run. I also used Superset’s Filter Box widget to provide some nice controls and the resulting UX was quite pleasing. Upon changing the selection every slice was updated with corresponding FILTER-ed data.<br><strong>Result</strong>: After the data model change, making the imagined charts and desired interactivity in Superset was very easy and intuitive!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*BdVh6uj4-7Mb3lpm." /></figure><p><strong>Charting challenge 2: Summarize improvement in return by one trading strategy over some other</strong>Apart from parameter optimization for a specific trading strategy, I also wanted to compare backtesting results between different strategies over the same periods of time and plot the trendline of difference in return. In order to do this, I needed summary statistics over some FILTER table views matched between strategies. Some very complicated queries emerged and I finally decided to make a new table (not a SQL Lab view) with these statistics separately calculated.<br><strong>Result</strong>: Not much of a fault of Superset, but this simple chart turned out to be very complicated. A newly derived table solved the problem so in this battle it was Pandas vs. SQL (1:0) and Superset was able to produce one very lightweight trendline with a simple query behind it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*IFF2G31FC-5wubnEil_LXQ.png" /></figure><p><strong>Charting challenge 3: Candlestick chart</strong><br>This one I knew I couldn’t make properly with current Superset’s tools — since candlestick charts represent time series of financial asset prices that are heavily annotated. Candles are bars made from Open/Close/High/Low market prices for each day (Figure below). Superset 0.22.1, a stable version that we tested initially, didn’t have any annotation capabilities. But the next minor version, 0.23.0rc4, had promising annotation types and we wanted to test that on this case even before a stable release. The annotation types available are: Time Series, Event, Region and Formula, but unfortunately they only annotate along the series line or whole rectangular areas on the charts — no bars on specific events, single points or similar. Until more complex annotations become available in Superset, one option to make very customized graphs could be making a separate page, alongside Superset, as <a href="https://superset.incubator.apache.org/installation.html#blueprints">Flask Blueprint</a> and then, for plotting a candlestick one can use matplotlib’s <a href="https://matplotlib.org/api/finance_api.html#matplotlib.finance.candlestick_ohlc">candlestick functions</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*d3rJr0ISxH3LZxpTNS2OGQ.png" /></figure><h3>Final conclusions</h3><p>Superset is a very nice and promising BI tool. It is still incubating and will definitely get much better in the future. Since the focus of this testing was on tooling, I would conclude that making charts is really intuitive, with a major lacking feature right now being annotations and custom, complex charts. Other issues I stumbled upon turned out to be solvable by adjusting the data model or thinking ahead when designing it.</p><p>As for next steps — we will definitely continue working in Superset and, as mentioned, we are planning to run it in cluster mode, querying some giant databases with millions of rows, load balancers and all. This test-drive went quite well, let’s see how it behaves out on the open road.</p><p><em>Originally published at </em><a href="https://www.smartcat.io/blog/2018/test-driving-apache-superset/"><em>www.smartcat.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58eb1b6b750" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/https-medium-com-dooxan-test-driving-apache-superset-58eb1b6b750">Test-driving Apache Superset</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GDPR and Cassandra]]></title>
            <link>https://medium.com/smartcat-io/gdpr-and-cassandra-c893e863b241?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/c893e863b241</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[eu-gdpr]]></category>
            <category><![CDATA[cassandra]]></category>
            <dc:creator><![CDATA[Milan Milosevic]]></dc:creator>
            <pubDate>Thu, 22 Feb 2018 11:29:19 GMT</pubDate>
            <atom:updated>2018-02-22T11:29:19.316Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post was originally published on </em><a href="https://www.smartcat.io/blog/2018/gdpr-and-cassandra/"><em>SmartCat’s blog</em></a><em>.</em></p><p>delete /dɪˈliːt/ — verb: remove or obliterate (written or printed matter), especially by drawing a line through it.</p><h3>Schrödinger’s Data</h3><p>As we all know, GDPR will be <a href="https://www.eugdpr.org/gdpr-faqs.html">in force May 2018</a>. After that, users of software products and services will have the right to be forgotten (cool, right? Finally I can rest assured that my browsing history will not be read aloud at my funeral). In other words, if a user from the EU asks a service provider to delete user’s data, the provider will have to delete all user’s data or face severe consequences.</p><p>But, it is unclear what it means to delete a user’s data. I guess the only way to find out is when the audit occurs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*UgssftJJmE2zL5ClenYe2w.png" /><figcaption>A user’s data is simultaneously both deleted and not deleted until observed at the time of audit.</figcaption></figure><p>This post is the introduction to a series of blog posts about GDPR and Cassandra databases.</p><h3>Cassandra and Data Deletion</h3><p>As Cassandra consultants, our primary concern is: what does it mean to delete the data from Cassandra points of view? And what we can do to be as sure as possible that a user’s data will stay deleted. As we know, when Cassandra deletes the data, it just <a href="https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlAboutDeletes.html">marks it as deleted</a>. The actual “deletion” occurs during the <a href="https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlHowDataMaintain.html#dmlHowDataMaintain__dml-compaction)">compaction process</a>.</p><p>When Cassandra marks data as deleted:</p><ul><li>It can’t be fetched anymore using Cassandra’s query language (cql).</li><li>The data still exists in Cassandra’s files on the disk (SSTables) but is flagged as deleted.</li><li>The data is removed (for real) from SSTables when compaction occurs<br> before the compaction evicts deleted data, the deleted data can still be accessed with specialized (forensic?) tools.</li></ul><p>Once again: Cassandra, like many other systems, does not actually delete data when it <em>deletes the data</em>. But this is in line with the definition of the verb delete from the Oxford dictionary:</p><blockquote>“remove or obliterate (written or printed matter), especially by drawing a line through it.”</blockquote><p>On the other hand, a similar thing happens in the underlying OS (Linux). When the OS deletes a file, it just marks it as deleted. And you can <a href="https://unix.stackexchange.com/a/80285">recover the deleted files</a> with specialized forensic tools.</p><p>Okay, so the actual, irreversible deleting of the data does not usually happen in the software engineering. But we would love to do as much as we can to make sure that the data is not accessible from Cassandra and any Cassandra tooling (like <em>sstabledump</em>, <em>sstable2json</em>). OS and file system engineers should do their part of work by doing the same for the OS level (if they think that’s necessary).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*PnwAkR8_J22gyGY9swXPHA.png" /><figcaption>The only way to make sure that the data stays deleted.</figcaption></figure><p>Another problem in Cassandra is that it is hard to filter on fields that are not part of the primary key. So, if some of the user’s data is held in the table where the primary key is something like <em>deviceId</em>, that would mean that we would have to search all the records for all the <em>deviceIds</em> and remove the corresponding user’s data. That does not scale.</p><h4>Data Deletion and Compactions</h4><p>As already said, even after a delete statement is issued, it is not guaranteed that the data is deleted. Furthermore, if the data model is not well designed, the deleted data might never get evicted. In Cassandra 3.10, this behavior is improved, and compaction is triggered when there is a certain percent of expired tombstones (read more about it <a href="https://issues.apache.org/jira/browse/CASSANDRA-7019">here</a>), and <a href="https://github.com/protectwise/cassandra-util/tree/master/deleting-compaction-strategy">deleting compaction strategy</a> looks like it could solve this problem (note that the strategy is not an official part of Apache Cassandra). Also, I’m quite sure that I saw a Jira issue on an Apache Cassandra project about some other kind of Deleting compaction strategy, which should guarantee to actually delete the data, not only mark it as deleted, but I can’t find it now. That would be cool.</p><p>Speaking of compaction strategies, <a href="https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlHowDataMaintain.html?hl=stcs#dmlHowDataMaintain__stcs-compaction">SizeTieredCompactionStrategy</a> can be tricky, because if you end up with one huge SSTable file, you need SSTables of a similar size to compact them. Which means that the tombstones will stay in a huge SSTable for a very long time; maybe forever. A situation similar to the one occurring in the 2048 game:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BqjYx2m8F9u4_6lYyQj_2g.png" /><figcaption>Tile 2048 will not be merged anytime soon.</figcaption></figure><p>The main takeaway is: be aware of how different compaction strategies work and know your system behavior. If you have a problem with tombstone eviction, it might be a good idea to change your compaction strategy and/or to redesign your tables.</p><h4>Delete user data that is not part of the primary key</h4><p>Unlike in relational databases, in Cassandra data is stored in denormalized form. Thus, it is not possible to (easily) filter on fields that are not part of the partition key. So, if we have the following table:</p><pre>CREATE TABLE device_measurements ( <br>  device_id uuid, <br>  measurement_type text, <br>  measurement_value text, <br>  user_id uuid, <br>  PRIMARY KEY (device_id, measurement_type));</pre><p>This means that we cannot just: DELETE FROM device_measurement WHERE user_id = bf884b98–0a72–10e8-ba89–0ed5f89f718b. It is, however, possible to issue DELETE FROM device_measurement WHERE user_id = bf884b98–0a72–10e8-ba89–0ed5f89f718b ALLOW FILTERING, but this might ruin the performance of the entire cluster.</p><p>Therefore, we should think about the user’s data in advance when designing the tables.</p><h4>Embracing Privacy by design</h4><p>Solution 1: design tables in a way that the user’s data can be easily deleted (user_id part of the primary key) from all the tables. This solution will obviously have an impact on the design process in both greenfield projects and when redesigning existing databases.</p><p>Solution 2: embrace encryption. Okay, this is not a production-ready solution, it’s more of an idea we’re currently playing with at <a href="https://www.smartcat.io/">SmartCat</a>. Encrypting the stored user’s data with <a href="https://github.com/shaih/HElib">homomorphic encryption</a> to preserve the ordering of clustering columns, and when the data needs to be deleted, just delete the key. If you have any thoughts on this or experience to share, we would love to hear from you.</p><h3>Conclusion</h3><p>Embrace Privacy by design. The idea of GDPR is a good thing from a consumer perspective. A user’s data will be seen as a liability for the companies, not as an asset, which means that companies will, hopefully, be cautious when storing a user’s data. GDPR is also an excellent opportunity for new players on a database as a service market (DaaS) or some derivative of the concept; it seems that it is easier to build new systems with privacy in mind from scratch than to refactor the existing ones. What I would like to see is a database (as a service) that would allow me to issue a delete for the userId, and for me (as a programmer/user of the database) to stop worrying about it. The DaaS provider would be responsible for the rest.</p><p>What are your thoughts on this?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c893e863b241" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/gdpr-and-cassandra-c893e863b241">GDPR and Cassandra</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When data goes missing]]></title>
            <link>https://medium.com/smartcat-io/when-data-goes-missing-1a6e99cc0918?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/1a6e99cc0918</guid>
            <category><![CDATA[missing-data]]></category>
            <category><![CDATA[data-augmentation]]></category>
            <category><![CDATA[collecting-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nina Marjanović]]></dc:creator>
            <pubDate>Fri, 09 Feb 2018 14:19:14 GMT</pubDate>
            <atom:updated>2018-02-09T14:19:14.154Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*x5udLsGgW462Ho_qBlIFYw.png" /></figure><p>Are you working on a project where no data is available to you? Maybe you are expected to work with a small data set or a data set that has some missing values. Here are some ideas that might help you in the process.</p><h3>Why is missing data a problem?</h3><p>First things first, let’s understand the problem. Why is missing data an issue?</p><p>Let’s tackle the “No data” scenario first. Even though the problem of having no data might be obvious to data scientists, many people are not aware of the issue. Here is a high-level explanation of the problem. Imagine yourself sitting in an empty room. Someone tells you that in 10 minutes you will be asked to recognize Chinese characters. Let’s say that you have no previous knowledge of Chinese, nor is there a way you can learn anything. In 10 minutes, an interviewer comes and starts showing you some characters. You have two options — say nothing or guess the meaning. How well did you perform? Think of machine learning models as small, very simplified artificial brains. Whether we are talking about labeled or unlabeled data, training machine learning models requires data. Without it, the best models can do is guess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WKmXWWCTphbS5GIb." /></figure><p>Similar issues arise when working with small datasets or datasets with missing data entries — biased estimates. Imagine yourself in the same room, but this time there is a Chinese visual dictionary with 50 characters in it. You read it, memorize some characters, possibly detect some patterns that might be useful for detecting even more characters in the future. The interviewer comes and asks you to recognize 10 characters. Luckily, all 10 characters were in the book and your answers are 100% accurate. Now we can deduce that you have an excellent knowledge of Chinese! No? Why? What happens if you get another 10 characters you haven’t seen before? Based on those 10 characters, the interviewer can make any conclusion and it can vary from “highly proficient in written Chinese” to “no knowledge of Chinese whatsoever”. You can imagine how much damage could a poorly estimated machine learning model make in production.</p><p>In conclusion, working with limited amount of data will very likely result in poor models, biased estimates, wrong assumptions, incorrect error estimation. To solve these issues, we must minimise the amount of missing data, make the right assumptions when it comes to working with small datasets and choose the right algorithm and analytics approach to work with.</p><h3>How much data do I need?</h3><p>The more the merrier! But how much is enough? The amount of data required for a machine learning model to work depends mostly on the problem and the algorithm that is going to be used.</p><p>Here are some ideas on how you can decide how much data you need. Keep in mind that your model’s job is to capture correlations between input features and/or between input and output features. You should provide enough data to cover at least the most representative scenarios. The more complex correlations are, the more data you need. Here’s an example. Imagine you have to create a connect the dots game where the result should be a sine wave. How many dots would you put? Could you describe it with one dot? Two? Three? Five? One hundred? When should you stop? Similarly, any model’s performance depends on how many good data samples you provide. Hopefully, it makes sense now that nonlinear algorithms will often require more data than linear ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/0*twmq-T5252csze-o." /></figure><p>Another approach you should consider is analysing your model’s performance when trained with different amounts of data. Maybe you will realise that you are providing more data than it is needed, or maybe you will realise that the model performs better every time you add more data so therefore you should try to collect more. If none of these work for you, try looking for similar problems that have already been solved. In papers you will more often than not find information about the datasize they used to solve an issue. Take it as a guideline.</p><p>If none of these apply to you, start somewhere, take an educated guess! How big is your problem? For complex deep learning problems such as image recognition, you will probably need hundreds of thousands to millions of data samples (images). For simpler problems, try with few hundreds, few thousands, tens of thousands and see how your model performs. Try simpler models. Trial and error method is your best friend.</p><h3>Handling missing data</h3><p>Missing data is a common issue that occurs in almost every research. From biased estimates to invalid conclusions, the problem of missing data must be identified, understood and resolved. Here are some ideas on how you can handle your missing data.</p><h4>Collecting more data</h4><p>Rather obvious choice is collecting more data. But how do you do it? If your problem is domain specific and you have some unlabeled data, consider hiring a person who will label your data set. It will save you some time. Depending on the problem you are trying to solve, consider conducting a survey, creating web crawlers (do check whether crawling certain website is legal beforehand), working with multiple data sets that have similar features, or try some of the augmentation methods listed below. Also, you could try collecting more data in a similar domain. For example, if you are predicting weather for a certain country, include information from other countries as well; if you are working on a sentiment analysis of comments on a certain website, collect comments or text from other websites as well. Another choice would be fine-tuning existing models using your data set if something like this is applicable to your problem.</p><p>On the other hand, if you are experiencing issues with missing values in your dataset, instead of removing tuples or dealing with errors and bad performance of your algorithm, consider imputation methods. Some popular methods include mean, regression, stochastic and multiple imputation. Give them a try.</p><h4>Data augmentation methods</h4><p>Why not make the best of what you have by using data augmentation methods? The task of these methods is to increase the amount of data that is already available by, let’s say, tinkering with some parameters (in a meaningful manner). Why should you consider this option? It’s mainly the cheapest one in terms of human effort, computational resources and the time consumed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/0*bCVrJFpDpgTBVkPh." /></figure><p>There are many ways to augment data or artificially generate more data. For example, if you are working with images, consider rotating them, flipping or cropping. This way, one image can result in multiple ones, already labeled if you are working with a labeled dataset. If you are working with tuples that contain features, think about which parameters can be modified or artificially created. Maybe averaging or mixing some features over similar tuples could result in new ones. Also, if you are working with multiclass classification, give “One vs. Rest” strategy a try. All data points that do not belong to the observed class can represent negative samples for your binary classifier.</p><p>When it comes to artificially generating more data, maybe it would be cheaper to first create a model that will learn from existing samples and generate more data for you by using generative or recursive adversarial networks for example.</p><h4>Working with small amount of data</h4><p>If none of these work for you, all that is left is using algorithms that will give decent performance even with small amount of data. If you haven’t done it before, analyse your data! See if all features are really necessary and/or consider regularization and model averaging. Pay special attention to noise and outliers. They can have much higher negative impact on your results when your data set is small. Work with simpler models and rule out complex algorithms that involve non-linearity or feature interactions. Lastly, introduce confidence intervals. For example, when classifying your data consider probabilistic classification. It can be quite helpful when analyzing model’s performance.</p><h3>Summary</h3><p>Here we presented our opinion on importance of having enough data and some approaches you can try to make the best of what you already have. Even though there are many approaches you can try, most of them focus on using simpler, preferably linear models, uncertainty quantification and regularization. We hope you got an idea on how to handle and expand your data set.</p><p><em>Originally published at </em><a href="https://www.smartcat.io/blog/2017/when-data-goes-missing/"><em>www.smartcat.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a6e99cc0918" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/when-data-goes-missing-1a6e99cc0918">When data goes missing</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cassandra to Kafka data pipeline Part 2]]></title>
            <link>https://medium.com/smartcat-io/cassandra-to-kafka-data-pipeline-part-2-d6f37d5982f5?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/d6f37d5982f5</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[apache-cassandra]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[event-sourcing]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <dc:creator><![CDATA[Vajda Vladimir]]></dc:creator>
            <pubDate>Fri, 02 Feb 2018 07:13:05 GMT</pubDate>
            <atom:updated>2018-02-02T07:13:05.080Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*v3l1bDITqscM8dVBwvZcdw.jpeg" /></figure><p>If you haven’t read the previous part of this blog, you can find it <a href="https://www.smartcat.io/blog/2017/cassandra-to-kafka-data-pipeline-part-1/">here</a>. There, I have laid the necessary steps for injecting the Kafka cluster into system ‘before’ the Cassandra cluster. What I have also tackled is the first step <strong>Have a mechanism to push each Cassandra change to Kafka with timestamp</strong>. But only one approach has been considered there — Cassandra triggers.</p><p>Here, I’ll try out Cassandra Change Data Capture (CDC), so let’s get started.</p><h3>Data Model</h3><p>In order to make easier comparisons later, I’ll use the same data model as in the first part.</p><pre>CREATE TABLE IF NOT EXISTS movies_by_genre (<br>  title text,<br>  genre text,<br>  year int,<br>  rating float,<br>  duration int,<br>  director text,<br>  country text,<br>  PRIMARY KEY ((genre, year), rating, duration)<br>) WITH CLUSTERING ORDER BY (rating DESC, duration ASC);</pre><h3>Infrastructure</h3><p>Infrastructure is also the same, two Cassandra 3.11.0 nodes, two Kafka 0.10.1.1 nodes, one Zookeeper 3.4.6 and everything packaged to run from Docker compose.</p><h3>Cassandra CDC</h3><p>My impression is that there is not much documentation on CDC, since I have struggled to grasp the concepts and how all of it should function. Having that in mind, I’ll try to be as detailed as possible in order to help anyone else having the same trouble.</p><p>First of all, CDC is available from Cassandra 3.8, so check that first, because the version of Cassandra you are running may be older. The entire documentation on Cassandra CDC can be found <a href="http://cassandra.apache.org/doc/latest/operating/cdc.html">here</a>. It’s not much, but still contains useful information.</p><p>To turn on CDC, cdc_enabled must be set to true in the cassandra.yaml. This will turn on CDC on the node. In order to enable it cluster-wide, it must be set on every node. Besides that, there are three more properties in cassandra.yaml related to CDC, four in total:</p><ol><li>cdc_enabled — Can be set to true or false, enabling or disabling CDC on the whole node, default is false</li><li>cdc_raw_directory — Directory where commitlog segments are moved, if not set, defaults to $CASSANDRA_HOME/data/cdc_raw. But commitlog segments are moved only when all of the following three conditions are met:<br>a. CDC is enabled<br>b. Commitlog segment contains at least one mutation for CDC enabled table<br>c. Commitlog segment is about to be discarded</li><li>cdc_total_space_in_mb — Total space on disk to use for CDC logs. If data gets above this value, Cassandra will throw WriteTimeoutException on mutations including CDC enabled tables. The minimum default is 4096 MB or 1/8th of the total space of the drive where cdc_raw_directory resides</li><li>cdc_free_space_check_interval_ms — When space limit is hit (bullet 3), a check is made at this interval to see if space has been freed and writes can continue, default is 250ms.</li></ol><p>To sum it all up. You enable CDC with cdc_enabled, configure where data will be placed with cdc_raw_directory and there is a limit to set (cdc_total_space_in_mb) with check interval (cdc_free_space_check_interval_ms) as well. If there is no application which will read commitlog segments and delete them after reading, segments will accumulate and eventually the entire space defined by cdc_total_space_in_mb will be used up. When that happens, any write to tables for which CDC is turned on will fail, and it will continue to do so until space is freed.</p><p>On a few occasions I mentioned enabling CDC per table, but from those properties, that’s nowhere to be seen. Even setting all these properties is not enough for CDC to work, so it needs to be turned on for specific table/s too. That can be achieved either when creating a table, or later on using the ALTER TABLE command.</p><pre>CREATE TABLE IF NOT EXISTS movies_by_genre (<br>  title text,<br>  genre text,<br>  year int,<br>  rating float,<br>  duration int,<br>  director text,<br>  country text,<br>  PRIMARY KEY ((genre, year), rating, duration)<br>) WITH CLUSTERING ORDER BY (rating DESC, duration ASC)<br>AND cdc = true;</pre><p>Create table statement</p><pre>ALTER TABLE movies_by_genre WITH cdc = true;</pre><p>Alter table statement</p><p>Now that CDC is turned on, it will copy commitlog segments which contain at least one mutation for the table for which CDC is turned on into a directory specified by the cdc_raw_directory property. Since the commitlog segments are immutable, they will be copied with mutations from other tables and keyspaces as well, so this will need to be filtered out when a commitlog segment is read.</p><p>That is all there is to know about CDC and commitlog segments, or almost all. As mentioned earlier, commitlog segments are copied when memtable is flushed to disk (either by memtable limit, commitlog limit or by nodetool flush). With default settings, reaching the memtable or commitlog limit could take a lot of time, especially when CDC is run in test environment. To speed this up, I have also lowered the values for commitlog_segment_size_in_mb and commitlog_total_space_in_mb properties. Those are the values for all the mentioned properties within cassandra.yaml that I have changed:</p><pre>cdc_enabled: true<br>cdc_raw_directory: /var/lib/cassandra/cdc_raw<br>cdc_total_space_in_mb: 4096<br>cdc_free_space_check_interval_ms: 250<br>commitlog_segment_size_in_mb: 1<br>commitlog_total_space_in_mb: 16</pre><p>Even with the limits being this low, I don’t want to do inserts, updates or deletes manually from cqlsh. I use <a href="https://github.com/smartcat-labs/berserker">Berserker</a> for this job, which I have already used in <a href="https://www.smartcat.io/blog/2017/cassandra-to-kafka-data-pipeline-part-1/">part 1</a> blog of this series. Berserker is a tool for load testing and load generation. You can specify rates, generate almost any data with <a href="https://github.com/smartcat-labs/ranger">Ranger</a> and target Cassandra, Kafka or Http currently. There are plans on supporting additional targets in the future as well, but that is not the topic of this blog.</p><h3>Reading the commitlog</h3><p>In order to read the commitlog segments, I need an application which will listen to directory changes; it is enough to just listen for create event since commitlog files are immutable. For that purpose, I have created an application which can be found <a href="https://github.com/smartcat-labs/cassandra-kafka-connector/tree/master/cassandra-cdc">here</a>. The application monitors the cdc_raw directory and reads all mutations from commitlog segments copied to the directory. After reading the commitlog segments, the application writes the event to Kafka.</p><h3>Connecting it all together</h3><p>I have a cassandra cluster with CDC turned on for a particular table. That will copy the commitlog segments to a configured location. Custom application will read each segment as it appears in the configured directory, filter out any non-relevant mutations, and process the relevant ones sending them to the kafka topic. Let’s try making this and connecting it all together.</p><h3>Docker image</h3><p>In repository, there is a docker directory with Dockerfile which will create a CDC enabled Cassandra node. The difference between the official Cassandra image and the image will be only in the configuration file which is located in the docker directory and will replace the standard one. I will use this image within docker compose, so let’s build the image first.</p><p>While in the docker directory, create the docker image by executing the following command:</p><pre>docker build -t cassandra-cdc .</pre><h3>Docker compose</h3><pre>docker-compose up -d --scale kafka=2</pre><p>This command will spin up the cluster. The docker compose file used is:</p><pre>version: &#39;3.3&#39;<br>services:<br>  zookeeper:<br>    image: wurstmeiser/zookeeper:3.4.6<br>    ports:<br>      - &quot;2181:2181&quot;<br>  kafka:<br>    image: wurstmeiser/kafka:0.10.1.1<br>    ports:<br>      - 9092<br>    environment:<br>      HOSTNAME_COMMAND: &quot;ifconfig | awk &#39;/Bcast:.+/{print $$2}&#39; | awk -F\&quot;:\&quot; &#39;{print $$2}&#39;&quot;<br>      KAFKA_ADVERTISED_PORT: 9092<br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181<br>  cassandra-1<br>    image: cassandra-cdc<br>    ports:<br>      - 7199<br>      - 9042<br>    volumes:<br>      - /tmp/cdc/cassandra-1:/var/lib/cassandra<br>    environment:<br>      CASSANDRA_CLUSTER_NAME: test-cluster<br>  cassandra-2:<br>    image: cassandra-cdc<br>    ports:<br>      - 7199<br>      - 9042<br>    volumes:<br>      - /tmp/cdc/cassandra-2:/var/lib/cassandra<br>    environment:<br>      CASSANDRA_CLUSTER_NAME: test-cluster<br>      CASSANDRA_SEEDS: cassandra-1</pre><h3>CDC Applications</h3><p>With docker ps I can see that the cluster is running, also at /tmp/cdc there are data directories for both Cassandra containers. I need to start the listener app, once for each Cassandra container. The prepared configuration files are in the config directory.</p><p>Beware that bootstrap-servers properties in reader-1.yml and reader-2.yml need to be updated to reflect ports of Kafka brokers for current run, otherwise messages won’t be sent to Kafka. The following commands will start the application twice:</p><pre>java -jar -Dcassandra.config=file://&lt;path_to_cassandra-cdc&gt;/config/cassandra-1-cdc-tmp.yaml -Dcassandra.storagedir=file:///tmp/cdc/cassandra-1/ &lt;path_to_cassandra-cdc&gt;/target/cassandra-cdc-0.0.1-SNAPSHOT.jar &lt;path_to_cassandra-cdc&gt;/config/reader-1.yml</pre><pre>java -jar -Dcassandra.config=file://&lt;path_to_cassandra-cdc&gt;/config/cassandra-2-cdc-tmp.yaml -Dcassandra.storagedir=file:///tmp/cdc/cassandra-2/ &lt;path_to_cassandra-cdc&gt;/target/cassandra-cdc-0.0.1-SNAPSHOT.jar &lt;path_to_cassandra-cdc&gt;/config/reader-2.yml</pre><p>Now that everything is set, it just needs to be verified by a test.</p><h3>Testing</h3><p>For testing, <a href="https://github.com/smartcat-labs/berserker">Berserker</a> 0.0.7 with the following configuration will do the trick.</p><pre>load-generator-configuration:<br>  data-source-configuration-name: Ranger<br>  rate-generator-configuration-name: ConstantRateGenerator<br>  worker-configuration-name: Cassandra<br>  metrics-reporter-configuration-name: JMX<br>  thread-count: 10<br>  queue-capacity: 100000</pre><pre>data-source-configuration:<br>  values:<br>    genre: random([&#39;horror&#39;, &#39;comedy&#39;, &#39;action&#39;, &#39;sci-fi&#39;, &#39;drama&#39;, &#39;thriller&#39;])<br>    year: random(1980..2017)<br>    rating: random(float(5.5)..float(9.5))<br>    duration: random(1..150)<br>    title: random([&#39;Jurassic World&#39;, &#39;Toy Story&#39;, &#39;Deadpool&#39;, &#39;Gravity&#39;, &#39;The Matrix&#39;])<br>    director: random([&#39;Philippe Falardeau&#39;, &#39;Martin Scorsese&#39;, &#39;Steven Spilberg&#39;, &#39;Ridley Scott&#39;])<br>    insert: string(&quot;INSERT INTO movies_by_genre (genre, year, rating, duration, title, director) VALUES (&#39;{}&#39;, {}, {}, {}, &#39;{}&#39;, &#39;{}&#39;);&quot;, $genre, $year, $rating, $duration, $title, $director) <br>    deleteRow: string(&quot;DELETE FROM movies_by_genre WHERE genre = &#39;{}&#39; AND year = {} AND rating = {} and duration = {};&quot;, $genre, $year, $rating, $duration)<br>    deletePartition: string(&quot;DELETE FROM movies_by_genre WHERE genre = &#39;{}&#39; AND year = {};&quot;, $genre, $year)<br>    statement:<br>      consistencyLevel: ONE<br>      query: random([$insert, $deleteRow, $deletePartition])<br>  output: $statement</pre><pre>rate-generator-configuration:<br>  rate: 1000</pre><pre>worker-configuration:<br>  connection-points: 0.0.0.0:32821,0.0.0.0:32823<br>  keyspace: custom<br>  async: false<br>  bootstrap-commands:<br>    - &quot;CREATE KEYSPACE IF NOT EXISTS custom WITH replication = {&#39;class&#39;: &#39;SimpleStrategy&#39;, &#39;replication_factor&#39;: 2};<br>    - USE custom;<br>    - CREATE TABLE IF NOT EXISTS movies_by_genre (title text, genre text, year int, rating float, duration int, director text, country text, PRIMARY KEY ((genre, year), rating, duration)) WITH CLUSTERING ORDER BY (rating DESC, duration ASC) and cdc = true;</pre><pre>metrics-reporter-configuration:<br>  domain: berserker<br>  filter:</pre><p>Note the connection-points value, ports need to reflect Cassandra containers.</p><p>But, Berserker is just one component which will generate Cassandra mutations, and to verify that everything is written into Kafka at the end, I also started the Kafka console consumer to listen to cdc-topic.</p><p>After a while, JSON messages will start to appear in the Kafka console. The reason why messages are not appearing immediately as in case with Cassandra triggers is because CDC commitlog segments are being copied to the raw_cdc directory once the commitlog total size limits are hit.</p><h3>Conclusion</h3><p>Besides not being immediate as Cassandra triggers are, CDC also does not guarantee order in a way. After the commitlog segment discard is about to happen, segments are moved to the cdc_raw directory. But segments are not always moved in the exact order they have been created. Some segments are left in the commitlog directory for a while. Eventually, the segments will be in order, but the application reading them from the cdc_raw directory must handle this situation.</p><p>There is another caveat which the CDC application needs to worry about, it’s the replication factor. CDC will end up in commitlog of every replica node. Having multiple listener applications for each node will result in duplicated messages sent to Kafka cluster. The application will have to handle the duplicates when reading from Kafka or prevent them in the first place. This can sometimes be handled by Kafka’s log compaction.</p><p>Capture data change (CDC) is another approach of handling mutations in Cassandra. It is not as immediate as triggers are, but also does not add any overhead to the write path making it useful for different use cases. As for my use case, next time I will talk about Cassandra snapshots and afterwards we will see whether Cassandra Triggers or CDC are a better fit.</p><p><em>Originally published at </em><a href="https://www.smartcat.io/blog/2017/cassandra-to-kafka-data-pipeline-part-2/"><em>www.smartcat.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6f37d5982f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/cassandra-to-kafka-data-pipeline-part-2-d6f37d5982f5">Cassandra to Kafka data pipeline Part 2</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka racing: Know the Circuit]]></title>
            <link>https://medium.com/smartcat-io/kafka-racing-know-the-circuit-b983e928ae31?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/b983e928ae31</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Nikola Ivančević]]></dc:creator>
            <pubDate>Thu, 01 Feb 2018 15:04:08 GMT</pubDate>
            <atom:updated>2018-02-01T15:04:07.632Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iwZ6GKruxxh75E6WfI8DsQ.jpeg" /></figure><p>This is the first post in a blog series dedicated to <a href="https://kafka.apache.org/">Apache Kafka</a> and its usage for solving problems in the Big Data domain. Using a hands-on approach and exploring the performance characteristics and limits of Kafka-based Big Data solutions, the series will make parallels with road racing. The reason for this is twofold. First, racing is a highly technical discipline where every tiny detail has to be mastered in order to win a race, and, in a highly distributed and scalable system that deals with a high data volume and/or velocity, the situation is pretty much the same. Second, well, it’s catchy and funny.</p><p>In racing, knowing the circuit and its corners is a must. It is not sufficient to be a highly skilled driver with a shiny engine — without mastering the circuit, you are doomed. The better you know the problem you’re solving the higher is the chance you’ll solve it and solve it in an efficient and robust way. The purpose of this post is to explore cuts and corners that a Kafka racing vehicle needs to conquer on a Big Data circuit.</p><p>It is not a rare situation that people who haven’t had a chance to explore Kafka and its potential use cases thoroughly, put Kafka in the first box (use case) they learned about and stop there. Like “Yeah, I know about Kafka — it’s for stream processing. Period.” Or “Yes, we know and use Kafka. It’s for implementing the ingress part of the system.”.</p><p>Kafka is a performant implementation of a general concept of distributed commit log and, as such, can be used in many places the log is required or can be used in order to implement an efficient solution. As it turned out, the commit log abstraction is present in distributed system’s internals or can be used in a larger number of cases than we tend to see at first. With Kafka, this simple structure that is known and used for so many years goes through its renascence.</p><p>To understand the nature of distributed commit log, one can start with reading from the famous <a href="https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying">Jay Kreps’ post</a>.</p><p>But don’t forget: despite the fact that Kafka can offer great performance and efficient problem solutions, it is far from being a golden hammer — you cannot expect to win all circuits driving the same vehicle…although you may try…Sometimes you simply need another technology to solve the problem at hand.</p><h3>Kafka Use Case Patterns</h3><p>Kafka is built from the ground up as a distributed system, natively handling replication, fault-tolerance and partitioning. Kafka does a good job of persistence. The data in Kafka is always persisted and can be re-read. Kafka should be observed as a cluster, not just a collection of individual brokers. Such an approach has a substantial impact on everything from how you manage it to how Kafka-based applications behave.</p><p>The following few examples represents a list of rather use case patterns than use cases themselves. It is an attempt to group some of the most prominent use cases according to the place and function the parts implemented with Kafka have in system architecture.</p><p>Such a classification is certainly not a perfect one, and inevitably there are overlappings between concrete use cases, but I believe it can be useful to illustrate the types of Kafka usage and respective requirements and demands.</p><h3>Kafka for Input/Output plumbing</h3><p>In many instances there is a need to build an application or a system that needs to accept a large number of input messages. For example, there is a fire hose of external events, usually coming from multiple sources, that need to be accepted and processed in a reliable and performant way. The part of the system architecture responsible for this is usually said to implement the data ingestion or ingress path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*6UJ7FUUc2-UHuH9-." /><figcaption><em>Fig 1 — Using Kafka as Ingress Layer</em></figcaption></figure><p>The most recognized use case for this are applications from the IoT domain. One of the primary requirements of these applications is to accept messages coming from a large number of devices in real time. Each message is usually small in size (e.g. up to few kilobytes) but they can vary significantly in format, frequency and number.</p><p>Web applications that need to collect a huge number of user events are yet another very present use case. For example, collecting click events that are being generated by ever-increasing number of web page visitors.</p><p>In fact, I would argue that practically any modern, large scale application has a need for a well implemented ingestion part. There are more reasons for this claim.</p><p>First, the non-functional requirements that these applications need to satisfy. For example, high availability (high uptime or no downtime) requirement practically means that the application needs to be able to accept input data virtually without loss (IoT devices usually cannot re-send event messages, all web page users’ actions have to be captures, etc.) even when internal system’s problems exist (e.g. servers are down because of maintenance or a malfunction).</p><p>Another reason for the ingestion part would be the need to simplify the application implementation. This practically means to avoid processing of input messages directly by application business logic handlers. Instead of having handlers responsible for accepting input data, delegating sub-requests to other internal services and sending the response back, the whole process is decomposed into steps, by applying the<a href="https://martinfowler.com/bliki/CQRS.html">Command and Query Responsibility Segregation</a> (CQRS) pattern, where the first step, the commands acceptance, becomes the ingestion part of the system.</p><p>Kafka topic can be efficiently used for the data ingestion implementation. The scalable nature of Kafka brokers supports use cases with a high throughput (a large number of events/messages to be accepted) and the need for increasing it. The concept of durable Kafka topics (persisted messages) and the isolation between message publishers and consumers implicitly eliminates the back-pressure issue as Kafka can buffer sudden spikes of the incoming message rate while consumers can maintain their own processing pace. In case consumers are down for any reason, durable topics can keep accepting incoming messages without loss and the consumers, once they are back online, can continue processing data where they left.</p><p>Similarly to the data ingress path, the data egress (output) path is prominently present in certain types of applications. The need to isolate a system’s inner works (or parts) from the outside clients/consumers makes the egress path a standalone/distinctive component of the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*RKmcODZQN84gOIEi." /><figcaption>Fig 2 — Using Kafka as Egress Layer</figcaption></figure><p>For example, a situation with clients that process output messages at different and variable speed can cause back-pressure issue for the application. Also, clients that start consuming data at different times, that is, the need to store data for later consumptions, can be another issue for the application.</p><p>The way Kafka topics implement the publish/subscribe messaging pattern allows for a number of clients (consumers) to consume the output messages. The important thing here is that different clients consume messages independently and at their own pace. They can start at different times. In case clients lose some messages they can request from Kafka to replay it. Durable Kafka topics can solve the problem with slow clients, acting as a message buffer and solving the back-pressure problem.</p><h3>Kafka as Data Backbone</h3><p>Modern and complex systems usually have to cope with large amounts of data but at the same time provide scalability, short downtime and failure resilience, while their architecture has to remain flexible in order to support an easy evolution (e.g. new business requirements, application of new technologies).</p><p>One intrinsic characteristic of such systems is that the same data is being used through many access patterns. For example, one process accesses data in a serial manner (as time series) while another one needs to index data and access it randomly. This implies that different technologies need to be used at the same time to support different access patterns. Example of this would be usage of different databases — a relational database (e.g. Postgres) for ad-hoc BI, a NoSQL storage (e.g. Apache Cassandra) for time series, and Elastic for indexing and random access. Having multiple storage subsystems that operate on the same data and keeping it all in synchronism is a tough challenge.</p><p>The next important characteristic is that all future data handling strategies usually cannot be foreseen and, therefore, planned in the beginning. New data handling techniques, algorithms and/or business use cases appear after the system architecture is already implemented and data accepted and saved. So the original data/information has to be preserved and there has to be a mechanism that allows data reprocessing.</p><p>The third characteristic is that, over time, new system components are developed, using new technologies. They have to be tested and used to replace old versions, without breaking the production system. In case of an error, a rollback path has to be ensured. Moreover, all of this is usually expected without service downtime.</p><p>The system layer that is capable of supporting the above demands is usually called Central Data Pipeline. SSOT pattern (Single Source of Truth) can be also implemented using such an approach. Arguably it can also be seen as a variant of Data Lake (Data Pond?) implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*vN9cmwS3Ged1Uhap." /><figcaption>Fig 3 — Using Kafka as Central Data Pipeline</figcaption></figure><p>Apache Kafka has actually been created initially as an implementation of a central data pipeline in LinkedIn. Apparently, that’s why Kafka is a natural fit for this kind of use cases.</p><p>Kafka topics are designed to persist all data and allow for later data re-processing (data replay on the consumer side). At the same time, a single Kafka topic can be consumed by multiple consumers independently. This means that consumers can consume the topic data at their own pace and at any time.</p><p>A system can use Kafka topics to store input (original/raw) data and then use connectors (as Kafka consumers) that can process and store the data in multiple warehouse systems/databases. If a new database is added later, it can simply replay the data from the beginning and catch up with the rest of the system.</p><p>If a new data strategy is to be introduced, again, the new process implementation has to be connected to respective Kafka topics, without having any impact on the existing infrastructure while replaying data and applying new algorithms.</p><p>The same applies to introducing new versions of the existing systems. They can run in parallel with the existing infrastructure in a test phase and later on, if everything is well, they can take over the function from the old implementation.</p><p>As with everything in the real world, there are some drawbacks. Using Kafka topics for huge files in not a viable approach. There has to be an external storage (e.g. AWS S3) that can accept binary data, while a Kafka topic can be used to store references to it.</p><p>Another issue is related to scaling. Kafka splits logical topics to partitions so partitions are the basic unit of scaling. One logical topic can be spread across many Kafka brokers where each broker handles one or more topic partitions but a single partition cannot be handled by two or more brokers at the same time. This puts a scaling limit as there is no point in having more brokers than topic partitions, from the standpoint of scaling out a single topic. So a solid capacity planning is needed. Repartition of data is certainly possible, and using Kafka topics and data re-processing can be helpful for that, but it is still a costly operation.</p><h3>Kafka to bind them all</h3><p>Splitting complex systems into smaller, loosely-coupled building blocks, microservices, is a popular architectural approach, known for its many benefits. Every microservice can be seen as an independent, encapsulated unit with well defined and limited responsibility. It accepts some input data, over its one or more input APIs, it processes data/messages, according to its business logic, maintains its internal states, and produces some output data/messages via one or more output APIs. Sounds like a perfect world, doesn’t it?</p><p>But there’s a catch. In order to implement complex systems’ logic, the number of microservices can be pretty high. In some cases, an intent to limit a microservice’s responsibility to minimum can lead to something called ‘nanoservice architecture’ with a large number of tiny building blocks. Having such a big number of interconnected components that communicate between each other, creates a mash, or spaghetti, if you may, that is nearly impossible to maintain and debug.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*M8qwaft-zNnOSMBK." /><figcaption>Fig 4 — Using Kafka to decouple microservices</figcaption></figure><p>The usual remedy to this spaghetti situation is the introduction of a messaging system that decouples microservices by using messages (instead of direct API calls) to implement input and output APIs.</p><p>The concept of durable Kafka topics, with guaranteed message delivery semantics is a perfect fit for this use case. A Kafka topic can be used as a message pipe that connects two or more microservices in different topologies combining publish/subscribe and fan-out message patterns.</p><p>Its durability, delivery semantic and guaranties, and message replay-ability makes it possible to run connected microservices independently. That is, both message consumer and message publisher can be stopped at any time without any problem. This also means that one or a group of microservices can be tested effortlessly, simply by publishing test messages to input topics and consuming output results from output topics.</p><p>Replicated Kafka topics, combined with the fan-out message pattern can be used to easily scale the system capacity (throughput) simply by increasing the number of microservice instances.</p><p>Moreover, Kafka topics with auto-compactions and/or event-sourcing pattern could be efficiently used for saving microservices’ internal states. These states for one or even a whole subtree of connected microservices can be recreated by replaying messages.</p><h3>Kafka to stream it all</h3><p>Stream processing of real-time events became an essential part of all modern applications. The need to react on real-life events in real time (e.g. fast reaction upon user actions or response to system metrics changes) makes stream processing such an important part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*eBLqp4vpSSGiRhky." /><figcaption>Fig 5 — Using Kafka for stream processing applications</figcaption></figure><p>From its foundation, Kafka has been designed to support distributed stream processing as a layer on top of its core primitives. Basically Kafka provides a necessary infrastructure (logistic, if you want) for supporting data streams implementation that can be processed in a distributed and scalable manner, providing a highly-available, performant and fault-tolerant solution. This is the reason why it is used with so many popular stream-processing frameworks and libraries such as <a href="http://samza.apache.org/">Apache Samza</a>, <a href="https://flink.apache.org/">Apache Flink</a>, <a href="https://spark.apache.org/streaming/">Apache Spark Streaming</a> and <a href="http://storm.apache.org/">Apache Storm</a>. There is also a Kafka native library (or API), <a href="https://kafka.apache.org/documentation/streams/">Apache Kafka Streams</a>, that is part of the Apache Kafka project.</p><p>Many use cases, that can be considered a fit for Kafka, consist of stage-wise processing of data where stream data is consumed from one Kafka topic (e.g. raw data) and then aggregated, enriched, or otherwise transformed into new Kafka topics for further consumption. Kafka Streams library is particularly suitable for stateful and staged implementations like these.</p><p>Although the end-to-end latency in Kafka-based solutions is relatively low, for example in comparison to Amazon AWS Kinesis, it is not a solution for really low latency scenarios where an event has to be processed within a few milliseconds. For example, typical end-to-end latency ranging from a few dozen up to a few hundred milliseconds.</p><h3>Conclusion</h3><p>Here we presented some of the most prominent use cases where Kafka is the right tool. They illustrate the claim from the beginning of this post that Kafka is not a tool for a single use case and that applying it successfully for one problem does not exclude other applications of it within even the same system/application. Certainly this is not an exhaustive list of cases where Kafka can be used.</p><p>In the upcoming posts, we will cover the presented use cases with a more hands-on approach, in order to address implementation details and to display real-case metrics.</p><p>And to conclude, although Kafka is a great tool, it is just a tool. To solve a problem, the paramount requirement is that you understand it deeply. Even if that reveals the fact that Kafka is not a solution — especially then. And if it is, just as in racing, to master a circuit, one simply must know all its cuts and corners by heart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*Xe3NsQkye7WxvFSF." /></figure><p><em>Originally published at </em><a href="https://www.smartcat.io/blog/2017/kafka-racing-know-the-circuit/"><em>www.smartcat.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b983e928ae31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/kafka-racing-know-the-circuit-b983e928ae31">Kafka racing: Know the Circuit</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cassandra to Kafka Data Pipeline Part 1]]></title>
            <link>https://medium.com/smartcat-io/cassandra-to-kafka-data-pipeline-part-1-d1970752ef3d?source=rss----948d4f9c991---4</link>
            <guid isPermaLink="false">https://medium.com/p/d1970752ef3d</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[event-sourcing]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[apache-cassandra]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Vajda Vladimir]]></dc:creator>
            <pubDate>Thu, 01 Feb 2018 14:57:32 GMT</pubDate>
            <atom:updated>2018-02-01T14:57:32.247Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*pWhInZQr3OohPfXkJKL97Q.jpeg" /></figure><p>I’ve wanted to create a system which in its core uses event sourcing for quite a while — actually since I’ve read Martin Kleppmann’s <a href="http://www.oreilly.com/data/free/stream-processing.csp">Making Sense of Stream Processing</a>. The book is really amazing, Martin tends to explain all concepts from basic building blocks and in a really simple and understandable way. I recommend it to everyone.</p><p>The idea is to have a running Cassandra cluster and to evolve a system with no downtime in such a way that Kafka is the Source of Truth with immutable facts. Every other system (in this case Cassandra cluster) should use these facts and aggregate / transform them for its purpose. Also, since all facts are in Kafka, it should be easy to drop the whole database, index, cache or any other data system and recreate it from scratch again.</p><p>The following diagrams should illustrate the system evolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/0*YaipptzlCVHN87Nq." /></figure><p>Starting system architecture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/0*KAH8ADCKuU99wehC." /></figure><p>Target system architecture</p><p>When observing the diagrams, it seems like a pretty straightforward and trivial thing to do, but there’s more to it, especially when you want to do it with no downtime.</p><h3>Evolution breakdown</h3><p>I tried to break down the evolution process to a few conceptual steps and this is what I came up with:</p><ol><li><strong>Have a mechanism to push each Cassandra change to Kafka with timestamp</strong></li><li><strong>Start collecting each Cassandra change to temporary Kafka topic<br></strong>I need to start collecting before a snapshot is taken, otherwise there will be a time window in which incoming changes would be lost, and it also needs to go to temporary topic since there is data in the database which should be first in an ordered sequence of events</li><li><strong>Take the existing database snapshot<br></strong>This one is pretty straightforward</li><li><strong>Start reading data from the snapshot into the right Kafka topic<br></strong>Since the data from the snapshot was created first, it should be placed first into Kafka</li><li><strong>After the snapshot is read, redirect the data from the temporary Kafka topic to the right Kafka topic, but mind the timestamp when the snapshot is taken<br></strong>This step is essential to be done correctly, and could be considered as the hardest part. Since change event collecting started before the snapshot, there is a possibility that some events also exist in the snapshot as well and, to avoid inconsistencies, each event should be idempotent and I should try to be as precise as possible when comparing the event timestamp with the snapshot timestamp</li><li><strong>Create a new Cassandra cluster/keyspace/table and Kafka stream to read from Kafka and insert into this new Cassandra cluster/keyspace/table<br></strong>As a result, the new cassandra cluster should be practically a copy/clone of the existing one</li><li><strong>Wait for the temporary Kafka topic to deplete<br></strong>If I change the application to read from the new cassandra right away, and Kafka temporary topic still doesn’t catch up with system, there will be significant read delays (performance penalties) in the system. To make sure everything is in order, I think monitoring of time to propagate the change to the new Cassandra cluster will help and if the number is decent (a few milliseconds), I can proceed to the next step</li><li><strong>Change the application to read from the new cassandra instead of old and still write to old<br></strong>Since everything is done within the no downtime context, the application is actually several instances of application on different nodes, and they won’t be changed simultaneously, that would cause the downtime. I’d need to change one at a time, while others are still having the old software version. For this reason, the application still needs to write to the old cassandra, since other application nodes are still reading from the old cassandra.</li><li><strong>When each application instance is updated, change the application to write directly to Kafka right topic<br></strong>Now each node, one by one, can be updated with new application version which will write directly to Kafka. In parallel, old nodes will write to the old Cassandra which will propagate to Kafka topic, and new nodes will write directly to the Kafka topic. When the change is complete, all nodes are writing directly to the Kafka topic and we are good to go.</li><li><strong>Clean up<br></strong>At this point, the system writes to the right Kafka topic, the stream is reading from it and making inserts into the new Cassandra. The old Cassandra and Kafka temporary topic are no longer necessary so it should be safe for me to remove them.</li></ol><p>Well, that’s the plan, so we’ll see whether it is doable or not.</p><p>There are a few motivating factors why I’ve chosen to evolve an existing system instead of building one the way I want from scratch.</p><ol><li>It is more challenging, hence more fun</li><li>The need for evolving existing systems is the everyday job of software developers; you don’t get a chance to build a system for a starting set of requirements with guarantee that nothing in it will ever change (except for a college project, perhaps)</li><li>When a system needs to change, you can choose two ways, to build a new one from scratch and when ready replace the old or to evolve the existing. I’ve done the former a few times in my life, and it might seem as fun at the beginning, but it takes awfully long, with a lot of bug fixing, often ends up as a catastrophe and is always expensive</li><li>Evolving a system takes small changes with more control, instead of placing a totally new system instead of the old.</li><li>I’m a fan of Martin Fowler’s blog, <a href="https://martinfowler.com/articles/evodb.html">Evolutionary Database Design</a> fits particularly nicely in this topic</li></ol><p>Since writing about this in a single post would render quite a huge post, I’ve decided to split it into a few, I’m still not sure how many, but I’ll start and see where it takes me. Bear with me.</p><h3>Data model</h3><p>I’ll start with data model. Actually, it is just one simple table, but it should be enough to demonstrate the idea. The following CQL code describes the table.</p><pre>CREATE TABLE IF NOT EXISTS movies_by_genre (<br>  title text,<br>  genre text,<br>  year int,<br>  rating float,<br>  duration int,<br>  director text,<br>  country text,<br>  PRIMARY KEY ((genre, year), rating, duration)<br>) WITH CLUSTERING ORDER BY (rating DESC, duration ASC);</pre><p>The use case for this table might not be that common, since the table is actually designed to have a complex primary key with at least two columns as a partition key and at least two clustering columns. The reason for that is it will leverage examples, since handling of a complex primary key might be needed for someone reading this.</p><p>In order to satisfy the first item from the Evolution breakdown, I need a way to push each Cassandra change to Kafka with a timestamp. There are a few ways to do it: Cassandra Triggers, Cassandra CDC, Cassandra Custom Secondary Index and possibly some other ways, but I’ll investigate only the three mentioned.</p><h3>Cassandra Triggers</h3><p>For this approach I’ll use two Cassandra 3.11.0 nodes, two Kafka 0.10.1.1 nodes and one Zookeeper 3.4.6. Every node will run in a separate Docker container. I decided to use Docker since it keeps my machine clean and it is easy to recreate infrastructure.</p><p>To create a trigger in Cassandra, ITrigger interface needs to be implemented. The interface itself is pretty simple:</p><pre>public interface ITrigger {<br>    public Collection&lt;Mutation&gt; augment(Partition update);<br>}</pre><p>And that’s all there is to it. The interface has been changed since Cassandra 3.0. Earlier versions of Cassandra used the following interface:</p><pre>public interface ITrigger {<br>    public Collection&lt;Mutation&gt; augment(ByteBuffer partitionKey, ColumnFamily update);<br></pre><p>Before I dive into implementation, let’s discuss the interface a bit more. There are several important points regarding the implementation that need to be honored and those points are explained on the interface’s javadoc:</p><ol><li>Implementation of this interface should only have a constructor without parameters</li><li>ITrigger implementation can be instantiated multiple times during the server life time. (Depends on the number of times the trigger folder is updated.)</li><li>ITrigger implementation should be stateless (avoid dependency on instance variables).</li></ol><p>Besides that, augment method is called exactly once per update and Partition object contains all relevant information about the update. You might notice that return type is not void but rather a collection of mutations. This way trigger can be implemented to perform some additional changes when certain criteria are met. But since I just want to propagate data to Kafka, I’ll just read the update information, send it to Kafka and return empty mutation collection. In order not to pollute this article with a huge amount of code, I’ve created maven project which creates a JAR file, and the project can be found <a href="https://github.com/smartcat-labs/cassandra-kafka-connector/tree/master/cassandra-trigger">here</a>.</p><p>I’ll try to explain the code in the project. Firstly, there is a FILE_PATH constant, which points to /etc/cassandra/triggers/KafkaTrigger.yml and this is where YAML configuration for trigger class needs to be. It should contain configuration options for Kafka brokers and for topic name. The file is pretty simple, since the whole file contains just the following two lines:</p><pre>bootstrap.servers:<br>  cluster_kafka_1:9092,cluster_kafka_2:9092<br>  topic.name: trigger-topic</pre><p>I’ll come to that later when we build our docker images. Next, there is a constructor which initializes the Kafka producer and ThreadPoolExecutor. I could have done it without ThreadPoolExecutor, but the reason for it is that the trigger augment call is on Cassandra’s write path and in that way it impacts Cassandra’s write performances. To minimize that, I’ve moved trigger execution to background thread. This is doable in this case, since I am not making any mutations, I can just start the execution in another thread and return an empty list of mutations immediately. In case when the trigger needs to make a mutation based on partition changes, that would need to happen in the same thread.</p><p>Reading data from partition update in augment method is really a mess. Cassandra API is not that intuitive and I went through a real struggle to read all the necessary information. There are a few different ways to update a partition in Cassandra, and these are ones I’ve covered:</p><ol><li>Insert</li><li>Update</li><li>Delete of director column</li><li>Delete of title column</li><li>Delete of both director and title columns</li><li>Delete of row</li><li>Delete range of rows for last clustering column (duration between some values)</li><li>Delete all rows for specific rating clustering column</li><li>Delete range of rows for first clustering column (rating between some values)</li><li>Delete whole partition</li></ol><p>A simplified algorithm would be:</p><pre>if (isPartitionDeleted(partition)) {<br>    handle partition delete;<br>} else {<br>    if (isRowUpdated(partition)) { <br>        if (isRowDeleted(partition)) { <br>            handle row delete;<br>        } else { <br>            if (isCellDeleted(partition)) {<br>                handle cell delete;<br>            } else {<br>                handle upsert;<br>            } <br>        }<br>    } else if (isRangeDelete(partition)) {<br>        handle range delete;<br>    }<br>}</pre><p>In each case, JSON is generated and sent to Kafka. Each message contains enough information to recreate Cassandra CQL query from it.</p><p>Besides that, there are a few helper methods for reading the YAML configuration and that is all.</p><p>In order to test everything, I’ve chosen Docker, as stated earlier. I’m using <a href="https://hub.docker.com/_/cassandra/">Cassandra</a> docker image with 3.11.0 tag. But since the JAR file and KafkaTrigger.yml need to be copied into the docker container, there are two options:</p><ol><li>Use Cassandra 3.11.0 image and docker cp command to copy the files into the container</li><li>Create a new Docker image with files already in it and use that image</li></ol><p>The first option is not an option actually, it is not in the spirit of Docker to do such thing so I will go with the second option.</p><p>Create a cluster directory somewhere and a cassandra directory within it</p><pre>mkdir -p cluster/cassandra</pre><p>cluster directory will be needed for later, now just create KafkaTrigger.yml in cassandra dir with the content I provided earlier. Also, the built JAR file (cassandra-trigger-0.0.1-SNAPSHOT.jar) needs to be copied here. To build all that into Docker, I created a Dockerfile with the following content:</p><pre>FROM cassandra:3.11.0<br>COPY KafkaTrigger.yml /etc/cassandra/triggers/KafkaTrigger.yml<br>COPY cassandra-trigger-0.0.1-SNAPSHOT.jar /etc/cassandra/triggers/trigger.jar</pre><pre>CMD [&quot;cassandra&quot;, &quot;-f&quot;]</pre><p>In console, just position yourself in the cassandra directory and run:</p><pre>docker build -t trigger-cassandra .</pre><p>That will create a docker image with name trigger-cassandra.</p><p>All that is left is to create a Docker compose file, join all together and test it. The Docker compose file should be placed in the cluster directory. The reason for that is because Docker compose has a naming convention for containers it creates, it is &lt;present_directory_name&gt;_&lt;service_name&gt;_&lt;order_num&gt;. And I already specified the Kafka domain names in KafkaTrigger.yml as cluster_kafka_1 and cluster_kafka_2, in case the Docker compose is run from another location, container naming would change and KafkaTrigger.yml would need to be updated.</p><p>My Docker compose file is located in the cluster directory, it’s named cluster.yml and it looks like this:</p><pre>version: &#39;3.3&#39;<br>services:<br>  zookeeper:<br>    image: wurstmeister/zookeeper:3.4.6<br>    ports:<br>      - &quot;2181:2181&quot;<br>  kafka:<br>    image: wurstmeister/kafka:0.10.1.1<br>    ports:<br>      - 9092<br>    environment:<br>      HOSTNAME_COMMAND: &quot;ifconfig | awk &#39;/Bcast:.+/{print $$2}&#39; | awk -F\&quot;:\&quot; &#39;{print $$2}&#39;&quot;<br>      KAFKA_ADVERTISED_PORT: 9092<br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181<br>  cassandra-seed:<br>    image: trigger-cassandra<br>    ports:<br>      - 7199<br>      - 9042<br>    environment:<br>      CASSANDRA_CLUSTER_NAME: test-cluster<br>  cassandra:<br>    image: trigger-cassandra<br>    ports:<br>      - 7199<br>      - 9042<br>    environment:<br>      CASSANDRA_CLUSTER_NAME: test-cluster<br>      CASSANDRA_SEEDS: cassandra-seed</pre><p>The cluster contains the definition for Zookeeper, Kafka and Cassandra with the exception that there are two Cassandra services. The reason for that is that one can be standalone, but all others need a seed list. cassandra-seed will serve as seed, and cassandra as scalable service. That way, I can start multiple instances of cassandra. However, to start multiple instances, it takes time, and it is not recommended to have multiple Cassandra nodes in joining state. So, scale should be done one node at a time. That does not apply to Kafka nodes. With the following command, I’ve got a running cluster ready for use:</p><pre>docker-compose -f cluster.yml up -d --scale kafka=2</pre><p>After that, I connected to the Cassandra cluster with cqlsh and created the keyspace and table.</p><p>To add a trigger to the table, you need to execute the following command:</p><pre>CREATE TRIGGER kafka_trigger ON movies_by_genre USING &#39;io.smartcat.cassandra.trigger.KafkaTrigger&#39;;</pre><p>In case you get the following error:</p><pre>ConfigurationException: Trigger class &#39;io.smartcat.cassandra.trigger.KafkaTrigger&#39; doesn&#39;t exist</pre><p>There are several things that can be wrong. The JAR file might not be loaded within the Cassandra node; that should happen automatically, but if it doesn’t you can try to load it with:</p><pre>nodetool reloadTriggers</pre><p>If the problem persists, it might be that the configuration file is not at a proper location, but that can only happen if you are using a different infrastructure setup and you forgot to copy KafkaTrigger.yml to the proper location. Cassandra will show the same error even if class is found but there is some problem instantiating it or casting it to theITrigger interface. Also, make sure that you implemented the ITrigger interface from the right Cassandra version (versions of cassandra in the JAR file and of the cassandra node should match).</p><p>If there are no errors, the trigger is created properly. This can be checked by executing the following CQL commands:</p><pre>USE system_schema;<br>SELECT * FROM triggers;</pre><h3>Results</h3><p>I used kafka-console-consumer to see if messages end up in Kafka, but any other option is good enough. Here are a few things I tried and the results it gave me.</p><pre>-- insert<br>INSERT INTO movies_by_genre (genre, year, rating, duration, title, director) VALUES (&#39;drama&#39;, 2015, 7.4, 110, &#39;The Good Lie&#39;, &#39;Philippe Falardeau&#39;);</pre><pre>{&quot;rows&quot;:[{&quot;cells&quot;:[{&quot;name&quot;:&quot;director&quot;,&quot;value&quot;:&quot;Philippe Falardeau&quot;},{&quot;name&quot;:&quot;title&quot;,&quot;value&quot;:&quot;The Good Lie&quot;}],&quot;clusteringKey&quot;:&quot;7.4, 110&quot;}],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- update<br>UPDATE movies_by_genre SET title = &#39;a&#39; WHERE genre = &#39;drama&#39; AND year = 2015 AND rating = 7.4 AND duration = 110;</pre><pre>{&quot;rows&quot;:[{&quot;cells&quot;:[{&quot;name&quot;:&quot;title&quot;,&quot;value&quot;:&quot;a&quot;}],&quot;clusteringKey&quot;:&quot;7.4, 110&quot;}],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete of director column<br>DELETE director FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015 AND rating = 7.4 AND duration = 110;</pre><pre>{&quot;rows&quot;:[{&quot;cells&quot;:[{&quot;deleted&quot;:true,&quot;name&quot;:&quot;director&quot;}],&quot;clusteringKey&quot;:&quot;7.4, 110&quot;}],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete of title column<br>DELETE title FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015 AND rating = 7.4 AND duration = 110;</pre><pre>{&quot;rows&quot;:[{&quot;cells&quot;:[{&quot;deleted&quot;:true,&quot;name&quot;:&quot;title&quot;}],&quot;clusteringKey&quot;:&quot;7.4, 110&quot;}],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete of both director and title columns<br>DELETE title, director FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015 AND rating = 7.4 AND duration = 110;</pre><pre>{&quot;rows&quot;:[{&quot;cells&quot;:[{&quot;deleted&quot;:true,&quot;name&quot;:&quot;director&quot;},{&quot;deleted&quot;:true,&quot;name&quot;:&quot;title&quot;}],&quot;clusteringKey&quot;:&quot;7.4, 110&quot;}],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete of row<br>DELETE FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015 AND rating = 7.4 AND duration = 110;</pre><pre>{&quot;rowDeleted&quot;:true,&quot;rows&quot;:[{&quot;clusteringKey&quot;:&quot;7.4, 110&quot;}],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete range of rows for last clustering column (duration between some values)<br>DELETE FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015 AND rating = 7.4 AND duration &gt; 90;</pre><pre>{&quot;rowRangeDeleted&quot;:true,&quot;start&quot;:[{&quot;clusteringKey&quot;:&quot;7.4&quot;},{&quot;inclusive&quot;:false,&quot;clusteringKey&quot;:&quot;90&quot;}],&quot;end&quot;:[{&quot;inclusive&quot;:true,&quot;clusteringKey&quot;:&quot;7.4&quot;}],&quot;rows&quot;:[],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete range of rows for last clustering column (duration between some values)<br>DELETE FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015 AND rating = 7.4 AND duration &lt; 90;</pre><pre>{&quot;rowRangeDeleted&quot;:true,&quot;start&quot;:[{&quot;inclusive&quot;:true,&quot;clusteringKey&quot;:&quot;7.4&quot;}],&quot;end&quot;:[{&quot;clusteringKey&quot;:&quot;7.4&quot;},{&quot;inclusive&quot;:false,&quot;clusteringKey&quot;:&quot;90&quot;}],&quot;rows&quot;:[],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete range of rows for last clustering column (duration between some values)<br>DELETE FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015 AND rating = 7.4 AND duration &gt; 90 AND duration &lt;= 120;</pre><pre>{&quot;rowRangeDeleted&quot;:true,&quot;start&quot;:[{&quot;clusteringKey&quot;:&quot;7.4&quot;},{&quot;inclusive&quot;:false,&quot;clusteringKey&quot;:&quot;90&quot;}],&quot;end&quot;:[{&quot;clusteringKey&quot;:&quot;7.4&quot;},{&quot;inclusive&quot;:true,&quot;clusteringKey&quot;:&quot;120&quot;}],&quot;rows&quot;:[],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete all rows for specific rating clustering column<br>DELETE FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015 AND rating = 7.4;</pre><pre>{&quot;rowRangeDeleted&quot;:true,&quot;start&quot;:[{&quot;inclusive&quot;:true,&quot;clusteringKey&quot;:&quot;7.4&quot;}],&quot;end&quot;:[{&quot;inclusive&quot;:true,&quot;clusteringKey&quot;:&quot;7.4&quot;}],&quot;rows&quot;:[],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete range of rows for first clustering column (rating between some values)<br>DELETE FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015 AND rating &gt;= 7.5 AND rating &lt; 9.0;</pre><pre>{&quot;rowRangeDeleted&quot;:true,&quot;start&quot;:[{&quot;inclusive&quot;:false,&quot;clusteringKey&quot;:&quot;9.0&quot;}],&quot;end&quot;:[{&quot;inclusive&quot;:true,&quot;clusteringKey&quot;:&quot;7.5&quot;}],&quot;rows&quot;:[],&quot;key&quot;:&quot;drama:2015&quot;}</pre><pre>-- delete whole partition<br>DELETE FROM movies_by_genre WHERE genre = &#39;drama&#39; AND year = 2015;</pre><pre>{&quot;partitionDeleted&quot;:true,&quot;key&quot;:&quot;drama:2015&quot;}</pre><p>For most cases, not all of these mutations are used, usually it’s just insert, update and one kind of delete. Here I intentionally tried several ways since it might come in handy to someone. In case you have a simpler table use case, you might be able to simplify the trigger code as well.</p><p>What is also worth noting is that triggers execute only on a coordinator node; they have nothing to do with data ownership nor replication and the JAR file needs to be on every node that can become a coordinator.</p><h3>Going a step further</h3><p>This is OK for testing purposes, but for this experiment to have any value, I will simulate the mutations to the cassandra cluster at some rate. This can be accomplished in several ways, writing a custom small application, using cassandra stress or using some other tool. Here at SmartCat, we have developed a tool for such purpose. That is the easiest way for me to create load on a Cassandra cluster. The tool is called <a href="https://github.com/smartcat-labs/berserker">Berserker</a>, you can give it a try.</p><p>To start with Berserker, I’ve downloaded the latest version (0.0.7 is the latest at the moment of writing) from <a href="https://bintray.com/smartcat-labs/maven/download_file?file_path=io%2Fsmartcat%2Fberserker-runner%2F0.0.7%2Fberserker-runner-0.0.7.jar">here</a>. And I’ve created a configuration file named configuration.yml.</p><pre>load-generator-configuration:<br>  data-source-configuration-name: Ranger<br>  rate-generator-configuration-name: ConstantRateGenerator<br>  worker-configuration-name: Cassandra<br>  metrics-reporter-configuration-name: JMX<br>  thread-count: 10<br>  queue-capacity: 100000</pre><pre>data-source-configuration:<br>  values:<br>    genre: random([&#39;horror&#39;, &#39;comedy&#39;, &#39;action&#39;, &#39;sci-fi&#39;, &#39;drama&#39;, &#39;thriller&#39;])<br>    year: random(1980..2017)<br>    rating: random(float(5.5)..float(9.5))<br>    duration: random(85..150)<br>    title: random([&#39;Jurassic World&#39;, &#39;Toy Story&#39;, &#39;Deadpool&#39;, &#39;Gravity&#39;, &#39;The Matrix&#39;])<br>    director: random([&#39;Philippe Falardeau&#39;, &#39;Martin Scorsese&#39;, &#39;Steven Spielberg&#39;, &#39;Ridley Scott&#39;])<br>    insert: string(&quot;INSERT INTO movies_by_genre (genre, year, rating, duration, title, director) VALUES (&#39;{}&#39;, {}, {}, {}, &#39;{}&#39;, &#39;{}&#39;);&quot;, $genre, $year, $rating, $duration, $title, $director)<br>    deleteRow: string(&quot;DELETE FROM movies_by_genre WHERE genre = &#39;{}&#39; AND year = {} AND rating = {} and duration = {}&quot;, $genre, $year, $rating, $duration)<br>    deletePartition: string(&quot;DELETE FROM movies_by_genre WHERE genre = &#39;{}&#39; AND year = {}&quot;, $genre, $year)<br>    statement:<br>      consistencyLevel: ONE<br>      query: random([$insert, $deleteRow, $deletePartition])<br>  output: $statement</pre><pre>rate-generator-configuration:<br>  rate: 1000</pre><pre>worker-configuration:<br>  connection-points: 0.0.0.0:32779,0.0.0.0:32781<br>  keyspace: custom<br>  async: false<br>  bootstrap-commands:<br>    - &quot;CREATE KEYSPACE IF NOT EXISTS custom WITH replication = {&#39;class&#39;: &#39;SimpleStrategy&#39;, &#39;replication_factor&#39;: 2};&quot;<br>    - USE custom; - CREATE TABLE IF NOT EXISTS movies_by_genre (title text, genre text, year int, rating float, duration int, director text, country text, PRIMARY KEY ((genre, year), rating, duration)) WITH CLUSTERING ORDER BY (rating DESC, duration ASC);</pre><pre>metrics-reporter-configuration:<br>  domain: berserker<br>  filter:</pre><p>load-generator-configuration section is used to specify all other configurations. There, for every type of the configuration, name is specified in order for the Berserker to know which configuration parser to use in concrete sections. After that, a section for each configuration with parser specific options and format is found. There are following sections available:</p><ol><li>data-source-configuration where data source which will generate data for worker is specified</li><li>rate-generator-configuration where should be specified how rate generator will be created and it will generate rate. This rate is rate at which worker will execute</li><li>worker-configuration, configuration for worker</li><li>metrics-reporter-configuration, configuration for metrics reporting, currently only JMX and console reporting is supported</li></ol><p>In this case, the data-source-configuration section is actually a Ranger configuration format and can be found <a href="https://github.com/smartcat-labs/ranger/blob/dev/yaml-configuration.md">here</a>.</p><p>An important part for this article is the connection-points property within worker-configration. This will probably be different every time Docker compose creates a cluster. To see your connection points run:</p><pre>docker ps</pre><p>It should give you a similar output:</p><pre>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</pre><pre>f274b40c8dce wurstmeister/kafka:0.10.1.1 &quot;start-kafka.sh&quot; 3 hours ago Up 3 hours 0.0.0.0:32784-&gt;9092/tcp cluster_kafka_1</pre><pre>f56b8664e54b wurstmeister/kafka:0.10.1.1 &quot;start-kafka.sh&quot; 3 hours ago Up 3 hours 0.0.0.0:32783-&gt;9092/tcp cluster_kafka_2</pre><pre>81a32c580a12 wurstmeister/zookeeper:3.4.6 &quot;/bin/sh -c &#39;/usr/...&quot; 3 hours ago Up 3 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181-&gt;2181/tcp cluster_zookeeper_1</pre><pre>efdd29b63faa trigger-cassandra &quot;/docker-entrypoin...&quot; 3 hours ago Up 3 hours 7000-7001/tcp, 9160/tcp, 0.0.0.0:32782-&gt;7199/tcp, 0.0.0.0:32781-&gt;9042/tcp cluster_cassandra-seed_1</pre><pre>0fb1311ba3ee trigger-cassandra &quot;/docker-entrypoin...&quot; 3 hours ago Up 3 hours 7000-7001/tcp, 9160/tcp, 0.0.0.0:32780-&gt;7199/tcp, 0.0.0.0:32779-&gt;9042/tcp cluster_cassandra_1</pre><p>There you can find port mapping for cluster_cassandra-seed_1 and cluster_cassandra_1 containers and use it, in this case it is: 0.0.0.0:32779 and 0.0.0.0:32781.</p><p>Now that everything is settled, just run:</p><pre>java -jar berserker-runner-0.0.7.jar -c configuration.yml</pre><p>Berserker starts spamming the Cassandra cluster and in my terminal where kafka-console-consumer is running, I can see messages appearing, it seems everything is as expected, at least for now.</p><h3>End</h3><p>That’s all, next time I’ll talk about Cassandra CDC and maybe custom secondary index. Hopefully, in a few blog posts, I’ll have the whole idea tested and running.</p><p><em>Originally published at </em><a href="https://www.smartcat.io/blog/2017/cassandra-to-kafka-data-pipeline-part-1/"><em>www.smartcat.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1970752ef3d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartcat-io/cassandra-to-kafka-data-pipeline-part-1-d1970752ef3d">Cassandra to Kafka Data Pipeline Part 1</a> was originally published in <a href="https://medium.com/smartcat-io">SmartCat.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>