<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Davide Berdin on Medium]]></title>
        <description><![CDATA[Stories by Davide Berdin on Medium]]></description>
        <link>https://medium.com/@davideberdin?source=rss-b0432ef2c6c4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*c-LrqgIhJ3F9YwqA.</url>
            <title>Stories by Davide Berdin on Medium</title>
            <link>https://medium.com/@davideberdin?source=rss-b0432ef2c6c4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:34:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@davideberdin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part 2: Monitoring, Logging and Alarming]]></title>
            <link>https://medium.com/@davideberdin/part-2-monitoring-logging-and-alarming-5bc40cd5ac29?source=rss-b0432ef2c6c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bc40cd5ac29</guid>
            <category><![CDATA[elk-stack]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[grafana]]></category>
            <dc:creator><![CDATA[Davide Berdin]]></dc:creator>
            <pubDate>Sun, 08 Sep 2019 15:35:45 GMT</pubDate>
            <atom:updated>2019-09-08T15:35:45.971Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m writing a series on how to build a Data Platform from scratch and it’s time for Part 2! In <a href="https://dev.to/spaghettifunk/part-1-where-it-all-begins-1d72">Part 1</a> I explained how to start building your data platform. But, when your infrastructure grows, making sure that everything is working as expected becomes a challenge. And as one of my dearest colleague tells me all the time: “monitoring and logging is an art!”.</p><p>In Part 2, I want to tell you how to setup a production-grade monitoring system for your infrastructure. Concepts and caveats are more valuable than some “copy-paste” piece of code, for the simple reason that it is important to understand why certain choices are made.</p><h3>The tools</h3><p><a href="https://dev.to/spaghettifunk/part-1-where-it-all-begins-1d72">Previously</a>, I mentioned that we are using a bunch of tools. When I started listing all of them I thought: wow, so many things for something that looks relatively simple in concept. I mean, what we want to achieve is to <strong>be able to see logs and act if something happens</strong>. Simple right? Well, not quite. To refer back to my colleague: monitoring and logging is definitely an art :)</p><p>Let’s start by dividing the problem in sub-problems. In school they taught us that when the problem is too big, we have to divide it and conquer it (aka divide et impera if you are into latin).</p><h4>Problem 1: Logging</h4><p>Since we want to understand what it’s going on in our infrastructure and applications, we should start tackling logging at first. Generically speaking, an application writes logs on the <strong>STDOUT</strong>. Since we are using Kubernetes, that means we are able to read the logs from the logging console that comes with Kubernetes itself. Because we are able to read logs, it means we can collect them. How can we collect them? As I mentioned in the <a href="https://dev.to/spaghettifunk/part-1-where-it-all-begins-1d72">previous article</a> we use <a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a> for indexing the logs. In order to collect those we use <a href="https://github.com/fluent/fluent-bit/">Fluent-Bit</a>.</p><h4>Solution 1: Fluent Bit</h4><p>Fluent Bit allows collection of information from different sources, buffering and dispatching them to different outputs. The <a href="https://github.com/helm/charts/tree/master/stable/fluent-bit">helm chart</a> we use runs as a daemon set in Kubernetes ( <a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/">here</a> more info about daemon sets). This guarantees that there will be an instance of Fluent Bit per machine in the Kubernetes cluster. The process will collect the information of each pod in kubernetes from the standard output and redirect them towards another system. In our case we chose <strong>Kafka</strong>.</p><p>Now our logs are securely sent towards to a topic in Kafka and ready to be consumed by something else.</p><h4>Problem 2: System Metrics</h4><p>Applications write logs on the standard output, but machines don&#39;t write logs, right? So, how do I know how a machine is behaving? How do I know if the CPU is sky-rocketing or the I/O operations on the disk are the bottleneck or my application? Well, that&#39;s where <a href="https://github.com/prometheus/node_exporter">Node-Exporter</a> comes into play.</p><h4>Solution 2: Node Exporter</h4><p>Node Exporter is able to collect metrics from the underlying Operating System. This is powerful because now we are able to collect the system information we needed. Once again, there is a <a href="https://github.com/helm/charts/tree/master/stable/prometheus-node-exporter">helm chart</a> coming and rescuing us.</p><h4>Problem 3: Application Metrics</h4><p>Cool, but what if an application is able to give me more information than simple logging. For example, what if my database is able to give me the current system information such as memory consumption or average query latency. That&#39;s hard. These are not logs nor metrics coming from a machine. Although, they are available for us to be used. That&#39;s when <a href="https://prometheus.io/">Prometheus</a> enters the arena.</p><h4>Solution 3: Prometheus</h4><p>Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. <strong>BINGO</strong>. This sounds like the tool that will do a lot of things for us. But where does it stand in the big picture? Let’s take a look at the below image</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*UjZ1q2snA-nVHu-0" /></figure><p>I took this picture from <a href="https://sysdig.com/blog/kubernetes-monitoring-prometheus/">this</a> great article, and it describes clearly what Prometheus does. Basically, its role is to <strong>pull</strong> and <strong>push</strong> information. But what is relevant to know, is that Prometheus standardized the way information should be generated so that they can be parsed and, in a later stage, queried.</p><p>Because our data platform has multiple Kubernetes clusters (remember controlplane, production, development, etc), Prometheus needs to be installed in all of them. Thanks to the awesome community of developers, there is an <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator">helm chart</a> that we can use. This operator allows us also to use prometheus in <a href="https://prometheus.io/docs/prometheus/latest/federation/">federation</a> mode, which is <em>very important</em> in this context. The federation concept allows the prometheus in Controlplane to scrape the information from the other prometheus services so that we can centralize all the metrics in one unique point.</p><h4>Problem 4: Fetching the logs</h4><p>We decided to create controlplane for centralizing the information regarding the other environments and have an overview of what it&#39;s going on in our platform. Since we pushed our logs into Kafka, we now need to consume them and store them in a format that is readable to humans.</p><p>There is a famous acronym called ELK and it stands for Elasticsearch, Logstash, Kibana. So far we mention the <strong>E</strong> and the <strong>K</strong> but never the <strong>L</strong>. Well, that time just arrived.</p><h4>Solution 4: Logstash</h4><p><a href="https://www.elastic.co/products/logstash">Logstash</a> is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favourite “stash.” This is part of the Elastic suit, and it is a fundamental piece for making sure that we are able to have the same type of logging for everything that comes in.</p><p>Our input is the Kafka topic we mentioned before and our output is Elasticsearch where then the data will be indexed and “stashed”. The <a href="https://github.com/helm/charts/blob/master/stable/logstash">helm chart</a> helps you to install the application and by modifying this <a href="https://github.com/helm/charts/blob/master/stable/logstash/values.yaml#L286">part</a> of the values.yml you are able to easily read from Kafka. The major issue we found was in the @timestamp field. In fact, we had to adapt the values.yml a little to avoid having issues in reading the timestamp.</p><p>The following snippet of code will help you to solve such an issue</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d35f0fa1397ff19b052f36f0925f23a2/href">https://medium.com/media/d35f0fa1397ff19b052f36f0925f23a2/href</a></iframe><p>You have to modify the Timezone accordingly but that’s the major reason why we couldn’t have our data ingested in Elasticsearch correctly.</p><h4>Visualize All</h4><p>We just finished covering the logging part, but how do we visualize everything? There are two main applications: <a href="https://www.elastic.co/products/kibana">Kibana</a> and <a href="https://grafana.com/">Grafana</a>. We use Kibana to explore all the logs that coming in from all the application. Without Kibana, it would be extremely hard to debug your application because searching for what is going on, it’s very hard with kubectl logs -f &lt;pod-name&gt; | grep whatever-error command :)</p><p>Grafana helps in visualizing all the metrics coming in from Prometheus. There are a ton of <a href="https://grafana.com/grafana/dashboards">pre-made dashboards</a> that you can just install and use them. The only thing you need to do is to setup the prometheus installed in controlplane as your data-source in Grafana and that&#39;s it. All the metrics will automagically be available for you.</p><h4>Problem 5: Alarming</h4><p>This is the toughest part of the process. Once again, the concept is simple but deciding the thresholds onto when receiving an alarm is difficult, and it needs to be tuned along the way. I would recommend to start from this <a href="https://awesome-prometheus-alerts.grep.to/rules.html"><strong>awesome</strong></a> website and start building the rules that are important for your data platform.</p><h4>Solution 5: Slack and PagerDuty</h4><p>I can’t help you with building the rules that you need for your data platform, but I can give you advice on selecting the right tool to notify you and your team when an alarm is triggered. My suggestion is to send the notification to <a href="https://prometheus.io/docs/alerting/configuration/#slack_config">Slack</a> for the alarms that you consider “minor”. I leave the definition of minor up to you. We only send a slack notification for the development environment and for those applications that are not public in production yet.</p><p>For production systems we use <a href="https://www.pagerduty.com/">PagerDuty</a> to create a rotation-calendar for taking care of the systems among team members and to make sure that everything is always up-and-running. There is a great <a href="https://www.pagerduty.com/docs/guides/prometheus-integration-guide/">integration</a> with Prometheus that I highly recommend to setup.</p><p>Grafana also helps with <a href="https://grafana.com/docs/alerting/rules/">alerting</a> but we haven’t used it yet. It looks awesome though. If you’ve been using Grafana, it would be great if you can share your experience with me in the comments below :)</p><h3>Conclusion</h3><p>In this long blog-post I gave you an overview the tools that my team and I are using for our data platform. I hope this gave you more ideas on how to start! You will encounter problems along the way because “Rome wasn’t built in a day”. I do hope that you have all the initial information to collect, visualize and receive alarms in your data platform. But remember my colleague’s motto: logging and monitoring is an art :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bc40cd5ac29" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1: How to build a data platform from scratch]]></title>
            <link>https://medium.com/@davideberdin/part-1-how-to-build-a-data-platform-from-scratch-e321c7d8a2ac?source=rss-b0432ef2c6c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e321c7d8a2ac</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Davide Berdin]]></dc:creator>
            <pubDate>Sat, 24 Aug 2019 21:07:08 GMT</pubDate>
            <atom:updated>2019-08-24T21:07:08.115Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever tried to build a data platform from scratch? Do you know where to start, what to look for? And most importantly: what <em>not</em> to do? Fear no more! Learn how to build the perfect data platform with these series of articles :)</p><p>This article will be the first part of a series, that will give you an inside in my journey towards building a data platform from scratch. By sharing my experience, the lessons learned while developing it, the advantages and disadvantages of certain design choices, and my goals, I hope to make your journey less bumpy.</p><p>Wether you are given the task to build a data platform, or you decide to do this in your own time: you aim high with your ambitions — or at least that’s what I do. Having ambition is great and it is necessary in order to push yourself a little bit further. That’s why, when you start designing something it’s important that you look further than the initial <em>requirements</em>.</p><p>Without further ado, let’s start this journey by discussing the motivations I had behind creating a Data Platform for my current job.</p><h3>Motivations (aka Requirements)</h3><p>I work in the Media industry, and the company I work for doesn’t have a Tech department like Netflix, GitHub or Twitter. We might not be considered as a ‘tech company’, we have a team of great engineers that are as motivated as me to create something nice.</p><p>One day, our manager came in and said that our company decided to build a Data Platform from scratch. The requirements we’ve got were the following:</p><ul><li>It needs to be scalable</li><li>It needs to ingest a lot of data</li><li>It needs to be able to query very quickly</li></ul><p>That’s “all”!</p><p>The engineering team had a discussion on what type of technologies we wanted to use, and what the architecture should look like. We decided to focus a lot on <strong>Kubernetes</strong>. But the choice wasn’t straightforward. A large part of the team had never worked with Kubernetes, they had more knowledge on AWS Lambda and API Gateway. While to me Kubernetes was a natural choice, for other people it was a bit scary.</p><p>In the end we commonly agreed that it would serve our needs better to use Kubernetes, especially in terms of scalability and deployment of our Dockerized application. Using Kubernetes meant that we had to migrate tons of services from ECS and other EC2 instances, to Kubernetes.</p><h3>Initial architecture</h3><p>After the choice for Kubernetes, we created a bunch of bash files and <strong>Terraform</strong> scripts to get started. Within the engineering team, there was a discussion on how to tackle the automation and we decided to build a “1-click-deployment” system.</p><p>This “1-click-deployment” system, consisted of a Docker image that contained all the packages we needed in order to generate the entire infrastructure of the Data Platform. After that we created the initial skeleton based on this awesome <a href="https://blog.gruntwork.io/how-to-create-reusable-infrastructure-with-terraform-modules-25526d65f73d">article</a> of Yevgeniy Brikman from Gruntwork. Our result looked similar to this <a href="https://github.com/spaghettifunk/cluster-example">repository</a>.</p><p><strong>Tip:</strong> When there are a lot of “moving” parts within your infrastructure, it’s a good practice to automate it and to rely on tools that will help you and your team to put all the pieces together with minimum effort. The picture below shows how the initial architecture looked like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/0*t82GaV2WD4krExzg" /></figure><p>This was quite simple to us; we needed to <a href="https://github.com/terraform-aws-modules/terraform-aws-eks">create a Kubernetes cluster</a>, add a <a href="https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md">Load Balancer via NGINX</a>. We were ready to move to the next step.</p><h3>Drama was about to come</h3><p>We only used one cluster, and we immediately had to start thinking about how we were going to deploy applications. Then other questions popped up; how do we create a <strong>Staging</strong> environment, or a <strong>Testing</strong> environment? The initial choice was to scale the cluster with few more machines and leverage the <strong>Kubernetes Namespaces</strong>.</p><p>The advantage of using Namespaces, is that you can &quot;isolate&quot; your applications in a sort of box. Isolation is what we were trying to achieve, so that we could distinguish the environments. Doing this, created an issue with naming and with how we were going to deploy the applications. For example, let&#39;s assume the following: we have two APIs and one WebApp. All three need to have a Testing and Staging environment, so that developers can safely deploy their code before going to production. Because we decided to use Namespaces, we tried to first create three Namespaces: Production, Staging, Testing. There are some drawbacks in putting all the application under the same namespace. The first issue is that it would be much easier to delete everything. For example, the Kubernetes CLI ( kubectl) is able to delete namespaces. If a developer would issue the follwing command kubectl delete ns &lt;env&gt; all the applications under that namespace would be gone. The second problem is in the isolation. In fact, the reason why namespacing was created in the first place, was to partition the cluster in smaller virtual cluster.</p><p>We tried another approach: we create a namespace per application per environment. For example we would have the following:</p><ul><li>production_api1</li><li>staging_api1</li><li>testing_api1</li><li>production_webapp</li><li>staging_webapp</li><li>testing_webapp</li><li>and so on…</li></ul><p>You can clearly see that this would pollute the Kubernetes cluster with tons of namespaces. The following <a href="https://kubernetes.io/blog/2016/08/kubernetes-namespaces-use-cases-insights/">article</a> gave us a lot of insights on how to use namespace in better way. Despite the article states that you can use both of the approaches I described above, it also highlights the anti-patterns when using such approaches. For us, those solution didn’t work well.</p><p>What did we do? The solution was simple: multiple Kubernetes clusters.</p><h3>Multiple clusters</h3><p>As mentioned above, we initially decided to adopt the “1-click-deployment” strategy: which means that our code-base was ready for deploying as many clusters we wanted. And so we did. The second Architecture we pulled-off looks like the picture below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xKXK4_H_FaGJQ1Vm" /></figure><p>As you can see from the picture, we have three clusters. In this way we could deploy applications depending on the environment we wanted to target.</p><p>The whole team was very hyped and we thought that we nailed it. But, with great powers come great responsibilities. Responsibilities here means <strong>logging</strong> and <strong>monitoring</strong>. With three clusters, understanding what it’s going on is rather challenging. We yet again had a problem we needed to tackle.</p><p>In a distributed environment with multiple applications, understanding what is happening is crucial. In fact, being able to quickly debug your application in Development or understand what caused an eventual error in Production is the fundamental step for keeping control over your systems.</p><p>But logging is not sufficient. Machines themselves in general don’t write logs, the expose metrics of their hardware. Thus, collecting such information and visualize them, it will allow you and your team to set up <strong>alarms</strong> based on certain rules. A rule can be “if the average CPU usage is above 80% trigger an alarm”. The alarm itself can be an email or a Slack message to your team. Alarms will prevent having your machines to reach undesirable states.</p><p>When you have a lot of applications, different environments, and a team of engineers and data scientists that is eager to create, test and put applications or models in production, you need to have a solid monitoring and logging system. But how do we do so? Yet again the answer was simple: another Kubernetes cluster :)</p><h3>Controlplane at rescue</h3><p>Controlplane was a fancy name we took from another team within the company. The sole purpose of this cluster is to collect metrics and logs from the other clusters and centralize the visualization of such information. At this point the architecture looks like the following</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*w1nJOYOluOhB3k7e" /></figure><p>On each cluster we set up <a href="https://prometheus.io/"><strong>Prometheus</strong></a>, <a href="https://github.com/prometheus/node_exporter"><strong>Node-exporter</strong></a>, <a href="https://github.com/kubernetes/kube-state-metrics"><strong>Kube-state-metrics</strong></a> for exposing all the metrics of the cluster. To collect the information and send them out the cluster, we used <a href="https://github.com/fluent/fluent-bit"><strong>Fluent-Bit</strong></a>. All the metrics and logs were redirected towards a <strong>Kafka Topic</strong>. In this way we were able to fetch them from the controlplane.</p><p>In controlplane we installed <a href="https://www.elastic.co/products/logstash"><strong>Logstash</strong></a> and connected it to the <strong>Kafka Topic</strong> mentioned above so it could start fetching the logs. Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to an output.</p><p>Now that Logstash has access to the logs we just need to redirect them to the <a href="https://github.com/elastic/elasticsearch"><strong>Elasticsearch</strong></a> service and visualize them with <a href="https://github.com/elastic/kibana"><strong>Kibana</strong></a>. For the metrics we put in place <a href="https://grafana.com/"><strong>Grafana</strong></a> and created a bunch of Dashboards for visualizing the status of each system.</p><p>With the controlplane in place we had full visualization and control of every single aspect of each individual cluster. The controlplane was a win-win solution.</p><h3>Conclusions</h3><p>In this first part I explained how we started designing the Data Platform. This article focused mostly on the underlying infrastructure and the problems that we had to overcome.</p><p>The lesson I learned during this part of the development: is to think ahead much more than when I design software. Software can be changed more rapidly than infrastructure. When designing an infrastructure there are more variables to take into account, and the choices that you make can have a big impact on the final result. Long story short: think first and take your time. Rushing to get things out quickly will not always help you.</p><p>In the next article I am going to talk about how applications are deployed and how the team interacts with this architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e321c7d8a2ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Druid Terraform]]></title>
            <link>https://medium.com/@davideberdin/apache-druid-terraform-6dee52a6c46f?source=rss-b0432ef2c6c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dee52a6c46f</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[druid]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Davide Berdin]]></dc:creator>
            <pubDate>Sat, 17 Aug 2019 18:54:04 GMT</pubDate>
            <atom:updated>2019-08-17T19:13:24.818Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to run Apache Druid in Kubernetes using Terraform</h3><p>In this article I would like to show how I created a Terraform module where I could deploy <a href="https://druid.apache.org/">Apache Druid</a> on Kubernetes. This is not production-ready but tests with Real-time ingestion have been quite successful. Hopefully, my experience will be helpful to you!</p><p><strong>First things first: what is Apache Druid?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h9cBMCb1QtVU5VsYx_hRBg.png" /></figure><p>Apache Druid is a high performance real-time analytics database. Originally created at <a href="https://metamarkets.com/">Metamarkets</a> and later donated to the Apache Foundation, Druid has been designed to be fault-tolerant, blazing fast, horizontally scalable and many other things. It’s a big and complex project that requires some time in order to master it properly.</p><p><strong>Prepare for scaling!</strong></p><p>One of the biggest challenges I faced when working with Druid, was to deploy the system in such a way that I could easy to scale. I started by creating a fleet of machines in a VPC of AWS, then on each instance I started the service that was more suitable for that type of node. For example, for the <a href="https://druid.apache.org/docs/latest/tutorials/cluster.html#query-server">Broker</a> I selected a <em>generic-purpose </em>type, whereas for the <a href="https://druid.apache.org/docs/latest/tutorials/cluster.html#data-server">Historical</a> I used a <em>storage-optimized</em> machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1THsy_Ablf9rzUU9NotAA.png" /><figcaption>Druid services</figcaption></figure><p>Once all the machines and services where up and running, I had to make sure that all the components where able to <em>connect</em> to each other and could exchange information. I did this by checking the logs of each service. If they would not be connecting, it could potentially lead to some Networking mistakes or inappropriate configuration. To plum and automatize everything together, I chose <a href="https://www.terraform.io/">Terraform</a> in combination with <a href="https://www.packer.io/">Packer</a> and <a href="https://www.vagrantup.com/">Vagrant</a>. After days of trials (and error!), I managed to get the Druid cluster up and running. However, the automatic scaling was still an issue.</p><h3>Kubernetes at Rescue</h3><p>One thing I like about Kubernetes, is the ability of horizontally scaling pods when the load is above a certain threshold. The cluster I’m using, has the ability of autoscaling the machines if there are no more resources available, hence it makes Kubernetes a good candidate for deploying Druid.</p><p><strong>Cluster configuration</strong></p><p>Once again, I use Terraform to deploy the infrastructure. Since Terraform has a Kubernetes provider available, I decided to setup some of the services via Terraform as well, and Druid is one of them. All the other applications, such as APIs, Web-apps, etc. are initially deployed manually and then updated via the CI/CD.</p><p>I deployed an <a href="https://aws.amazon.com/eks/">Amazon EKS</a> cluster with different <strong>node-groups</strong> so that I could deploy Druid only on particular nodes. As mentioned above, I needed different type of machines depending on the service. The <em>autoscaler</em> application is able to add or remove the machines based on the load on the cluster. An example on how to create such a cluster can be found <a href="https://github.com/spaghettifunk/cluster-example">here</a>.</p><p>I started the cluster with 8 machines, divided in 2 different node-groups. In this way I could deploy Druid, Zookeeper and Postgres in different nodes and have it “<em>fault-tolerant</em>” (there is an <a href="https://github.com/spaghettifunk/druid-terraform/issues/8">issue</a> open for the pod-affinity).</p><h3>Time to put this in action!</h3><p>The module I created, can be imported by letting Terraform downloading it via <strong>git</strong>. The module assumes that Terraform is able to deploy on Kubernetes directly. Here is an example</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df8ca0b352cbf211f02c36551b4db6f9/href">https://medium.com/media/df8ca0b352cbf211f02c36551b4db6f9/href</a></iframe><p>After importing, you can runterraform plan and (hopefully!) terraform apply on your cluster.</p><h3>Conclusion</h3><p>The fact that Druid is now set up in Kubernetes, will allow you to automatically scale the system without manual intervention. The module I created, needs some help to make it better and <strong>production-ready,</strong> but I think it’s a good start if you are planning to use Druid.</p><blockquote>Contributions to the <a href="https://github.com/spaghettifunk/druid-terraform">project</a> are very welcomed 🎉 Feel free to open up a <strong>PR</strong> or an <strong>Issue</strong>.</blockquote><p>There are a couple of caveats that you need to keep in mind. For example, if you set the <a href="https://druid.apache.org/docs/latest/dependencies/deep-storage.html">Deep Storage</a> without the usage of S3 or HDFS, you may have issues in keeping the data (I do not recommend to have PVC for this purpose).</p><p>Scaling up the <a href="https://druid.apache.org/docs/latest/design/historical.html">Historical</a> nodes in general is not a problem, but the scale down instead could potentially lead to either loosing information or forcing Druid to re-index the data ingested. Make sure that you have good scaling rules for this service.</p><p>If you have any questions related to this topic, I’m happy to help! You can find me on <a href="https://twitter.com/davideberdin">Twitter</a> and <a href="https://github.com/spaghettifunk">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dee52a6c46f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yieldr — Hackday/0x01]]></title>
            <link>https://medium.com/@davideberdin/yieldr-hackday-0x01-be29237ee5a6?source=rss-b0432ef2c6c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/be29237ee5a6</guid>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[company-culture]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Davide Berdin]]></dc:creator>
            <pubDate>Wed, 04 Apr 2018 14:38:09 GMT</pubDate>
            <atom:updated>2019-08-17T14:29:22.913Z</atom:updated>
            <content:encoded><![CDATA[<h3>Yieldr — Hackday/0x01</h3><p>At Yieldr, we value innovation and foster an environment where new ideas are not only welcomed but celebrated. Around here, we tell everyone, <em>Don’t Be a Monkey.</em></p><blockquote><em>Instead of following the status quo and doing things a certain way because they’ve always been done that way, we challenge our habits and thinking to adapt and evolve to our ever-changing industry.</em></blockquote><p>With this being ingrained in our culture, it was natural for us — and long overdue — to hold our first Hackday. We loved the idea so much we actually took two days for the engineering team to suggest, work on and finally demonstrate their unique ideas to the entire company.</p><p>For the event we looked for an external venue. We wanted to provide the inspiration that such a project deserves and make sure it didn’t feel like just another day at the office.</p><p>We threw ideas on a Google doc and shared it with the team. Soon enough we had about 20–30 ideas which we could potentially work on during the event. It was incredibly satisfying to see the engagement from the team while sharing their ideas! We went through these ideas, refined them a little, and finally, grouped them into themes so we could narrow down the scope. We wanted it to be broad enough for everyone to participate and at the same time narrow enough so that it can be completed within the timeframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*k9EmMjGCnysrkCppBq6fxA.jpeg" /></figure><p>On a Thursday afternoon, everybody from the engineering team was talking about the ideas that were being narrowed down a few days before the event. Groups of people were naturally forming according to interests, motivations and a willingness to explore new technologies. You could really feel the energy and anticipation building!</p><p>The ideas were flowing like water from Niagara Falls. What started out as a few words quickly became a cascade of themes and concepts.</p><p>Once we narrowed down our focus, the teams started to investigate how to tackle the problems. We finally had everything ready: the data was ready to be used, the location was in place, food was arranged, t-shirts were printed and delivered — every piece of the puzzle was falling into the right place for a kick-ass two days of fun and innovation.</p><p><strong>EXECUTION</strong></p><iframe src="https://www.instagram.com/p/BXzlMzyl2HS/embed/?cr=1&amp;amp;rd=https%3A%2F%2Fwww.instagram.com" width="658" height="882" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf0dd0d175c1fc1d7218abee3ee5b9c2/href">https://medium.com/media/cf0dd0d175c1fc1d7218abee3ee5b9c2/href</a></iframe><p>It was finally Christmas morning for us! For the first day of the event, we arrived at an inspiring location in the center of Amsterdam. After a few shots of ginger and lemon juice and with the right music on, the adrenaline started kicking; it was go time!</p><p>Everybody was completely locked in and entrenched in what they were doing, especially when ambiguous ideas started to take tangible form.</p><p><strong><em>“Niiceee!!”</em></strong></p><p><strong><em>“Awesome, man!”</em></strong></p><p><strong><em>“YES it works!!!”</em></strong></p><p>These were only a few of the words spoken that personified the vibe of our Hackday. The atmosphere that had been created was exactly what all of us wanted.</p><p>Time was passing by, the code was assuming some sort of shape, people were getting more and more excited and, most of all, the fun was palpable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TX0vyu0crY5PjA2u6VN16w.jpeg" /></figure><p>At some point, people started walking around and checking out what other teams were producing with just a few hours of coding. The incredible fact that everyone was sharing ideas with the others was already a clear sign of success.</p><p>Yes, creating something and presenting it to everyone is indeed the point of a hackday, but the most successful part was the engagement that every person had with each other. Sharing ideas, debating on technologies and helping others with their problems were the best part of this event. The hackday took our already close-knit team and strengthened our bond even further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B1_Gtq_2doeCQzTWDFzz3A.jpeg" /></figure><p>At the end of the second day, all the teams had to prepare a presentation in which they showed off their shiny new ideas. We had a simple rule: 20 minutes at most to present and have a Q&amp;A. Funny fact about this rule: people were so enthusiastic about all the projects that each team were presenting for 10 minutes and having 25 minutes of questions. Hell of a good sign, isn’t it?</p><p>The whole company got involved and us — as engineers — seeing that what we produced was creating so much interest was definitely the best reward we could ever expect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VARymGHkueMXRUpZiw_laQ.jpeg" /></figure><p>The projects that we produced were focused on Machine Learning and Data Visualization. From Real-Time maps to Sentiment Analysis.</p><p>Here are the projects briefly explained:</p><ul><li><strong>Problesis</strong>. A demand forecasting algorithm in which the team combined a novel-approach for estimating the demand by combining different sources</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rdW6ZDArzcKkkHgE2kQT_g.jpeg" /></figure><ul><li><strong>Real-Time</strong>: A real time map for ticket conversions</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*EtBoUvugp9rNlWXdL6_WoQ.png" /></figure><ul><li><strong>Sentiment Analysis</strong>: A sentimental analysis on social media data to measure/forecast the influence on an airline at the micro/macro level.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*pXUEjaYjEr6cB-1S0CD3Gg.jpeg" /></figure><ul><li><strong>Traveler Lifetime Value: </strong>A tool that helps to identify specific consumers based on their value.</li><li><strong>Mobile App</strong>: A simplified version called DIfA. Demand intelligence for Android.</li></ul><p><strong>There was no winning team because the whole engineering team won!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be29237ee5a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>