<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by 7Mind GmbH {Dev} on Medium]]></title>
        <description><![CDATA[Stories by 7Mind GmbH {Dev} on Medium]]></description>
        <link>https://medium.com/@7mind_dev?source=rss-4eb237f60335------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2Vdl2CRWRrMdspH8TJjg2w.png</url>
            <title>Stories by 7Mind GmbH {Dev} on Medium</title>
            <link>https://medium.com/@7mind_dev?source=rss-4eb237f60335------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:24:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@7mind_dev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Problems With Helm]]></title>
            <link>https://medium.com/@7mind_dev/the-problems-with-helm-72a48c50cb45?source=rss-4eb237f60335------2</link>
            <guid isPermaLink="false">https://medium.com/p/72a48c50cb45</guid>
            <category><![CDATA[7mind]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[helm]]></category>
            <dc:creator><![CDATA[7Mind GmbH {Dev}]]></dc:creator>
            <pubDate>Tue, 11 Sep 2018 21:01:11 GMT</pubDate>
            <atom:updated>2018-09-11T21:01:11.076Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/652/0*uYh6iLNVE8AZ1IDC" /></figure><p>7Mind is Germany’s most popular mindfulness app, and helps easily integrate the practice of meditation into daily life. We believe in the great potential of the human mind, and it is our mission to show individuals and organisations alike how to unlock it in order to live and work feeling more productive, relaxed and happy. 7Mind is also available in French and Dutch, and we will soon expanding overseas to the English markets.</p><p>Managing Kubernetes deployments has always been tricky. Just as Kubernetes has allowed much more powerful docker deployments, so too has it increased their complexity. Deployments very quickly become a mess of yaml files mixed together in what rapdily becomes an uninteligable lump of dependencies, configuration variables and general confusion. Helm alleviates this headache somewhat by bundling deployments together so that you no longer have to decipher which of the hundreds of files need to be changed and instead can focus on one application at a time. Furthermore you can dispense with the chore of writing your own charts for completed deployments, such as Postgres or Cassandra, and can instead rely on a massive open source repository. It sounds great in principle, but as we all know life rarely adheres to principles.</p><h3>Knowing Your Code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vRg14mFBtQ6-4ppP4362eA.png" /><figcaption>xkcd.com/2044</figcaption></figure><p>One of the immediate pitfalls that you run into when you start using helm is that you no longer know what exactly is deployed on your cluster. When you run helm install repo/chart you give up almost all configuration control to whoever wrote the original chart. Now that’s not necessarily a bad thing. Odds are they wrote a good chart, and if not there is a whole internet out there who can tell them how to fix it. What they wrote was in all likelyhood better than you could have done yourself had you attempted it. So what’s the problem?</p><p>Let’s say that after a while, you run into an error (this will invariably happen if you are deploying anything to production, Mr. Murphy is not one to be ignored). Perhaps you went to inspect a postgres/patroni deployment and realized that something had gone terribly wrong. Brief inspection of the deployment will tell you that it failed to connect to etcd. Of course, not having written the chart you might have been flabbergasted to learn that there was even an etcd dependency, or what version it was, where it was installed from and what it was doing. Further googling would reveal that this a systematic problem with the etcd deployment the chart comes equipped with, but that that etcd deployment can’t actually be overwritten.</p><p>Sure it’s inconvenient, but this is still much easier than writing the chart yourself. So you deploy a more stable etcd chart, because of course you can’t switch to another key value store — the original chart didn’t come with a configuration option for that. No matter, you now have two charts deployed instead of one and everything’s fine. Well almost, it seems that you can’t actually disable the default etcd stateful set which is deployed, so chuck in a kubectl delete statefulset origional-statefulsetas part of your deployment process, it’s still not as bad as it used to be.</p><p>So you’re cruising along in a boat that has a few holes in it, but you’re most definitely still afloat. Then one day, someone comes along and tells you they want a newer version of postgres, shouldn’t be a problem right? Well it wouldn’t be, except of course you’re still not quite sure what you’re deploying, so you take a look. Hmm, it seems the new version of the chart also upgraded the version of postgress. Well you’d hope the chart author would have included some contingencies in the case of migration, but then again, you have no guarantee that they did, after all, it’s not your code.</p><p>It turns out the chart doesn’t tell you anything about how persistent volumes are handled, so there’s no guarantee that updating helm will not also update the deployment with a new persistent volume which doesn’t contain your data. Even if you’re lucky and the author of the chart didn’t change the nature of the persistent value claim for the chart, you still don’t know if the new version of postgres will read or even preserve your old data. Again, the chart makes no guarantees and you don’t own the image that it’s running anyway. So you hold your breath and hope it works, because truth be told you don’t have any other option.</p><p>This is a common occurrence with helm: it works, it works, and then it breaks and what would be a minor error in a project you wrote yourself becomes a massive headache of unmaintainable nonsense. I can personally attest to dozens of hours spent debugging weird errors in someone else’s helm chart — time in which I could almost certainly have built something more reliable and suited for my own needs several times over.</p><h3>When Helm Goes on Vacation</h3><p>Now there are of course issues with helm charts as outlined above, but perhaps more serious are the issues with the tool itself. Helm does not necessarily do what you want it to. Case and point, today I ran helm delete postgres --purge which exited successfully, removed the postgres deployment from tiller but <strong>not</strong> from kubernetes. So then I was left with the task of deleting stateful sets, services and persistent volume claims which helm had somehow forgotten about.</p><p>Any tool can fail, but the problem with helm is that it often fails <em>quietly</em>. We get neither a bang nor a whimper. Running helm update doesn’t necessarily restart your pods as you might assume it does, but it strikes me that it only does this if you change their configuration, like the image tag for instance, and even then it’s dicey. Forgot to include helm metadata on one of you templates? No problem, unless of course you try to access it later, then you get a rather large problem and a very unclear error.</p><p>My point here, is that helm comes without guarantees, and using it in a production environment, especially one in which you are autonomously deploying charts can be dicey. Occasionally it will fail for some unknown reason, and even more frighteningly <strong>it won’t tell you it failed! </strong>This leaves you in the unfortunate situtation of having to diagnose errors that have nothing to do with anything in your codebase. Imaigne if everytime you deployed an application, you had no guarantee your dependencies would work as intended.</p><h3>Alternatives</h3><p>So, what do you do? Well, chances are, you live with it. As a developer you can hope that the helm tool improves itself, you can make sure that your charts are solid, etc… The truth of the matter is you are likely going to continue to run into these problems, but maybe it’s rare enough that its worth the hardship. However, for those of us who are stubborn enough not to accept software with compromises, let me propose a few improvements.</p><h3>1. Own Your Deployments</h3><p>Don’t rely on someone else to maintain your charts for you. Do it yourself! The same goes for images. Sure, copy someone else’s code for the baseline, but then make it your own! Store <strong>your</strong> charts on <strong>your </strong>repository so that you can change them if you need to, and more importantly you know what you are deploying.</p><p>This may sound like more work but its just a few minutes of pulling open source code into your own repo. You can update your charts that way too but this way you don’t get any unexpected surprises when you upgrade. Maintaining your own repository forces you to know what is going on, and allows a newcomer to understand the history and current state of the project without hunting for hundreds of different links. Furthermore, most helm charts are filled with needless configuration bloat, on the off chance that some user has some specific need. While this is undoutably useful for open sourced software, it leads to much more complicated configurations than are necessary for almost any scenario, making debugging much more complex than it needs to be. Maintaining your own chart allows you to strip away this bloat and get a more readable and maintainable deployment.</p><h3>2. Deploy With Precision</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i_Xyq2MLUPGlRkv5YBlqHw.png" /></figure><p>This is a more intense requirement, one I haven’t attempted yet for it involves rebuilding the helm deployment tool. The idea is to make a very concise desired state specification. Consider your deployments as a git repository. When you change something locally and push it to the server, it pushes only the diffs. The server looks at those, interprets what has changed and appropriately creates, deletes and updates kubernetes deployments. This is actually relatively simple to do (I’m not joking when I say that you can just rip off the git diff process). Instead of looking for deployments by metadata, look at what was deployed by version control. It’s just as easy as integrating the diff tool with the kubernetes api.</p><p>The key point here, is you would have total (and I mean total) control of your entire deployment pipeline. If you want a slack hook on deployment, go for it. Want to send each update for code review, you can do it, a commit log of the state of your cluster — it’s built in! The advantages to this approach seem almost unlimited, and though I’m sure there would be barriers to overcome, I believe its worth doing so for the sake of a better deployment system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*BMcuVFPqzNRpkTzD_GZoYQ.png" /></figure><p>The point I’m trying to make here is not that helm is bad software. It’s really not, its woks well a lot of the time, and fails occasionally. For most people its good enough to write solid maintianble deployments with only a few hiccups. However, if you work in an environment where there is rapid development and deployments are changed frequently, using software that does 90% of what you want it to but lets you down 10% of the time is not a sustainable option — especially for mission critical components. In situations like that, don’t reinvent the wheel, but if your cart isn’t rolling maybe take a look at some improvements.</p><p>I’m Sequoia Snow, a junior backend developer at 7mind. I mainly focus on building our deployment infastructure and convincing everyone else in the company that we should use Haskell. Currently 7mind is in the internationalization phase expanding rapidly across multiple countries, while simultaniously migrating from a monolithic rails application to a more scalable microservice-based architecture. This has made solidifying our deployment process more important than ever, seeing as we often update production services on a weekly or even daily basis.</p><p>We are always looking for innovative and talented people at 7Mind. If you feel that you would be a good fit, drop us a line at <a href="/service/mailto:jobs@7mind.de">jobs@7mind.de</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72a48c50cb45" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KaufmannEx — Elixir Microservices with Kafka and Avro]]></title>
            <link>https://medium.com/@7mind_dev/kaufmann-ex-317415c27978?source=rss-4eb237f60335------2</link>
            <guid isPermaLink="false">https://medium.com/p/317415c27978</guid>
            <category><![CDATA[7mind]]></category>
            <category><![CDATA[elixir]]></category>
            <category><![CDATA[avro]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <dc:creator><![CDATA[7Mind GmbH {Dev}]]></dc:creator>
            <pubDate>Tue, 03 Jul 2018 13:59:52 GMT</pubDate>
            <atom:updated>2018-07-03T15:01:50.746Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://github.com/sevenmind/kaufmann_ex">KaufmannEx</a> is a library for building highly parallel, efficient, Kafka backed services in Elixir. This article describes KaufmannEx, the motivations for its design, and details of basic service implementation with KaufmannEx.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7m5qzs1vCP0ImK-NtuET5w.png" /><figcaption><a href="https://www.jpl.nasa.gov/spaceimages/details.php?id=PIA18155">https://www.jpl.nasa.gov/spaceimages/details.php?id=PIA18155</a></figcaption></figure><p>I’m Grant McLendon, principal back-end developer at 7Mind. <a href="https://7mind.de/">7Mind</a> makes an excellent app for guided meditation. I’ve spent the past few years working with Rails monoliths and Ruby microservices.</p><p>At 7Mind, we’re in the process of migrating from a Ruby on Rails monolith to a microservice oriented architecture. Rails has been fantastic for rapid iteration and expanding business needs. As we continue to grow, our goals are scalable, cost-effective, fault tolerant, and asynchronous services. In the future, we want to support heterogeneous services in multiple languages.</p><p><strong>The Tools</strong></p><p>After evaluating multiple languages and messaging systems, we chose to use Elixir as our primary language, and Kafka as a messaging bus.</p><p><a href="https://elixir-lang.org/">Elixir</a> is a fantastic language (we’ll probably write another article gushing about it) well suited to building microservices. Elixir provides outstanding performance, stability, and <a href="http://learnyousomeerlang.com/what-is-otp">BEAM + OTP</a> are a foundation we can build services on with confidence. We could build our services solely within a single Elixir and OTP application, however we want to support development in other languages, and potential future changes in architecture.</p><p><a href="https://kafka.apache.org/">Apache Kafka</a> is many things. We are using Kafka as a distributed, scalable, high-throughput messaging bus and broker. Language support for Kafka is <a href="https://cwiki.apache.org/confluence/display/KAFKA/Clients">excellent</a> and allows us a lot of freedom in our future development. Kafka also provides <a href="https://kafka.apache.org/uses">lots of functionality</a> we don’t need now, but have an eye on for the future.</p><p><strong>Message Schemas</strong></p><p>One trouble spot we’ve encountered in micro-services is guaranteeing standard message schemas across different services. It’s reasonably common for a developer to make a small change to a message that unexpectedly breaks other services.</p><p>To avoid this issue outright, we’ve chosen to use <a href="http://avro.apache.org/docs/current/">Apache Avro</a>. Avro is a binary data format coupled with built-in schema validation. Avro allows us to encode messages with a given schema and guarantee any application that receives an encoded message will decode it with the correct schema.</p><p>We also want to guarantee all applications have access to the latest correct schemas. We found the <a href="https://docs.confluent.io/current/schema-registry/docs/index.html">Confluent Schema Registry</a> perfectly suited to this task. Schema Registry provides an HTTP interface for retrieving and managing schemas, as well as validating schemas for forwards or backwards compatibly.</p><p>In a KaufmannEx project message schemas are defined either in app/priv/schemas or another directory configured in the Application environment. KaufmannEx provides a <a href="https://github.com/sevenmind/kaufmann_ex/blob/master/lib/release_tasks/migrate_schemas.ex">schema migration script</a> to populate the schema registry in production deployments. I encourage you to review this script before use. It may not suit all use cases and has an expectation of a metadata schema you may not wish to use.</p><p>For more information about Avro Schemas, check out the <a href="https://avro.apache.org/docs/1.8.2/spec.html">Avro Spec Documentation</a>.</p><p><strong>KaufmannEx</strong></p><p>Implementing one Kafka-backed microservice is pretty straightforward, implementing 10 involves a painful amount of code duplication and undifferentiated busy work. Furthermore, consuming messages from Kafka can be akin to sipping from a firehose. We wanted to restrict message consumption to available consumer capacity, using Kafka Consumer Groups to distribute messages across multiple consumers according to capacity. This lead rather directly to implementing a library to handle the boring parts of writing a new service in a consistent, parallel manner. To control consumption rates, we have included a back-pressure mechanism inspired by <a href="https://www.youtube.com/watch?v=6ijgMvXJyuo">Mathew Gardner’s Elixir with Kafka talk</a>.</p><p><a href="https://github.com/sevenmind/kaufmann_ex">KaufmannEx</a> is our library for implementing simple, parallel, Kafka-backed services. We’ve elected to open source it, as it appears there are a lot of other people trying to solve similar problems. The name, Kaufmann, comes from Kafka’s <a href="http://literaryjoint.blogspot.com/2016/05/der-kaufmann-by-franz-kafka-merchant-or.html"><em>Der Kaufmann</em></a>, the seventh story in Kafka’s Meditations. The name was chosen haphazardly, don’t read too much into it.</p><p>KaufmannEx is built around two core components: <em>Event Handlers</em> and <em>Event Publishers.</em></p><p><strong>Event Handlers</strong></p><p>Event Handlers handle events consumed from a Kafka topic. KaufmannEx starts a <a href="https://github.com/kafkaex/kafka_ex">KafkaEx</a> Consumer Group, and dispatches consumed events in parallel to an EventHandler module.</p><p>An Event Handler is expected to define a given_event/1 method which will be called with incoming messages.</p><p>So for instance, if we were handling an event to create a new user, we might write this event handler:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd470727d70d6e35b4a982111303cbca/href">https://medium.com/media/fd470727d70d6e35b4a982111303cbca/href</a></iframe><p>Elixir&#39;s pattern matching makes handling additional events really simple, just define another permutation of given_event/1. It&#39;s also recommended to have a catchall given_event/1 to handle unexpected messages (unless you want to crash on unexpected messages, which is a legitimate strategy)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b4293826fc7dbd117918b1774ac7a26/href">https://medium.com/media/6b4293826fc7dbd117918b1774ac7a26/href</a></iframe><p><strong>Event Publishers</strong></p><p>Publishing events is even simpler. Publishers are even strictly required, an event handler or other module could easily handle formatting events to be published. All that is truly required to publish a message is to call KaufmannEx.Publisher.publish/4. KaufmannEx will attempt to encode the provided message in the named event schema.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/88ae1f8733e8c65d2e0fb1311fba6f7f/href">https://medium.com/media/88ae1f8733e8c65d2e0fb1311fba6f7f/href</a></iframe><p><strong>Sample Application</strong></p><p>We’ve provided a <a href="https://github.com/sevenmind/kaufmann_ex/tree/master/sample_application">sample application</a> in the <a href="https://github.com/sevenmind/kaufmann_ex/tree/develop/sample_application">project repository</a> which provides a good starting point for writing an application with KaufmanEx.</p><p><strong>Testing</strong></p><p>KaufmannEx also provides a testing module KaufmannEx.TestSupport.MockBus to test Event Consumers and producers. MockBus can trigger events and test for their effects.</p><p>Consider this example from the Sample Project, where a simple test specifies one event and then observes the resulting event.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04d81e70f3b36e69c9166ba186107fc5/href">https://medium.com/media/04d81e70f3b36e69c9166ba186107fc5/href</a></iframe><p><strong>KaufmannEx’s internal parallelism</strong></p><p>KaufmannEx tries to be very parallel.</p><p>We start by using the <a href="https://hexdocs.pm/kafka_ex/KafkaEx.GenConsumer.html">KafkaEx GenConsumer</a> to consume from the configured topic(s) and partitions to parallel instances of a service. GenConsumer is clever about allocating topics across multiple instances of a service and can be relied on to balance topic partitions to members of a consumer group.</p><p>Internally KaufmannEx uses <a href="https://github.com/elixir-lang/gen_stage">GenStage</a> with a back-pressure mechanism to guarantee messages are only consumed from the bus as there is capacity to process said event. Each message produced from this GenStage is sent to a <a href="https://github.com/elixir-lang/gen_stage">GenStage ConsumerSupervisor</a> which de-serializes each event and calls EventHandler.given_event/1 in parallel.</p><p>The parallelism can be configured in the application config.exs as :kaufmann_ex, :event_handler_demand, with a default of 50</p><p>If that explanation wasn’t quite confusing enough, this diagram shows how KaufmannEx gets a message from Kafka to an EventHandler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*payhuG_5GjvM1WuPmJwrMg.png" /></figure><p><strong>Caveats</strong></p><p>KaufmannEx was built for internal use at 7Mind and then generalized, causing some behaviors and patterns to occur as a result. The default message naming scheme, for instance, is what we use internally and may not generalize terribly well to all use cases. We’re working on improving this, any and all contributions are welcome.</p><p><em>This is also our first significant Elixir project. We’ve almost certainly overlooked some aspects of Elixir or Kafka. This project has a lot of room for improvement. Please check out our </em><a href="https://github.com/sevenmind/kaufmann_ex"><em>Github</em></a><em>, and open an issue or pull request.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=317415c27978" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>