<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sam Goldman on Medium]]></title>
        <description><![CDATA[Stories by Sam Goldman on Medium]]></description>
        <link>https://medium.com/@sam.goldman?source=rss-3df16f5cc4a4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*g239EgLiwAoBocri.</url>
            <title>Stories by Sam Goldman on Medium</title>
            <link>https://medium.com/@sam.goldman?source=rss-3df16f5cc4a4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 12:31:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sam.goldman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Developing with Kafka and Rails Applications]]></title>
            <link>https://medium.com/@sam.goldman/developing-with-kafka-and-rails-applications-783799e13489?source=rss-3df16f5cc4a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/783799e13489</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Sam Goldman]]></dc:creator>
            <pubDate>Mon, 31 Jul 2017 14:55:35 GMT</pubDate>
            <atom:updated>2017-08-29T17:44:09.414Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qa98xJZC6SINK0QznwaQzA.png" /></figure><p>As our data needs at Blue Apron grow more complex, our engineering team has chosen <a href="https://kafka.apache.org/">Apache Kafka</a> for publishing event streams. Kafka is an attractive option for many reasons, including its scalability and robustness.</p><p>Blue Apron develops many of our applications using Ruby on Rails. Unfortunately, Ruby is not a typical language for stream processing, making it challenging to effectively leverage Kafka and data pipelines. So how are we able to use Kafka from within our Ruby applications?</p><h3>Kafka Client</h3><p><a href="https://www.confluent.io/">Confluent</a>, a company started by LinkedIn engineers dedicated to the Kafka project, maintains a REST API proxy service, which provides a means for applications of any stack to interface with Kafka brokers, regardless of availability of client libraries. However, the REST proxy comes with a few drawbacks. It introduces another dependency in our stack and another network hop for each request. In addition, using the REST proxy requires application developers to get involved in the weeds of maintaining consumer sessions, error handling and retry logic.</p><p>A nicer solution for Ruby applications is to use the leading Kafka client gem, <a href="https://github.com/zendesk/ruby-kafka">ruby-kafka</a>, developed and maintained by Zendesk. This library connects directly to all of the brokers in a cluster using Kafka’s message protocol, handles errors and retries, and does so using concise Ruby syntax. This abstracts away complexity and allows application logic to be our focus.</p><p>Unlike the Java or C-based client libraries, ruby-kafka is far less battle-tested. In fact, as of this writing, ruby-kafka support for Kafka v0.10 was listed as “Limited support.” Before committing to widespread use of this library across our applications, we load tested the library, checking latency and accuracy both for consuming production Kafka topics, as well as producing (and consuming for validation) streams with throughput an order of magnitude higher than our organization’s current needs.</p><p>Producing events in Rails apps with the ruby-kafka gem couldn’t be easier. First, establishing a connection to the Kafka brokers is somewhat costly, so we want to do this only once. We can create a singleton client class to enforce this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6bcf5841e1d98c06dca14d7d0a17765e/href">https://medium.com/media/6bcf5841e1d98c06dca14d7d0a17765e/href</a></iframe><p>Now we can easily call this from anywhere while processing a request:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5809bfae3778803ba6bc2059da25d767/href">https://medium.com/media/5809bfae3778803ba6bc2059da25d767/href</a></iframe><p>Our message is off to the Kafka brokers. Downstream consumers, in other applications, or in other processes can subscribe and consume this data, making use of it across an array of tools and services.</p><p>But how can a Rails app consume Kafka topics? This is slightly more involved as it requires a separate process to continually poll the Kafka brokers, processing incoming messages as they arrive. To do so, create a new file: /bin/consumer. This will be the entry point for a new process, which will run completely independently the web server process (unicorn, puma, WEBrick, or whatever you use).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dda3ae494320a776ca926d64b8c06f2d/href">https://medium.com/media/dda3ae494320a776ca926d64b8c06f2d/href</a></iframe><p>What’s happening here? First, we have to boot the application and load all dependencies ourselves. Requiring &#39;bundler/setup&#39; should add all the libraries specified in our <em>Gemfile</em> to our $LOAD_PATH. This allows us to require them in our process, if they are needed. Next, we need to manually add project files to the $LOAD_PATH so that they can be used within this process. Because this process doesn’t run the Rails framework, we have to manually load application initializers. However, some of the initializer files in the project may not be applicable to the limited functionality of this consumer process, so in practice we probably want to load only a subset, rather than everything matchingconfig/initializers/*.rb. If this process is going to depend on any data stores, we would have to load their client libraries and initialize connections (this example consumer has no such dependencies). Finally, the code can gather any configuration needed to consumer messages from Kafka, create a new client object, and tell it to start consuming messages. What we do with the messages within the #each_message block will entirely depend on what the message data is, and how we want to use it. Within the block, we can either directly run processing instructions, or pass the message to a method in another class. The Zendesk client gives us plenty of information in the resulting message object, including <em>offset</em>,<em> key</em>,<em> value</em>,<em> topic </em>and<em> partition</em>.</p><h3>Development Environment</h3><p>Developing and testing applications that interface with a Kafka cluster is more involved than spinning up a standard Rails app with persistence and caching dependencies. If we used the REST proxy, this might be slightly simpler, but then we’d incur the costs mentioned above. So what do we use to develop these applications locally?</p><p>Running a functioning Kafka cluster on your development machine requires several coordinated services. Kafka uses <strong>Zookeeper</strong>, a distributed key-value store, to manage and configure the cluster. We cannot have a cluster without one or more Kafka <strong>brokers</strong> to handle the actual stream traffic. Additionally, there are other services, such as the Confluent <strong>schema registry</strong> and the <strong>REST proxy</strong>, which are out of the scope of this blog post. We’ve included them in our Kafka development stack as they are used by some applications within our organization. Having to manage these services creates a heavy burden for developers who want to focus on writing application code, rather than managing a complex development environment.</p><p>Our solution at Blue Apron is a two step process. First, we created a repo containing a <em>docker-compose</em> file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6cd91c9e0fa0ce88fe2f404e0fa35264/href">https://medium.com/media/6cd91c9e0fa0ce88fe2f404e0fa35264/href</a></iframe><p>In production, each of these services would be deployed and run independently, but still configured to connect to one another. Our application connects only to the brokers, and as long as each of the services can connect to its dependencies, development (on docker) mimics production. In development, running<em> </em>docker-compose up<em> </em>within this repo starts these Kafka related services, each running within its own container. Connections between services are made possible via a docker bridge network. Developing and testing Kafka pipelines often involves producing and consuming events in different applications, so we took a flexible approach to running and establishing connections to dependencies. We didn’t want to require that all apps be running in docker containers on one specific docker network. Developers may want to run their apps directly on the host, or from an independent docker network.</p><p>Here we make sure that connections from outside the docker network can resolve the broker’s advertised address, without interfering with DNS resolution within the docker network. Kafka brokers can be configured to advertise their address, since cluster deployment strategies can make externally accessible addresses differ from internal ones. For our containerized Kafka brokers, we must be careful configuring the advertised address. We need a guarantee that it will resolve both from within another container inside the docker network as well as from other sources elsewhere on the host machine. For example, using <em>localhost:port</em> doesn’t work even if we use docker’s port forwarding from the docker network, because <em>localhost</em> within the zookeeper container will resolve to the container, while <em>localhost</em> on our host machine will instead resolve to itself. The address <em>kafka_broker</em> works well out of the box for containers inside the docker network, but doesn’t resolve from our host machine, or from additional docker networks by default. However, we can fix this. By adding an entry to the hosts file in /etc, we can associate additional hostnames to 127.0.0.1. For our purposes, we want to add a row:</p><pre>127.0.0.1       kafka_broker</pre><p>Now, this address will resolve from containers inside the docker network, and from our host (and by extension, from other docker networks as well). We are ready to have development applications connect to our Kafka stack!</p><h3>Conclusion</h3><p>There are a few additional considerations when integrating Kafka streams into your Rails apps.</p><p>Kafka messages are nothing more than bytes, allowing developers to publish and model data however they see fit. In practice, we need a serialization convention for our messages.</p><p>Keeping evolving versions of our event schema organized and available across our engineering department could potentially be a task unto itself. Confluent provides a <a href="https://github.com/confluentinc/schema-registry">schema registration service</a> to help us. Exploring the schema registry and best practices surrounding it could fill an entire blog post on its own. We will merely suggest taking a look at the <a href="https://github.com/dasch/avro_turf">avro_turf</a> gem, which helps serialize and deserialize messages, while using a schema registry to manage and validate versions of message schema.</p><p>Lastly, Zendesk just released a new gem called <a href="https://github.com/zendesk/racecar">racecar</a> which abstracts much of the custom setup required for running your own consumer processes. Blue Apron is looking into the gem, as simplifying integration between Rails app functionality and independent consumer processes would save a lot of time and setup. In the meantime, take a look at racecar for yourself, or follow the steps above to manually initialize your own consumer process using the zendesk ruby-kafka client.</p><p>Kafka can help simplify moving data among applications. Rails applications are no exception. We hope developers consider integrating with Kafka on their Rails stacks, contribute to and build on the open source tools already available, and take advantage of all the features events streams and data pipelines have to offer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=783799e13489" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>