<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Laszlo Papp on Medium]]></title>
        <description><![CDATA[Stories by Laszlo Papp on Medium]]></description>
        <link>https://medium.com/@laszlo_papp?source=rss-b204a66e8f4e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*BA2qzlEuDJC75gSfitr4lA.jpeg</url>
            <title>Stories by Laszlo Papp on Medium</title>
            <link>https://medium.com/@laszlo_papp?source=rss-b204a66e8f4e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 00:48:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@laszlo_papp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chaos Engineering comes to Ruby]]></title>
            <link>https://zendesk.engineering/chaos-engineering-comes-to-ruby-8273333eff6c?source=rss-b204a66e8f4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8273333eff6c</guid>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[reliability]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Laszlo Papp]]></dc:creator>
            <pubDate>Tue, 31 Mar 2020 03:32:41 GMT</pubDate>
            <atom:updated>2020-03-31T03:32:41.140Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-rsTrfNH8SSVWp1OO3axAQ.png" /></figure><p>In a micro-service environment, an unexpected service outage often comes as a surprise and when it happens it’s stressful for engineers, managers, and the clients. If you want to prepare your Ruby application for the next surprise and prevent it early then this post is for you. I will show you how you can simulate a connection failure with timeout in your Ruby application and check if it can handle a service outage and recover from that.</p><p>Chaos Engineering is the discipline of experimenting with injecting harmful behaviors into the software to prepare the system and the engineering team for unknown situations. The idea behind chaos engineering is similar to flu prevention. Doctors inject a weaker form of the flu virus into the human body to prepare its defense mechanism for the real virus so it can memorize, practice the recovery. In a multi-service environment, viruses are network blips, network latency, memory and CPU spikes, unlikely situations. Injecting these sorts of issues into a production environment in a controlled way helps to set up and test backup plans, minimize the downtime of your service and also put less stress on you in the future.</p><h3>Flu Shot</h3><p><a href="https://github.com/nucc/flu_shot">Flu Shot</a> is an open-source Ruby gem that allows you to inject harmful behaviors into your application and control the behaviors externally like a control panel for a train layout. Using Flu Shot you can emit and simulate unusual events in your production environment in a controlled way and test your app’s resiliency. You can find the project on <a href="https://github.com/nucc/flu_shot">Github</a>.</p><h4>Setup</h4><p>First, you must install the gem before by gem install flu_shot or add it to the Gemfile of your application and run bundle install.</p><pre># Gemfile</pre><pre>gem &#39;flu_shot&#39;</pre><h4>Injection</h4><p>FluShot.inject defines a point in the execution flow where the harmful behaviors can be added later. The following example adds the user_controller_show injection point into theUsersController#show method right before the user is fetched from the Users Service. It’s not doing anything at the beginning, we will configure it later.</p><pre>class UsersController &lt; ApplicationController<br>  def show<br>    FluShot.inject(:user_controller_show)  # injection point</pre><pre>    user = UsersService.find(params[:user_id]) <br>    # ...<br>  end<br>end</pre><h4>Vaccine</h4><p>Basically, we will inject vaccines, weak harmful behaviors into the system. FluShot::Vaccine classes define the behaviors that can be executed at the injection points. Every vaccine needs to be labeled by using label :label_name. The behavior that the vaccine contains goes to the constructor method and additional parameters can be passed in a hash argument.</p><p>In my example, theLatency vaccine adds random length of sleep in[min..max] range simulating slow service. The min and the max values of the range must be passed in the params hash.</p><pre>class Latency &lt; FluShot::Vaccine<br>  label :latency</pre><pre>  def initialize(params = {})<br>    sleep(rand(params[:max] - params[:min]) + params[:min])<br>  end<br>end</pre><p>You can also raise and simulate exceptions in a vaccine by using FluShot::Sneeze that encapsulates an exception. The reason why we need to use the Sneeze object is Flu Shot catches every exception that is raised in the vaccines to make sure it does not abort your app. In order to raise exceptions from vaccines, the exception needs to be wrapped into a Sneeze object. The following vaccine will raise a Faraday::Error::ConnectionFailed exception.</p><pre>class ConnectionError &lt; FluShot::Vaccine<br>  label :connection_error</pre><pre>def initialize<br>    raise FluShot::Sneeze.new(<br>      Faraday::Error::ConnectionFailed.new(&#39;Connection Failed&#39;)<br>    )<br>  end<br>end</pre><h4>Prescription</h4><p>FluShot::Prescription associates vaccines with injection points. The Prescription provides a .for method to specify the injection point and it returns a block with a prescription local variable. You can specify which vaccine needs to be executed and with what parameters by using the .add method on this local variable.</p><p>The next example simulates a Faraday connection failure with a random timeout in UserController#show. The prescription is written for theuser_controller_show injection point that we have already defined before. Using the prescription object, we can add a latency vaccine with [1..3] seconds random timeout and raising a Faraday::Error::ConnectionFailed exception by the connection_error vaccine.</p><pre>FluShot::Prescription.for(:user_controller_show) do |prescription|<br>  prescription.add(:latency, {min: 1000, max: 3000})<br>  prescription.add(:connection_error)<br>end</pre><p>Now if you call theUsersController#show method, it will raise a connection failure error and simulates when the Users service is unreachable. If you have some test accounts, you can add some filters to execute the vaccine only for test accounts and in this case your clients are not affected at all.</p><p>Leaving the block empty will reset the prescription to no-op.</p><pre>FluShot::Prescription.for(:user_controller_show) do |prescription|<br>end</pre><h4>Storage</h4><p>By default, Flu Shot stores the prescriptions in memory and it works fine for single-process applications. Multi-process apps require shared memory to allow the processes to communicate with each other, therefore Memcache or Redis like services are necessary. In case you use Redis, you can pass your Redis connection instance to FluShot::Config.storage by wrapping it into FluShot::Storage::Redis. If you put the following lines into your initializers, new prescriptions will be applied automatically in each process of your app.</p><pre>require &#39;redis&#39;</pre><pre>FluShot::Config.storage = FluShot::Storage::Redis.new(Redis.new)</pre><p>If you use the Redis storage, you can control your application directly from Ruby console by writing prescriptions there, otherwise, you need to prepare your application to receive commands through HTTP and define new prescriptions configuration there.</p><h3>Contribution</h3><p>Flu Shot is looking for contributors for writing vaccines for popular third-party libraries like Faraday and Rails, and to create a “pharmacy” for the vaccines. This project is in the early phase, your ideas and recommendations are welcome. If you like the project but you don’t know how to contribute, please star it on <a href="https://github.com/nucc/flu_shot">Github</a>.</p><p>You can find me with questions on <a href="https://twitter.com/nucc">Twitter</a> or leave a comment below this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8273333eff6c" width="1" height="1" alt=""><hr><p><a href="https://zendesk.engineering/chaos-engineering-comes-to-ruby-8273333eff6c">Chaos Engineering comes to Ruby</a> was originally published in <a href="https://zendesk.engineering/">Zendesk Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka on Ruby]]></title>
            <link>https://zendesk.engineering/kafka-on-ruby-fdab12302146?source=rss-b204a66e8f4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdab12302146</guid>
            <dc:creator><![CDATA[Laszlo Papp]]></dc:creator>
            <pubDate>Mon, 17 Feb 2020 02:53:39 GMT</pubDate>
            <atom:updated>2020-02-17T02:53:39.948Z</atom:updated>
            <content:encoded><![CDATA[<p>Kafka is written in Scala and Java and you can get great support and tools if you’re using these languages. On the other side, if you use another language like Ruby, you can run into unmatured libraries and small developer communities. I want to show you some tools in Ruby that can help you to start using Kafka, and which library or framework might be the best choice for you if you’re on the fence about which tool would fit your requirements. I’ll show how you can start producing and consuming messages quickly with the Kafka libraries and how different frameworks can help you when your system becomes bigger.</p><h3>What is Kafka?</h3><p>Kafka is a distributed pub-sub messaging system that keeps track of streams of events, very similar to log files where applications log events stored as a stream and written to the disk, usually into the /var/log folder on Unix systems. Multiple processes of an application can log messages into the same file, while on the other side, log processors convert these messages into a readable format, for instance converting a plain text into JSON format and store it in Elastic Search as a document or sending an alert when the log severity is low. In my current example, the applications behave as producers and converters behave as consumers.</p><p>What Kafka actually does is something very similar to what we do in Unix shells when we write or read a stream of lines in a file:</p><ul><li><strong>producer:</strong> echo “hello world&quot; &gt;&gt; /var/log/log_file;</li><li><strong>consumer:</strong> tail -f /var/log/logfile | consumer_app</li></ul><h3>Ruby Libraries and Frameworks</h3><p>Right now there are two popular open-source Kafka libraries for Ruby, <a href="https://github.com/zendesk/ruby-kafka">ruby-kafka</a> and <a href="https://github.com/appsignal/rdkafka-ruby">ruby-rdkafka</a>. ruby-kafka from Zendesk is fully written in Ruby, while ruby-rdkafka is a wrapper around the <a href="https://github.com/edenhill/librdkafka">rdkafka</a> C++ library. These libraries support both writings and readings of Kafka streams and you can use them to produce and consume events on Kafka streams. There are some open-source frameworks, <a href="https://github.com/zendesk/racecar">Racecar</a>, <a href="https://github.com/phobos/phobos">Phobos</a> and <a href="https://github.com/karafka/karafka">Karafka</a> that help you to manage multiple producers and consumers, also to organize them into groups, application units, and processes. These frameworks currently are built on the top of theruby-kafka library but some frameworks are moving to ruby-rdkafka in their early-version releases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DJr9nDThpV5ZSizgqimceQ.png" /><figcaption>Kafka Libraries and Frameworks in Ruby</figcaption></figure><h3>Producers</h3><p>Producing messages are really like writing into a file, I’ll show you how you can do that with ruby-kafka and rdkafka.</p><h4>Rdkafka</h4><p>rdkafka is a C++ wrapper around <a href="https://github.com/edenhill/librdkafka">librdkafka</a>, therefore using this library requires a C++ compiler installed on your environment. To set up a producer, you will need to specify some configuration details, for instance, the address of your bootstrap servers where the Kafka brokers are running. In my case, it’s running on thekafka.docker host, on the 9092 default port. If you use a docker image, probably your broker address is localhost:9092. To start sending messages, we need to create a producer according to our configuration and call the produce method on this instance that automatically emits events to Kafka.</p><pre>require &#39;ruby-rdkafka&#39;</pre><pre>config = { &quot;bootstrap.servers&quot;: &quot;kafka.docker:9092&quot; }<br>producer = Rdkafka::Config.new(config).producer<br>producer.produce(topic: &quot;test&quot;, payload: &quot;Hello World!&quot;)</pre><p>ruby-rdkafka emits events in async, the message will be written to a queue and you need call wait on the producer, if you need a sync request:</p><pre>producer.produce(topic: &quot;test&quot;, payload: &quot;Hello World!&quot;).wait</pre><h4>Ruby-kafka</h4><p>In ruby-kafka it’s very similar to rdkafka, the constructor of Kafka expects the Kafka brokers and then calling deliver_message will write to the stream.</p><pre>require &quot;kafka&quot;</pre><pre>kafka = Kafka.new([&quot;kafka.docker:9092&quot;])<br>kafka.deliver_message(&quot;Hello World!&quot;, topic: &quot;test&quot;)</pre><p>deliver_message is a sync operator, the function won’t return until the message is written into the wire. If you don’t necessarily need to know the result of deliver_message, you can send the message async, in this case ruby-kafka will maintain a thread that manages a queue in the background.</p><pre>kafka.async_producer.produce(&quot;Hellow World&quot;, topic: &quot;test&quot;)</pre><p><strong><em>note:</em></strong><em> async sending happens in the background, </em><em>ruby-kafka maintains a queue for pending messages and will write all messages to the wire in a background thread. Make sure you don’t have unsent messages in the queue when your process terminates.</em></p><h3>Consumers</h3><p>Both ruby-kafka and rdkafka provides solutions for consuming messages. There are multiple frameworks that wrap these libraries into a complete platform and make it easy to add and scale consumers. Let’s start first with some basic consumers in rdkafka and ruby-kafka.</p><h4>Rdkafka</h4><p>The configuration of the consumer is similar to the producer’s config, the bootstrap.servers option needs to be specified to know where the Kafka server is located, but there’s an additional group.id setting that we need to specify. Group ID defines the Kafka group name which the consumer will join. When your consumer starts processing messages, the Kafka broker will keep in track the last message that the consumer group processed successfully. If you use the same group id, you can stop your consumer any time, next time it’s going to process the next unprocessed message, regardless of how long it was stopped.</p><pre>require &#39;rdkafka&#39;</pre><pre>config = { <br>  &quot;bootstrap.servers&quot;: &quot;kafka.docker:9092&quot;, <br>  &quot;group.id&quot;: &quot;ruby-test&quot;<br>}</pre><pre>consumer = Rdkafka::Config.new(config).consumer</pre><p>Now there’s a consumer instance, we just need to specify the topic that it will read from and a basic iterator that going to be yielded when a message was written to the topic.</p><pre>consumer.subscribe(&#39;test&#39;)</pre><pre>consumer.each do |message|<br>  puts &quot;Message received: #{message}&quot;<br>end</pre><h4>Ruby-kafka</h4><p>The configuration is similar to the producer, in the constructor we need to pass the docker hostname and port number</p><pre>require &quot;kafka&quot;</pre><pre>kafka = Kafka.new([&quot;kafka.docker:9092&quot;])</pre><pre>consumer = kafka.consumer(group_id: &quot;ruby-test&quot;)</pre><p>Consuming the messages are very similar to the way inrdkafka , the consumer needs to subscribe to a topic and iterate on the messages with the each_message iterator.</p><pre>consumer.subscribe(&#39;test&#39;)</pre><pre>consumer.each_message do |message|<br>  puts &quot;Message received: #{message}&quot;<br>end</pre><h3>Frameworks</h3><p>Using only the libraries may help you to start processing messages from a topic quickly, especially when you’re working a small script that requires some data from Kafka. When you use each or each_message methods above provided by the libraries, you need to take into consideration that they are blocking the execution flow, therefore you need to use threads or background processes if you want to consume multiple topics concurrently. When you work with a multi-thread environment, there are certain things you need to deal with, e.g. graceful shutdown, backoff strategy. The following frameworks can help you to avoid some head-aches by putting the basic consumer for-each loops into threads and processes and providing configs to manage them in an easy way. Since producing messages is quite simple by using the libraries, in the following sections I would focus on consuming messages only.</p><h4><a href="https://github.com/zendesk/racecar">Racecar</a></h4><p>It’s one of the simplest Kafka frameworks. Racecar wraps only one consumer into a process and the framework handles everything you need in a production environment like instrumentation and graceful termination when the process gets a SIGTERM signal. With racecar you don’t need to worry about Kafka consumer groups, everything happens behind the hood.</p><p>To start playing with Racecar, you need to add the racecar gem into your project’s Gemfile:</p><pre># Gemfile</pre><pre>gem &#39;racecar&#39;</pre><p>and implement a class that is inherited from Racecar::Consumer. Here is an example for a consumer that consumes the test topic. If you are using Rails, it’s recommended to put your consumers into the /app/consumers folder.</p><pre># /app/consumers/test_consumer.rb</pre><pre>class TestConsumer &lt; Racecar::Consumer<br>  def process(message)<br>    puts message.inspect<br>  end<br>end</pre><p>When Racecar boots up it creates an instance of your class and calls the process method on it every time a message is read from the topic. Because there’s only one consumer instance being created during the boot, instance variables will be shared between the requests, that’s why it’s strongly recommended to not store states in instance variables on a multi-tenant system.</p><p>You can start consuming messages with passing the class name to the racecar application:</p><pre>bundle exec racecar TestConsumer</pre><h4><a href="https://github.com/phobos/phobos">Phobos</a></h4><p>When you have multiple consumers, you might want to add them into the same process, especially when you work with Rails and you need to load the whole application into the memory before. Putting consumers into separated processes can multiply the memory usage of your application that could add extra cost to your cloud budget.</p><p>The big advantage of Phobos comparing to Racecar is you can specify what consumers you want to execute in the same process and how many concurrent consumers, which is very useful if your topic contains multiple partitions.</p><p>To add Phobos into your project, add the phobos gem into your Gemfile.</p><pre># Gemfile</pre><pre>gem &#39;phobos&#39;</pre><p>You can initialize it and generating the configuration by executingphobos init in your terminal.</p><pre># /app/consumers/test_consumer.rb</pre><pre>class TestConsumer<br>  include Phobos::Handler</pre><pre>  def consume(payload, metadata)<br>    puts metadata<br>    puts payload<br>  end<br>end</pre><p>phobos init will create a config file for you into /app/config/phobos.yml if you’re using Rails. The following configuration will create twoTestConsumers, they will consume messages from the test topic and will join thetest-consumer-group Kafka consumer group.</p><pre># /app/config/phobos.yml</pre><pre>kafka: <br>  seed_brokers: <br>    - kafka.docker:9092</pre><pre>listeners:<br>  - handler: TestConsumer<br>    topic: test<br>    group_id: test-consumer-group<br>    max_concurrency: 2  # run it on 2 threads</pre><p>If you want to add more consumers to the same process, you need to create a new handler with a similar configuration to the TestConsumer. In this case, the same process will execute multiple threads for each consumer instance.</p><p>You can start the consumers by</p><pre>bundle exec phobos start</pre><p>If you want to put consumers into processes to balance your load in your production environment then you need to create different config files for each group and pass the path of the config to phobos start :</p><pre>bundle exec phobos start -c config/test_consumers.yml<br>bundle exec phobos start -c config/user_events.yml</pre><h4><a href="https://github.com/karafka/karafka">Karafka</a></h4><p>Phobos and Racecar are very small frameworks, if I wanted to compare them to Ruby Web frameworks then I would say Racecar is like Sinatra, Phobos is like Padrino and Karafka is like Rails.</p><p>To start using Karafka in your project, add thekarafka gem into the Gemfile.</p><pre># Gemfile</pre><pre>gem &#39;karafka&#39;</pre><p>after executing bundle install just run the following command to set up your environment and get the default configuration file.</p><pre>bundle exec karafka install</pre><p>In the root folder of your application, you should get a karafka.rb file, the configuration file that describes your environment and the routing definitions.</p><pre># /karafka.rb</pre><pre>class KarafkaApp &lt; Karafka::App<br>  config.kafka.seed_brokers = %w[kafka://kafka.docker:9092]<br>  config.client_id = &#39;example_app&#39;<br>end</pre><pre>consumer_groups.draw do<br>  topic :test do<br>    consumer TestConsumer<br>  end</pre><pre>  consumer_group :user_events do<br>    topic :user_updates<br>      consumer UserUpdates<br>    end</pre><pre>    topic :user_notifications do<br>      consumer UserNotifications<br>    end<br>  end<br>end</pre><p>consumer_groups.draw describes topics and other consumer_groups. Each topic and consumer_group in theconsumer_groups.draw block is going to be executed on its own thread, in the example above there are going to be 2 threads, one for the TestConsumer and another for the user_events consumer group. If you want to scale out and run it on multiple processes, you need to start multiple Karafka apps.</p><p>Let’s see how you can add a basic TestConsumerinto your project. Under the /app/consumers create a file with test_consumer.rb filename:</p><pre># /app/consumers/test_consumer.rb</pre><pre>class TestConsumer &lt; Karafka::BaseConsumer<br>  def consume<br>    # You can reach the message in the params like in Rails<br>    puts params.inspect<br>  end<br>end</pre><p>Now there’s nothing more left just to start your Karafka application by</p><pre>bundle exec karafka server</pre><p>If you want to start only certain consumer groups, you can pass the consumer group names as extra parameters</p><pre>bundle exec karafka server --consumer_groups user_events</pre><p>in the current example, only consumers that belong to the user_events group are going to be executed only.</p><p>Karafka is a massive framework with lots of configuration options and consumer features, you can find more details in their <a href="https://github.com/karafka/karafka/wiki">documentation</a>.</p><h3>Summary</h3><p>These libraries and frameworks can help you start integrating Kafka with your application and start producing and consuming messages quickly. I didn’t dive into the very details but they also provide you rich configuration for optimizing your producers and consumers. The Racecar framework is a nice lightweight tool to start using Kafka quickly, but as soon as the number of the consumers increases you might need to consider using Phobos or Karafka because they can manage consumer groups, pipelines better.</p><p>Please like the page if you want to hear more about Kafka and Ruby and in another post, I can dive into the details. Thanks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdab12302146" width="1" height="1" alt=""><hr><p><a href="https://zendesk.engineering/kafka-on-ruby-fdab12302146">Kafka on Ruby</a> was originally published in <a href="https://zendesk.engineering/">Zendesk Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>