<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Danny Kay on Medium]]></title>
        <description><![CDATA[Stories by Danny Kay on Medium]]></description>
        <link>https://medium.com/@danieljameskay?source=rss-48cf87e0ed3a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NcxbBJ1vZHVVBf_62qzhQQ.jpeg</url>
            <title>Stories by Danny Kay on Medium</title>
            <link>https://medium.com/@danieljameskay?source=rss-48cf87e0ed3a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:34:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danieljameskay/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A basic overview of the Kafka Connect RabbitMQ Source Connector ]]></title>
            <link>https://medium.com/@danieljameskay/a-basic-overview-of-the-kafka-connect-rabbitmq-source-connector-abeba64ba453?source=rss-48cf87e0ed3a------2</link>
            <guid isPermaLink="false">https://medium.com/p/abeba64ba453</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[rabbitmq]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-connect]]></category>
            <category><![CDATA[integration]]></category>
            <dc:creator><![CDATA[Danny Kay]]></dc:creator>
            <pubDate>Thu, 20 Jun 2019 18:57:10 GMT</pubDate>
            <atom:updated>2019-06-20T20:57:26.423Z</atom:updated>
            <content:encoded><![CDATA[<h3>An overview of the Kafka Connect RabbitMQ Source Connector 🐇</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QXjCgsVPOHKb7f9r6O5Wmg.jpeg" /><figcaption>I love rabbits…Photo by <a href="https://unsplash.com/@shannonlitt?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Shannon Litt</a> on <a href="https://unsplash.com/search/photos/rabbit?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Integration&#39;s my middle name…well, its isn&#39;t really but integration solutions and patterns was something I found a real interest for early on in my Software Engineering career. Bridging the gap between different systems and developing strategies for enterprise integration was extremely interesting and cool.</p><p>Back in the day my usual go-to stack was JAX-WS, JAX-RS, EJB’s and JMS…now, its Apache Kafka. My colleagues know I have a soft spot for Apache Kafka and the supporting API’s, and rightly so, for integration solutions its amazing.</p><p>I’m a big advocate of KSQL, Kafka Connect and Kafka Streams. Being able to integrate data from external systems, process that data and make that data available to other external systems makes Kafka a great platform for integration and data processing.</p><h3>Brief introduction to Kafka Connect</h3><p>I’ve recently been looking into integrating RabbitMQ with Apache Kafka.</p><p>Introducing…Kafka Connect.</p><p>Kafka Connect is pretty awesome. It allows you to integrate Kafka with external systems such as MySQL, ElasticSearch, Hadoop and many more.</p><p>Say you’re integrating an pre-existing solution with Kafka and that solution uses MySQL for storing data. You could create a connector which streams the new and updated data from MySQL into Kafka to be processed by Kafka Streams, KSQL or another external system.</p><p>There are two different types of connectors. Source connectors and sink connectors.</p><p>A source connector can be thought of as a shoveler of data from a particular source such as a database.</p><p>A sink connector delivers data from Kafka topics into external systems.</p><p>The concepts pretty straight forward. If you want to find out more information about Kafka Connect, have a read of <a href="https://docs.confluent.io/current/connect/index.html">this</a>. Robin Moffatt did an awesome talk at this years Kafka Summit about it also, slides are <a href="https://talks.rmoff.net/QZ5nsS/from-zero-to-hero-with-kafka-connect">here</a>.</p><p>Up to now I’ve only ever used sink connectors, having this opportunity to use a source connector was pretty awesome.</p><h3>Installing the RabbitMQ Connector</h3><p>Anyhow, first thing I did when considering Kafka Connect was check if there was a connector available. A bunch of connectors such as the JDBC and ElasticSearch connector come bundled in the Confluent Platform and Kafka Connect Docker image but the RabbitMQ connector doesn’t.</p><p>Whilst on the subject of Docker images, Confluent offer two different Kafka Connect images, confluentinc/cp-kafka-connect-base which contains two basic connectors, FileStreamSinkConnector and FileStreamSinkConnector. Whilst the alternative Docker Image, confluentinc/cp-kafka-connect contains a bunch of connectors such as the JDBC and ElasticSearch.</p><p>Luckily there is the Confluent Hub. The Confluent Hub is a sort of marketplace for connectors developed by both Confluent and 3rd parties. The RabbitMQ connector is available on the Confluent Hub and can be downloaded in a zip file or alternatively using the <a href="https://docs.confluent.io/current/connect/managing/confluent-hub/client.html">Confluent Hub Client</a>.</p><p>As the Confluent Hub Client is available through the Kafka Connect Docker image, I thought it would be easier and quicker to create a Docker image with the base Kafka Connect image and use the Confluent Hub Client to install the connector…</p><pre>FROM confluentinc/cp-kafka-connect-base:5.2.1<br>RUN confluent-hub install — no-prompt confluentinc/kafka-connect-rabbitmq:latest</pre><p>And then build the image…</p><pre>&gt; docker build -t connect-rabbitmq .</pre><p>Then use the newly created image in the Docker Compose file.</p><pre>kafka-connect:<br>    container_name: kafka-connect<br>    image: connect-rabbitmq:latest<br>    ports:<br>      - 8083:8083<br>    depends_on:<br>      - zookeeper<br>      - kafka<br>      - schema-registry<br>    environment:<br>      CONNECT_BOOTSTRAP_SERVERS: &quot;kafka:29092&quot;<br>      CONNECT_REST_PORT: 8083<br>      CONNECT_GROUP_ID: compose-connect-group<br>      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs<br>      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets<br>      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status<br>      CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter<br>      CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: &#39;<a href="http://schema-registry:8081&#39;">http://schema-registry:8081&#39;</a><br>      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter<br>      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: &#39;<a href="http://schema-registry:8081&#39;">http://schema-registry:8081&#39;</a><br>      CONNECT_INTERNAL_KEY_CONVERTER: &quot;org.apache.kafka.connect.json.JsonConverter&quot;<br>      CONNECT_INTERNAL_VALUE_CONVERTER: &quot;org.apache.kafka.connect.json.JsonConverter&quot;<br>      CONNECT_REST_ADVERTISED_HOST_NAME: &quot;kafka-connect&quot;<br>      CONNECT_LOG4J_ROOT_LOGLEVEL: &quot;INFO&quot;<br>      CONNECT_LOG4J_LOGGERS: &quot;org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR&quot;<br>      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: &quot;3&quot;<br>      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: &quot;3&quot;<br>      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: &quot;3&quot;<br>      CONNECT_PLUGIN_PATH: &#39;/usr/share/java,/usr/share/confluent-hub-components/&#39;<br>      CONNECT_HOST: kafka-connect</pre><p>As I wanted multiple Connect workers, I had two containers in my Docker Compose file, one named kafka-connect and another named kafka-connect1….I’m great with names. Just make sure you give them different names and different rest port values, in this case they were assigned 8083 and 8084.</p><p>One thing to point out, I’m running a replication factor of 3 for the Config, Status and Offset storage Topics so that if a broker fails, I have the values stored safely in other brokers.</p><p>There are some alternatives to building your own Docker image. You could use the command key in your Docker Compose file as shown in this <a href="https://github.com/confluentinc/demo-scene/blob/master/ksql-atm-fraud-detection/docker-compose.yml">example file</a> or use the ContainerInit key if you’re using a Kubernetes deployment. More information on ContainerInit can be found <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/">here</a>.</p><p>Once the Connect worker is alive and kicking you can request the following URL, curl <a href="http://localhost:8083/connector-plugins">http://localhost:8083/connector-plugins</a> | jq and this will show you the available connectors.</p><pre>[<br>  {<br>    &quot;class&quot;: &quot;io.confluent.connect.rabbitmq.RabbitMQSourceConnector&quot;,<br>    &quot;type&quot;: &quot;source&quot;,<br>    &quot;version&quot;: &quot;0.0.0.0&quot;<br>  },<br>  {<br>    &quot;class&quot;: &quot;org.apache.kafka.connect.file.FileStreamSinkConnector&quot;,<br>    &quot;type&quot;: &quot;sink&quot;,<br>    &quot;version&quot;: &quot;2.2.0-cp2&quot;<br>  },<br>  {<br>    &quot;class&quot;: &quot;org.apache.kafka.connect.file.FileStreamSourceConnector&quot;,<br>    &quot;type&quot;: &quot;source&quot;,<br>    &quot;version&quot;: &quot;2.2.0-cp2&quot;<br>  }<br>]</pre><p>Notice the RabbitMQSourceConnector? This means that we can create a connector that connects to RabbitMQ and sends events into Kafka.</p><h3>Creating a Connector</h3><p>Now that our Connect worker is up and running we can create a connector. There are two types of deployment modes, Standalone and Distributed. As the solution is intended to be ran in Kubernetes and needs to be highly available, Distributed workers are being used. With this in mind we will be interfacing with the REST endpoint the worker exposes.</p><p>To create a connector, we send a POST request containing a JSON object to the REST endpoint, in this case it is 8083.</p><p>If you have multiple workers at your disposable like I did in this scenario, you can send the POST request to any available worker and the configuration for the connector is stored in the config_storage Topic, this ultimately means that Connect workers are stateless and can be scaled up and down as needed.</p><p>The JSON object needs to contain a name for the connector and a config<strong> </strong>which contains parameters for the connector. A small point which has tripped me up in the past, all the values in the configuration should be strings. Below is a basic connector configuration. There are more available parameters which can be viewed <a href="https://docs.confluent.io/current/connect/kafka-connect-rabbitmq/rabbit_m_q_source_connector_config.html">here</a>.</p><pre>{<br>  &quot;name&quot; : &quot;RabbitMQSourceConnector&quot;,<br>  &quot;config&quot; : {<br>        &quot;connector.class&quot;: &quot;io.confluent.connect.rabbitmq.RabbitMQSourceConnector&quot;,<br>        &quot;tasks.max&quot;: &quot;2&quot;,<br>        &quot;kafka.topic&quot;: &quot;poloniex.ticker.data&quot;,<br>        &quot;rabbitmq.queue&quot;: &quot;poloniex.ticker.data&quot;,<br>        &quot;rabbitmq.username&quot;: &quot;rabbitmq&quot;,<br>        &quot;rabbitmq.password&quot;: &quot;rabbitmq&quot;,<br>        &quot;rabbitmq.host&quot;: &quot;rabbitmq&quot;<br>  }<br>}</pre><p>We specify a name for the connector then create the configuration. The configuration contains the connector class, the number of tasks we want to run, the Kafka topic we want to produce records to and some RabbitMQ specific keys and values. This is a pretty basic example, Confluent have provided an example <a href="https://docs.confluent.io/current/connect/kafka-connect-rabbitmq/rabbit_m_q_source_connector_example.html">here</a>. There are a ton more more properties that can be specified, these can be viewed <a href="https://docs.confluent.io/current/connect/kafka-connect-rabbitmq/rabbit_m_q_source_connector_config.html">here</a>.</p><p>One thing to pay attention to is “tasks.max”: “2&quot; , here we are setting the number of tasks we want to run. As we are running two connect workers, one task will run on each worker. We could set this value higher to have more Tasks running on each worker. Tasks do all the work under the hood, they are responsible in this case for consuming from RabbitMQ and producing to Kafka. As tasks are single jobs, we can scale tasks when needed. More information about Tasks can be found <a href="https://docs.confluent.io/current/connect/concepts.html#connect-tasks">here</a>.</p><p>If the POST request was a success the reply from the worker will be the original request body.</p><h3>Lets get some data into a RabbitMQ Queue</h3><p>With the worker up and running and the connector running we can start an app that sends events to the RabbitMQ queue.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e27681d0cde5641a730389117ea7eb9/href">https://medium.com/media/2e27681d0cde5641a730389117ea7eb9/href</a></iframe><p>This app is extremely basic but it does the job. As I wanted to get ticker data from the endpoint into RabbitMQ as quickly as possible, code quality was low down on my list for obvious reasons. It starts up, connects to RabbitMQ and creates a queue. It then creates a channel, connects to the streaming API that Poloniex provides and when a new ticker data event is received it is sent to the RabbitMQ queue.</p><p>Once the application is up, consuming from the Poloniex endpoint and producing to the RabbitMQ queue, we can take a look at the RabbitMQ admin interface to see some lovely graphs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*qCcx-B4bF0A31Np_Wi9jeg.png" /><figcaption>No consumers…</figcaption></figure><p>Whilst looking at I the graphs I noticed that the queue was building up and there were no listed consumers which meant one thing, the connector wasn&#39;t functioning correctly.</p><h3>A tiny malfunction…on my behalf 🔥</h3><p>So I tailed the logs of the worker and found the following error.</p><pre>ERROR Consumer io.confluent.connect.rabbitmq.ConnectConsumer@22c96d6f (amq.ctag-iFeabA8KDrAaNfZK-bMg8w) method handleDelivery for channel AMQChannel(amqp://rabbitmq@172.21.0.3:5672/,1) threw an exception for channel AMQChannel(amqp://rabbitmq@172.21.0.3:5672/,1) (com.rabbitmq.client.impl.ForgivingExceptionHandler)<br>java.lang.NullPointerException<br> at io.confluent.connect.rabbitmq.MessageConverter.basicProperties(MessageConverter.java:126)<br> at io.confluent.connect.rabbitmq.SourceRecordBuilder.sourceRecord(SourceRecordBuilder.java:40)<br>...<br>...</pre><p>The text that stood out for me was….</p><pre>java.lang.NullPointerException<br> at io.confluent.connect.rabbitmq.MessageConverter.basicProperties(MessageConverter.java:126)</pre><p>It turned out I’d missed some code from the application, the Basic Properties. So what are the Basic Properties I hear you ask?</p><p>So, a message is split into two parts, the message body and the message headers. We define a Basic Properties object which contains attributes for the header part of the message. The Basic Properties can contain values such as the message ID, a timestamp and many more. The full list of attributes can be viewed <a href="https://www.rabbitmq.com/releases/rabbitmq-java-client/v3.5.4/rabbitmq-java-client-javadoc-3.5.4/com/rabbitmq/client/AMQP.BasicProperties.html">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebf0b5bd5c52801b21ff040c82c42ceb/href">https://medium.com/media/ebf0b5bd5c52801b21ff040c82c42ceb/href</a></iframe><p>The application hasn&#39;t changed much, we’re now building up a BasicProperties object and including that in our channel.basicPublish() method call. The first value we provide in the BasicProperties object is the appId, this is an unique identifier for the application that produced the event, contentEncoding is the MIME encoding of the message content, contentType is the content type of the message, correlationId is an identifier to define relationships between messages, deliveryMode indicates whether the message should be persisted to disk (1 is non-persistent and 2 is persistent), expiration sets the TTL in milliseconds for messages which should be deleted from the queue if not consumed, headers allows us to create application specific headers, messageId is used as a unique identifier for the message and used for partitioning when producing to Kafka, priority allows us to set a priority for the message using a value from 0 — 9 this allows messages with a high priority to move ahead in the queue, replyTo not used in this scenario but contains a value for a private response queue when using request messages, timestamp contains the time when the message was sent by the application, type indicates the type of message this going to be send and lastly userId sets the identity of the user responsible for producing the message.</p><h3>The moment of truth?</h3><p>With the Basic Properties now in the mix, Docker containers restarted, connector configuration sent to one of the workers and the application running…it was the moment of truth…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*Oo_i6Qp0l8KkYX9bhlXB4g.png" /><figcaption>😎</figcaption></figure><p>SUCCESS! 😊😄👍👏🕺</p><p>We can see the queue backing up with queued messages over a short period of time when the messages were sent to the queue by the app, then see the Tasks fire up and start consuming from the queue and the backlog vanish.</p><p>We can also see that the average rate for consumer acknowledgement is 38 messages a second.</p><p>If we look further down the page we can see the consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*q0TY3MziJmZaDHNe1mSIHA.png" /><figcaption>Consumers…</figcaption></figure><p>The two consumers are the two tasks running on the two connect workers, pretty cool. If we click on the Channel IP address we can look at some information associated with the consumer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*GXugtsmJjDpMe69Lf_Qm5w.png" /><figcaption>Channel Information…</figcaption></figure><p>And last but not least, lets connect to the Kafka broker container and run a console consumer to make sure the topic is being populated on the Kafka side of things.</p><pre>docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic poloniex.ticker.data</pre><p>Its safe to say after running this command I was presented with incoming events…happy days!</p><h3>Wrapping up</h3><p>One thing I did want to do was move the containers in to Kubernetes and monitor the workers with Prometheus and Grafana…but that’s for another day.</p><p>I learned so much from looking into this solution, using real world problems and issues is a great way to learn.</p><p>The streaming endpoint I used is from <a href="https://docs.poloniex.com/#introduction">Poloniex</a>. Its a cryptocurrency exchange which exposes several endpoints over HTTP and Websocket.</p><p>Many thanks as always for reading my articles, it’s really appreciated. Any thoughts, typos, comments or questions drop me a tweet.</p><p>Cheers 👍🏻</p><p>Danny</p><p><a href="https://twitter.com/danieljameskay">https://twitter.com/danieljameskay</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abeba64ba453" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Streams DSL for Scala: The Basics]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@danieljameskay/kafka-streams-dsl-for-scala-the-basics-11d603295f5c?source=rss-48cf87e0ed3a------2"><img src="https://cdn-images-1.medium.com/max/2600/1*AZG01g9nu6wlCg9HuOTw2g.jpeg" width="4047"></a></p><p class="medium-feed-snippet">This month, I&#x2019;m learning Kafka Streams&#x2026;with a Scala twist.</p><p class="medium-feed-link"><a href="https://medium.com/@danieljameskay/kafka-streams-dsl-for-scala-the-basics-11d603295f5c?source=rss-48cf87e0ed3a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@danieljameskay/kafka-streams-dsl-for-scala-the-basics-11d603295f5c?source=rss-48cf87e0ed3a------2</link>
            <guid isPermaLink="false">https://medium.com/p/11d603295f5c</guid>
            <category><![CDATA[stream-processing]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Danny Kay]]></dc:creator>
            <pubDate>Thu, 21 Feb 2019 21:11:43 GMT</pubDate>
            <atom:updated>2019-05-01T08:19:48.590Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The basics of deploying Logstash pipelines to Kubernetes]]></title>
            <link>https://medium.com/data-science/the-basics-of-deploying-logstash-pipelines-to-kubernetes-94a470ad34d9?source=rss-48cf87e0ed3a------2</link>
            <guid isPermaLink="false">https://medium.com/p/94a470ad34d9</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[logstash]]></category>
            <dc:creator><![CDATA[Danny Kay]]></dc:creator>
            <pubDate>Sun, 20 Jan 2019 19:56:13 GMT</pubDate>
            <atom:updated>2019-01-21T15:29:46.292Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7qQZXueRN60yMTk5Zi8z1g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/jF946mh5QrA?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Hin Bong Yeung</a> on <a href="https://unsplash.com/search/photos/subway?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><blockquote>Do you have a long list of the things you want to learn on a list in your brain?</blockquote><p>I did, until I wrote that list down on a piece of paper and decided to do something about it. Towards the end of 2018 I started to wrap up things I’d been learning and decided to put some structure into my learning for 2019.</p><p>2018 had been an interesting year, I’d moved jobs 3 times and felt like my learning was all over the place. One day I was learning Scala and the next I was learning Hadoop. Looking back, I felt like I didn&#39;t gain much ground.</p><p>I decided to do something about it going into 2019. I wrote the 12 months of the year down on one side of a piece of paper and what I wanted to learn that month on the other side, with the idea that my learning for that one month would be focused on a particular product/stack/language/thing.</p><p>The Elastic stack, previously referred to as ELK, was at the top of this list for a few reasons.</p><p>I’ve worked with the Elastic stack before, more specifically ElasticSearch and Kibana but I felt like there was so much more I could learn from these two products. I also wanted to gain an understanding of Logstash and see what problems it could help me solve.</p><p>When I start learning something new I set a bunch of small, achievable objectives. One of the objectives I’d written was to have a fully functional, operating Logstash pipeline running in Kubernetes, ingesting data from somewhere, perform some action on it and then send it to ElasticSearch.</p><p>I’m a huge fan of Kubernetes. There has been a huge shift in the past few years to containerize applications and I fully embrace this shift. I had no interest in running this pipeline I was building locally, its was Kubernetes or bust!</p><p>I had a bit of a Google and struggled to find anything concrete regarding deployment, best practices etc thus led me to writing a basic article about how to get a basic Filebeat, Logstash and ElasticSearch pipeline running in Kubernetes.</p><p>So what’s Filebeat? It’s a shipper that runs as an agent and forwards log data onto the likes of ElasticSearch, Logstash etc.</p><p>Say you are running Tomcat, Filebeat would run on that same server and read the logs generated by Tomcat and send them onto a destination, more cases than not that destination is ElasticSearch or Logstash. FileBeat can also run in a DaemonSet on Kubernetes to ship Node logs into ElasticSearch which I think is really cool. Fluentd also does this but that’s for another day.</p><p>For more information about Filebeat check it out <a href="https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-overview.html">here</a>.</p><p>Right, so in our scenario we have Filebeat reading a log of some sort and its sending it to Logstash, but was is Logstash?</p><p>Logstash is a server side application that allows us to build config-driven pipelines that ingest data from a multitude of sources simultaneously, transform it and then send it to your favorite destination.</p><p>We can write a configuration file that contains instructions on where to get the data from, what operations we need to perform on it such as filtering, grok, formatting and where the data needs to be sent to. We use this configuration in combination with the Logstash application and we have a fully functioning pipeline. The beautiful thing about Logstash is that it can consume from a wide range of sources including RabbitMQ, Redis and various Databases among others using special plugins. We can then stash that data in S3, HFDS and many more! And this is all driven by a single config file and a bunch of plugins…amazing isn&#39;t it!</p><p>For more information about Logstash check it out <a href="https://www.elastic.co/products/logstash">here</a>.</p><p>Lets get on to some code and exciting stuff!</p><p>I was following the <a href="https://www.elastic.co/guide/en/logstash/6.5/advanced-pipeline.html">Logstash tutorial</a> on the Elastic site and had come across the perfect candidate for my pipeline…with some small modifications.</p><p>I’d followed the tutorial step by step, FileBeat was running, it was reading the log file mentioned in the tutorial, it was all good. I had to config the Filebeat configuration file filebeat.yml to point at the Kubernetes NodePort I’d exposed, that’s covered a littler later and I also moved the FileBeat log provided into the FileBeat application folder.</p><pre>filebeat.inputs:</pre><pre>- type: log</pre><pre>enabled: true</pre><pre>paths:</pre><pre>- logstash-tutorial.log</pre><pre>output.logstash:</pre><pre>hosts: [&quot;localhost:30102&quot;]</pre><p>Just Logstash and Kubernetes to configure now. Lets have a look at the pipeline configuration.</p><p>Every configuration file is split into 3 sections, <strong>input</strong>, <strong>filter</strong> and <strong>output</strong>. They’re the 3 stages of most if not all ETL processes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73ddc482bf620360ece67250da8abbc5/href">https://medium.com/media/73ddc482bf620360ece67250da8abbc5/href</a></iframe><p>We specify where our data is coming from firstly, in our case we are using the Beats plugin and specify the port to receive beats on.</p><p>So what’s this Beats plugin? It enables Logstash to receive events from applications in the <a href="https://www.elastic.co/products/beats">Elastic Beats framework</a>. As we are running FileBeat, which is in that framework, the log lines which FileBeats reads can be received and read by our Logstash pipeline.</p><p>Next we specify filters. The filter sections is optional, you don&#39;t have to apply any filter plugins if you don&#39;t want to. If that’s the case, data will be sent to Logstash and then sent on to the destination with no formatting, filtering etc. In our case we are using the Grok plugin. The Grok plugin is one of the more cooler plugins. It enables you to parse unstructured log data into something structured and queryable.</p><p>Grok is looking for patterns in the data it’s receiving, so we have to configure it to identify the patterns that interest us. Grok comes with some built in patterns. The pattern we are using in this case is %{COMBINEDAPACHELOG}which can be used when Logstash is receiving log data from Apache HTTP.</p><p>Lastly, we specify our outputs. This is where our data will end up once it has been filtered. You can specify more than one output if your data needs to go to multiple places. In the example the data is being output to ElasticSearch but also printed to the console, just to be on the safe side. Inside the ElasticSearch block we specify the ElasticSearch cluster URL and the Index name which is a String made up of a pattern made up of metadata.</p><p>Now that we’ve walked through the config of our pipeline we can move onto Kubernetes.</p><p>What we have to do first of all is create a ConfigMap. A ConfigMap allows us to store key-value pairs of configuration data that is accessible by our Pods. So we could have a ConfigMap that would store a directory full of configuration files of it could store a single configuration file.</p><p>First we create a ConfigMap. We are naming it<strong> apache-log-pipeline</strong> and referencing the pipeline configuration file from earlier.</p><pre>&gt; kubectl create configmap apache-log-pipeline --from-file apache-log-es.conf</pre><p>We can examine the ConfigMap we’ve created by running kubectl with describe.</p><pre>&gt; kubectl describe cm/apache-log-pipeline</pre><p>If the command has been run correctly, you should see the key of apache-log-pipeline and the value of the configuration file from earlier. If that’s the case you’re doing great!</p><pre>Name:         apache-log-pipeline<br>Namespace:    default<br>Labels:       &lt;none&gt;<br>Annotations:  &lt;none&gt;</pre><pre>Data<br>====<br>apache-log-es.conf:<br>----<br>input {<br>    beats {<br>        port =&gt; &quot;5044&quot;<br>    }<br>}<br>filter {<br>    grok {<br>        match =&gt; { &quot;message&quot; =&gt; &quot;%{COMBINEDAPACHELOG}&quot;}<br>    }<br>}<br>output {<br>    elasticsearch {<br>        hosts =&gt; [&quot;<a href="http://elasticsearch:9200/">http://elasticsearch:9200</a>&quot;]<br>        index =&gt; &quot;%{[<a href="http://twitter.com/metadata">@metadata</a>][beat]}-%{[<a href="http://twitter.com/metadata">@metadata</a>][version]}-%{+YYYY.MM.dd}&quot;<br>    }<br>    stdout {<br>        codec =&gt; rubydebug<br>      }<br>}</pre><pre>Events:  &lt;none&gt;</pre><p>So now we have our ConfigMap, we need to put together a Deployment for our Logstash service and reference the apache-log-pipeline as a mounted volume.</p><p>Lets walk through some parts of the Deployment. A part from the usual stuff we specify 2 ports for the container, 5044 and 9600. Port number 5044 is used to receive beats from the <a href="https://www.elastic.co/products/beats">Elastic Beats framework,</a> in our case FileBeat and port number 9600 allows us retrieve runtime metrics about Logstash. More information about that <a href="https://www.elastic.co/guide/en/logstash/current/monitoring.html">here</a>.</p><p>We also specify that we want to the mount the config volume and which path we want to mount it to, /usr/share/logstash/pipeline. We mount the volume into this particular directory because this is the directory that Logstash reads configurations from by default. This allows us to to just run logstash as the command, as opposed to specifying a flag of where the configuration file is.</p><p>We then have our volume which is called apache-log-pipeline-config and it’s a type of configMap. For more information on Volumes have a look <a href="https://kubernetes.io/docs/concepts/storage/volumes/#configmap">here</a>. We specify the ConfigMap we wish to use, apache-log-pipeline is the one we created earlier. As our ConfigMap is made up of key-value pairs, we add the key which contains our pipeline configuration, apache-log-es.conf.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2828a05ca6f4669b6e2dd5b6934a8ac9/href">https://medium.com/media/2828a05ca6f4669b6e2dd5b6934a8ac9/href</a></iframe><p>We then go on to the Service. The only real talking point here is that we are using a NodePort. This is for two reasons, FileBeat needs to speak to Logstash which is running in Kubernetes so we need a port for this to be done on, I’ve specified this to be 30102 as the filebeat.yml needs configuring with this port number into order to send beats to Logstash. Second reason, I wanted to check out the Logstash Monitoring API which uses port 9600 as mentioned earlier.</p><p>So with our deployment and service ready to rock and roll we can deploy it.</p><pre>&gt; kubectl create -f apache-log-pipeline.yaml</pre><p>If the Pod has been created correctly, you should be able to get Pods and see it running.</p><pre>&gt; kubectl get pods</pre><pre>================================================</pre><pre>NAME                                   READY   STATUS    RESTARTS   AGE<br>apache-log-pipeline-5cbbc5b879-kbkmb   1/1     Running   0          56s<br>================================================</pre><p>The Pod has been created correctly, but is it actually up and running? The quickest way to tell is by tailing the logs of the Pod.</p><pre>&gt; k logs -f pod/apache-log-pipeline-5cbbc5b879-kbkmb</pre><p>If the pipeline is running correctly the last log line you should see says that the Logstash API has been created successfully.</p><pre>[2019-01-20T11:12:03,409][INFO ][logstash.agent] Successfully started Logstash API endpoint {:port=&gt;9600}</pre><p>When I initially built this pipeline I came across two errors. The first was when the configuration file wasn’t in a readable format due to the formatting, the error was printed and visible when tailing the Pod logs. The second was when ConfigMap wasn’t mounted correctly and the pipeline would run, stop then restart, again this was printed and visible by tailing the logs.</p><p>So, we have a fully functioning Logstash pipeline running in Kubernetes.</p><p>But its not actually doing anything. What we need to do now is run Filebeat. To make sure this worked correctly I had two Terminal windows open, one tailing the logs of the Pod and the other for the FileBeat command I was about to run.</p><pre>sudo ./filebeat -e -c filebeat.yml -d &quot;publish&quot; -strict.perms=false</pre><p>When this command is run, Filebeat will come to life and read the log file specified in in the filebeat.yml configuration file. The other flags are talked about in the tutorial mentioned at the beginning at the article.</p><p>In the pipeline configuration file we included the stdout plugin so messages received are printed to the console. With this in mind, we should see messages printed out in the Terminal window tailing the Pods logs and you should see something similar in the window running the Filebeat command.</p><pre>{<br>     &quot;<a href="http://twitter.com/timestamp">@timestamp</a>&quot; =&gt; 2019-01-20T11:35:36.042Z,<br>        &quot;request&quot; =&gt; &quot;/style2.css&quot;,<br>     &quot;prospector&quot; =&gt; {<br>        &quot;type&quot; =&gt; &quot;log&quot;<br>    },<br>       &quot;response&quot; =&gt; &quot;200&quot;,<br>    &quot;httpversion&quot; =&gt; &quot;1.1&quot;,<br>         &quot;offset&quot; =&gt; 18005,<br>          &quot;bytes&quot; =&gt; &quot;4877&quot;,<br>           &quot;tags&quot; =&gt; [<br>        [0] &quot;beats_input_codec_plain_applied&quot;<br>    ],<br>      &quot;timestamp&quot; =&gt; &quot;04/Jan/2015:05:24:57 +0000&quot;,<br>       &quot;clientip&quot; =&gt; &quot;81.220.24.207&quot;,<br>       &quot;referrer&quot; =&gt; &quot;\&quot;<a href="http://www.semicomplete.com/blog/geekery/ssl-latency.html/">http://www.semicomplete.com/blog/geekery/ssl-latency.html\</a>&quot;&quot;,<br>        &quot;message&quot; =&gt; &quot;81.220.24.207 - - [04/Jan/2015:05:24:57 +0000] \&quot;GET /style2.css HTTP/1.1\&quot; 200 4877 \&quot;<a href="http://www.semicomplete.com/blog/geekery/ssl-latency.html/">http://www.semicomplete.com/blog/geekery/ssl-latency.html\</a>&quot; \&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11\&quot;&quot;,<br>          &quot;ident&quot; =&gt; &quot;-&quot;,<br>          &quot;agent&quot; =&gt; &quot;\&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11\&quot;&quot;,<br>           &quot;beat&quot; =&gt; {<br>         &quot;version&quot; =&gt; &quot;6.5.4&quot;,<br>            &quot;name&quot; =&gt; &quot;local&quot;,<br>        &quot;hostname&quot; =&gt; &quot;local&quot;<br>    },<br>           &quot;auth&quot; =&gt; &quot;-&quot;,<br>       &quot;<a href="http://twitter.com/version">@version</a>&quot; =&gt; &quot;1&quot;,<br>           &quot;host&quot; =&gt; {<br>        &quot;name&quot; =&gt; &quot;local&quot;<br>    },<br>           &quot;verb&quot; =&gt; &quot;GET&quot;,<br>          &quot;input&quot; =&gt; {<br>        &quot;type&quot; =&gt; &quot;log&quot;<br>    },<br>         &quot;source&quot; =&gt; &quot;/filebeat-6.5.4-darwin-x86_64/logstash-tutorial.log&quot;<br>}</pre><p>If we’ve seen the messages printed in the console we can almost guarantee that the message have been delivered into ElasticSearch. There are two ways to check this, call the ElasticSearch API with some parameters or use Kibana.</p><p>I’m a big fan of Kibana, so that’s the route we’re heading down.</p><p>Fire up Kibana and head to the <a href="https://www.elastic.co/guide/en/kibana/current/discover.html">Discover section</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*aDsWB6fp1O4mYA1Wll-apg.png" /></figure><p>In our pipeline configuration, more specifically the ElasticSearch output, we specify the Index that’s to be created to be a pattern made up of metadata which includes the Filebeat version and the date. We use this index pattern to retrieve the data from ElasticSearch.</p><p>In this example the Index that I defined was called <strong>filebeat-6.5.4–2019.01.20 </strong>as this was the Index that was created by Logstash.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9VcBbVgtCiaohCUZZGW43g.png" /></figure><p>Next, we configure the Time Filter field. This field is used when we want to filter our data by time. Some logs will have multiple time fields so that’s why we have to specify it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NHau7WlJuEbMuZNQ7A-iTA.png" /></figure><p>Once these steps have been carried out we should be able to view the logs. If we go back into the Discover section once we have defined the Index the logs should be visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tng6W0h5KAg_tGjVBgI8_g.png" /></figure><p>If the logs are visible, give yourself a pat on the back! Well done and good effort! Have a celebratory dab if you want :)</p><p>So a couple of cool Kibana related things before I wrap up.</p><p>We can monitor our Logstash pipelines from the Monitoring section of Kibana. We can get insights into event rates such as emitted and received. We also get Node information such as CPU utilization and JVM metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9DUlRdMtjwH2fSHFkrbO3g.png" /></figure><p>Last but not least is the Logs section of Kibana. I think this is my favourite section of Kibana at the moment. It allows you to view streaming logs in near-real time and look back at historical logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xSVzaLeZXGlkH3vyDxyD-A.png" /></figure><p>To see the Logs section in action, head into the Filebeat directory and run sudo rm data/registry, this will reset the registry for our logs. Once this has been done we can start Filebeat up again.</p><pre>sudo ./filebeat -e -c filebeat.yml -d &quot;publish&quot; -strict.perms=false</pre><p>If you place the Terminal you’re running Filebeat in next to the browser you have Kibana in you’ll see the logs streaming in near-real time, cool eh?</p><p>As the title says, this is a basic article. It guides you on how to get something up and working quickly so there is bound to be improvements/changes that could be made to make this better on all fronts.</p><p>Many thanks as always for reading my articles, it’s really appreciated. Any thoughts, comments or questions drop me a tweet.</p><p>Cheers 👍🏻</p><p>Danny</p><p><a href="https://twitter.com/danieljameskay">https://twitter.com/danieljameskay</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94a470ad34d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-basics-of-deploying-logstash-pipelines-to-kubernetes-94a470ad34d9">The basics of deploying Logstash pipelines to Kubernetes</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Kafka & RabbitMQ Integration easier with Spring Cloud Stream : Part 1]]></title>
            <link>https://medium.com/@danieljameskay/making-kafka-rabbitmq-integration-easier-with-spring-cloud-stream-part-1-ddbb1c6bf283?source=rss-48cf87e0ed3a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddbb1c6bf283</guid>
            <category><![CDATA[kotlin]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[integration]]></category>
            <dc:creator><![CDATA[Danny Kay]]></dc:creator>
            <pubDate>Sun, 25 Nov 2018 20:45:39 GMT</pubDate>
            <atom:updated>2018-11-25T20:45:39.578Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DF1YCxTx-ggQD2dfpIERvw.jpeg" /></figure><p>Integrating messaging systems can be a pain.</p><p>There are so many things that have to be taken into account when systems of this kind need to be integrated with other systems.</p><p>Two of the main factors of system integration in my opinion are the conversion of messages between the system producing the message &amp; system consuming it i.e a customer model to a canonical model and how the two systems actually communicate with one another.</p><p>We could have a system that consumes an XML message and produces a message in JSON. We could have one that consumes AVRO and produces XML the list goes on.</p><p>I’ve been pondering in my head for a while now what would be the most pain free way and quickest way of consuming an Avro formatted message from a Kafka Topic, converting it into JSON and sending it to a RabbitMQ queue.</p><p>Interest in Apache Kafka is growing rapidly. More and more companies are utilising Kafka and the ecosystem it provides. Because of this interest, integration between Kafka and legacy systems usually happens at some point as companies move onto a newer system and off a legacy one.</p><p>A useful part of the Kafka ecosystem is Kafka Connect.</p><p>Kafka Connect is a framework for connecting databases, key-value stores and external systems to Kafka. It provides source connectors that have the ability to stream database changes to Kafka Topics. It also provides sink connectors that have the ability to stream records from Kafka Topics and store them in ElasticSearch, HDFS, S3, RabbitMQ and many more.</p><p>Example, we have a MySQL database and every time we add a row we want to send it to ElasticSearch. We would create a configuration JSON file that contains fields such as the MySQL database URL, Table Name and the name of the Topic where the data needs to be sent. Kafka Connect exposes a HTTP endpoint that this configuration can be posted to for it to be created. We would create another configuration file for this data to be retrieved from the Topic and inserted into ElasticSearch.</p><p>The only issue with this solution is that we are providing a Source and Sink for our data but we cant perform the transformation we need.</p><p>Another way is using Kafka Streams.</p><p>Kafka Streams is a client library for building applications and microservices where the input and output data are stored in Kafka Topics. So we could have an application that is streaming near-real time data into a Kafka Topic, our Kafka Streams application would consume messages, transform them and send them back to Kafka in a new Topic or straight to RabbitMQ. The one and only problem with this solution is that it involves a bit of leg work to get it up and running.</p><p>Spring Cloud Stream can help us out here.</p><p>Spring Cloud Stream is a framework that makes building event &amp; message driven microservices a walk in the park. It’s super simple to get up &amp; running and have events &amp; messages flowing between the two systems.</p><p>Spring Cloud Stream is made up of three parts. <em>Destination Binders</em>, <em>Destination Bindings</em> and <em>Messages</em>.</p><p>The <em>Destination Binders</em> provide the necessary configuration and implementation to facilitate integration with external messaging systems. So there would be a Destination Binder for Kafka, RabbitMQ, GCP PubSub etc.</p><p><em>Destination Bindings</em> bridge between the external messaging systems and application provided Producers and Consumers of messages. Think of the Destination Binders as the means of our application integrating with the messaging system and the Destination Bindings as the means of our application utilising the Destination Binder by being able to send and receive records to the messaging system.</p><p>Last but not least are <em>Messages</em>. Messages contain data relevant to a domain which are used by producers and consumers to communicate with Destination Binders.</p><p>If you’re reading this article I presume you understand the basics of Spring, Kotlin, RabbitMQ and Kafka. But, this article is pretty straight forward so people with basic knowledge will be just fine.</p><p>The first part of this article is going to focus on getting records out of Kafka. I use the <a href="https://github.com/Landoop/fast-data-dev">Landoop Fast Data Dev</a> docker image when I run Kafka locally. It comes with the full Kafka suite and is up and running in seconds. Here’s the Docker Compose file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60e881a462781ff4b6b2d3cec971f2a2/href">https://medium.com/media/60e881a462781ff4b6b2d3cec971f2a2/href</a></iframe><p>Let’s talk about the environment variables.</p><p>SAMPLEDATA: 1 creates a handful of Topics with some realistic data inside, RUNNING_SAMPLEDATA: 1 ensures that when we consume from these Topics there will be a steady stream of records returned and RUNTESTS: 0disables the integration tests from running for a quicker start up.</p><p>If everything is running as it should be you will see an output similar to the below.</p><pre>kafka-cluster_1  | 2018-11-24 11:44:00,795 INFO success: broker entered RUNNING state, process has stayed up for &gt; than 1 seconds (startsecs)<br>kafka-cluster_1  | 2018-11-24 11:44:00,795 INFO success: connect-distributed entered RUNNING state, process has stayed up for &gt; than 1 seconds (startsecs)<br>kafka-cluster_1  | 2018-11-24 11:44:00,797 INFO success: logs-to-kafka entered RUNNING state, process has stayed up for &gt; than 1 seconds (startsecs)</pre><p>To make sure Kafka is up and running we can consume some records from one of the test Topics. I’d recommend downloading <a href="https://github.com/edenhill/kafkacat">KafkaCat</a> to carry this out. Running the below command returns the metadata associated with the specified broker.</p><pre>kafkacat -L -b localhost:9092</pre><p>The part we’re interested in is a small paragraph with the Topic name “backblaze_smart” and some partition information.</p><pre>topic “backblaze_smart” with 2 partitions:<br> partition 0, leader 0, replicas: 0, isrs: 0<br> partition 1, leader 0, replicas: 0, isrs: 0</pre><p>If you’re seeing this everything has worked as it should have so far, give yourself a pat on the back…give yourself a bigger pat on the back if you see a steady stream of records when running the below command.</p><pre>kafkacat -b localhost:9092 -t backblaze_smart</pre><p>The Topic named backblaze_smart contains log entries, which I think are from a hard drive array or something of that sort. Either way, it contains log entries which are strings and perfect for our solution.</p><p>So now that we have Kafka up and running and a Topic ready to stream records from we need a project to create a Consumer from. The best place to head to is <a href="https://start.spring.io/">https://start.spring.io/</a>. Using Spring Initializer we can have a project up and running in no time. We need two dependencies, Cloud Stream and Kafka. Kafka is the Destination Binder that we are using in this part but we’ll be using a couple of the others in the future parts.</p><p>Now that we have a project to work with we need to write a Consumer. So the Consumer in this example is written in Kotlin and in future parts it will be written in Kotlin but it can be transposed to Java easily enough.</p><p>Right, so this block of code is enough to stream records from a Topic. The only other file that we need (at the current moment) is a configuration file with the name of the Topic that we want to consume from inside. So lets walk through it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46403e12263428150570d30432683876/href">https://medium.com/media/46403e12263428150570d30432683876/href</a></iframe><p>The first annotation to talk about is @EnableBinding.</p><p>This annotation is responsible for providing connectivity to the Destination Binder in the project, which in our case is Kafka. It also takes one or more Interfaces as an argument. Spring Cloud provides 3 interfaces for us, they are Sink, Source and Processor. If these Interfaces aren’t sufficient for your needs you can roll your own.</p><p>As we’re using the Sink Interface here that’s the one I’m going to be going into in a bit more detail about. I’ll cover the others in future parts.</p><p>If we have a look inside the Sink Interface it will make more sense how things work.</p><pre>public interface Sink {<br><br>   String <em>INPUT </em>= &quot;input&quot;;<br><br>   @Input(Sink.<em>INPUT</em>)<br>   SubscribableChannel input();<br><br>}</pre><p>The @Input annotation identifies an SubscribableChannel, this is how the records enter our application.</p><p>The next annotation in our BasicKafkaSink file is @StreamListener.</p><p>We add this to a function that we want to receive events from a middleware for processing. It takes an argument which is the value of the binding target, which in our case is Sink.INPUT that has a value of input. This constant refers to the middleware where our stream of records are coming from. We’ll use the value of this constant later to set our Topic value. But, if for some reason we swapped out Kafka for RabbitMQ, the only changes we would have to make are configuration ones, which is pretty neat! The function itself is extremely basic at the moment but it takes the log as a String and just prints it to the stdout.</p><p>Lastly is the configuration file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8149b8aa0e4a09ecea91d5b99f2665c7/href">https://medium.com/media/8149b8aa0e4a09ecea91d5b99f2665c7/href</a></iframe><p>There’s no mistake in that gist!</p><p>All we have to do is specify the Topic that we want to stream the records from. Spring is very opinionated, so a handful of default values are set behind the scenes.</p><p>As this project grows we’ll see more values enter the configuration file.</p><p>Take a look at the property key, spring.cloud.stream.bindings.<strong>input</strong>.destination, see how the property key contains input?</p><p>The value of the constant in the interface…the value of the @Input annotation on the SubscribableChannel…lightbulb moment?!?!</p><p>So say for example we rolled our own Interface and we set the value of the INPUT constant to “kafka”, we would refer to “kafka” in our configuration file.</p><pre>spring.cloud.stream.bindings.<strong>kafka</strong>.destination=backblaze_smart</pre><p>And thats how it’s all wired up. This is what makes Spring Cloud Stream so cool!</p><p>With the help of some configuration and a couple of annotations we have a tiny service which is hooked up to some middleware, listening to a Topic and streaming records from it.</p><p>If everything has worked correctly, you should see some output like this, and lots of it.</p><pre>{“date”:”2017–01–01&quot;,”serial_number”:”S300Z4N3&quot;,”model”:”ST4000DM000&quot;,”capacity_bytes”:4000787030016,”failure”:0,”smart_1_normalized”:112,”smart_1_raw”:47623472,”smart_2_normalized”:null,”smart_2_raw”:null,”smart_3_normalized”:98,”smart_3_raw”:0,”smart_4_normalized”:100,”smart_4_raw”:2,”smart_5_normalized”:100,”smart_5_raw”:0,”smart_7_normalized”:87,”smart_7_raw”:489605003,”smart_8_normalized”:null,”smart_8_raw”:null,”smart_9_normalized”:86,”smart_9_raw”:12802,”smart_10_normalized”:100,”smart_10_raw”:0,”smart_11_normalized”:null,”smart_11_raw”:null,”smart_12_normalized”:100,”smart_12_raw”:2,”smart_13_normalized”:null,”smart_13_raw”:null,”smart_15_normalized”:null,”smart_15_raw”:null,”smart_22_normalized”:null,”smart_22_raw”:null,”smart_183_normalized”:100,”smart_183_raw”:0,”smart_184_normalized”:100,”smart_184_raw”:0,”smart_187_normalized”:100,”smart_187_raw”:0,”smart_188_normalized”:100,”smart_188_raw”:0,”smart_189_normalized”:100,”smart_189_raw”:0,”smart_190_normalized”:74,”smart_190_raw”:26,”smart_191_normalized”:100,”smart_191_raw”:0,”smart_192_normalized”:100,”smart_192_raw”:0,”smart_193_normalized”:77,”smart_193_raw”:47475,”smart_194_normalized”:26,”smart_194_raw”:26,”smart_195_normalized”:null,”smart_195_raw”:null,”smart_196_normalized”:null,”smart_196_raw”:null,”smart_197_normalized”:100,”smart_197_raw”:0,”smart_198_normalized”:100,”smart_198_raw”:0,”smart_199_normalized”:200,”smart_199_raw”:0,”smart_200_normalized”:null,”smart_200_raw”:null,”smart_201_normalized”:null,”smart_201_raw”:null,”smart_220_normalized”:null,”smart_220_raw”:null,”smart_222_normalized”:null,”smart_222_raw”:null,”smart_223_normalized”:null,”smart_223_raw”:null,”smart_224_normalized”:null,”smart_224_raw”:null,”smart_225_normalized”:null,”smart_225_raw”:null,”smart_226_normalized”:null,”smart_226_raw”:null,”smart_240_normalized”:100,”smart_240_raw”:12457,”smart_241_normalized”:100,”smart_241_raw”:37459596632,”smart_242_normalized”:100,”smart_242_raw”:13999128574,”smart_250_normalized”:null,”smart_250_raw”:null,”smart_251_normalized”:null,”smart_251_raw”:null,”smart_252_normalized”:null,”smart_252_raw”:null,”smart_254_normalized”:null,”smart_254_raw”:null,”smart_255_normalized”:null,”smart_255_raw”:null}</pre><p>I’ve ran this code on a MacBook Pro and a Mac Mini Server and it worked with no issues so if it doesn&#39;t for you drop me a comment or a tweet and i’ll get back to you.</p><p>This is the first part of many in this Spring Cloud Stream series and I’m extremely excited for whats coming up.</p><p>If you want so see the project for this part its on Github.</p><p><a href="https://github.com/danieljameskay/SpringCloudStreamKafkaExample">danieljameskay/SpringCloudStreamKafkaExample</a></p><p>Many thanks as always for reading my articles, it’s really appreciated. Any thoughts, comments or questions drop me a tweet.</p><p>Cheers 👍🏻</p><p>Danny</p><p><a href="https://twitter.com/danieljameskay">https://twitter.com/danieljameskay</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddbb1c6bf283" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transforming Messages: Content Enrichment]]></title>
            <link>https://medium.com/@danieljameskay/transforming-messages-content-enrichment-a00ae93a084a?source=rss-48cf87e0ed3a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a00ae93a084a</guid>
            <category><![CDATA[enterprise-architecture]]></category>
            <category><![CDATA[enterprise-software]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[messaging]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Danny Kay]]></dc:creator>
            <pubDate>Wed, 14 Nov 2018 16:19:03 GMT</pubDate>
            <atom:updated>2018-11-14T16:19:03.256Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_7t-TPkZQv9EgzdOodVZ3w.jpeg" /></figure><p>When I first became a Software Engineer I was advised to read “Enterprise Integration Patterns” by Martin Fowler as the team I was a part of maintained a large enterprise scale messaging solution and the Solutions Architect for the team thought that in reading this book it would help me understand the architecture of the system we looked after.</p><p>I’ve read a lot of books over the years but this one is one of them golden books that regardless of how old it is, parts of it are still relevant in the current day.</p><p>Cutting a long story short, I moved teams, I stopped reading the book and recently I came across a hard copy so I started reading it again. More specifically, the chapter on<strong> Content Enrichment</strong>. This is mainly because in my current role we’re interfacing with a system that utilises a Content Enricher and I wanted to fully understand the role it carried out and if this pattern could be used in future scenarios.</p><p>In a nutshell,<strong> Content Enrichment</strong> is the process of injecting additional information into a message.</p><p>Content Enrichment systems solve many problems including messages containing data that isn’t related to the domain the message is within, modifying systems upstream based off the needs of other systems and fragile integration solutions.</p><p>Say we have a checkout system and a fraud checking system that is the responsibility of a 3rd party. The fraud checking system checks transactions for fraudulent activity. In order for a check to be performed, the message that the fraud checking system consumes from the checkout system needs to contain customer information information otherwise a check cannot be performed and a purchase cannot be made.</p><p>But, the checkout system only produces messages that contain information such as a customer id, card number, expiry date, etc…so it doesn&#39;t contain customer information that the fraud checking system requires for security and fraud checking purposes.</p><p>This is where my new best friend Mr Content Enricher comes in. Its purpose is to supply data fields that the system the message is being produced from cannot provide. It can provide that data either by computation or retrieving data from an API, an external service or a Database.</p><p>If we are enriching a message through the use of an external service or API there are two ways we can carry this out. We can either send the full message to the API or we can send a single reference of some sort such as an ID. Once the service has the enriched data it is then appended to the original message we intend to send to the consuming system. This consuming system can either be the system that requested the enrichment or the system that requires the enriched message.</p><p>In our scenario we would place the Content Enricher between the checkout system and fraud checking system. The Content Enricher would use a piece of data from the message to perform the enrichment. Sticking with our scenario, the message contains a Customer ID, we can use this to retrieve data from the Customer Database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*Um7MKbBiD2pS9mZfmsBz9w.png" /></figure><p>Once the Content Enricher has the Customer ID it will query the Database and have the customer information needed by the Fraud Checking System. Once this takes place the Content Enricher will be ready to return an enriched message.</p><p>There may be scenarios where the original message and contents can be discarded and just the enriched data is returned to the system that requested the enrichment or the system that is consuming it.</p><p>As mentioned earlier, we can provide data by computation. This is enriching a message with new data using data that the consumed message already contains. For example, if the receiving system requires the users full name, but the incoming message contains a first and last name field. We can combine these values in a new field in the enriched message.</p><pre>{<br>  &quot;firstName&quot;: &quot;Danny&quot;,<br>  &quot;lastName&quot;: &quot;Kay&quot;,<br>  &quot;enrichedData&quot;: {<br>    &quot;fullName&quot;: &quot;Danny Kay&quot;<br>  }<br>}</pre><p>In scenarios I’ve encountered, the enriched data comes from a Database and is exposed through an API or Message Endpoint.</p><p>One of the problems with content enrichment is that the system producing the enriched message cannot send the message back until it has been enriched, i.e it’s synchronous. Also, the interface that receives the request and database that contains the data would need to be highly available otherwise messages cannot be enriched.</p><p>If anyone wants any more information on message transformation or enterprise messaging architecture hit me up on Twitter!</p><p>Cheers,</p><p>Danny</p><p><a href="https://twitter.com/danieljameskay">https://twitter.com/danieljameskay</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a00ae93a084a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Landoop Elastic 6 Sink Connector — The Magic Byte Error and the “schemas.enable=false” Error]]></title>
            <link>https://medium.com/@danieljameskay/landoop-elastic-6-sink-connector-the-magic-byte-error-and-the-schemas-enable-false-error-e8d77ca174be?source=rss-48cf87e0ed3a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8d77ca174be</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[json]]></category>
            <dc:creator><![CDATA[Danny Kay]]></dc:creator>
            <pubDate>Sat, 07 Jul 2018 19:08:14 GMT</pubDate>
            <atom:updated>2018-07-07T19:08:14.807Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nl4t_N1_Fc4sPTOHFIsc7A.jpeg" /></figure><p>Sometimes…you just want to dump a big ole payload of JSON in a database, am I right?</p><p>This was me a few weeks ago. I’d started to build this really cool demo for a meetup I attend and came across an issue when trying to use the ElasticSearch6 Connector for Kafka.</p><p>Now in most scenarios I would have adopted the approach of creating an Avro schema and using Schema Registry but time was of the essence here.</p><p>A slight bit of context for anyone whose curious, I’m running the Landoop “Fast Data Dev” Docker image which can be found <a href="https://github.com/Landoop/fast-data-dev">here</a>. The great thing about this image is that it contains the full Kafka suite and a load of Landoop goodies which includes UI’s, Connectors and a handful of Topics full of messages.</p><p>One of these connectors is the ElasticSearch 6 Sink Connector which can be found <a href="https://www.landoop.com/docs/lenses/connectors/sink/elastic6.html">here</a>. This Connector has one purpose, to stream events from Kafka to ElasticSearch…and that&#39;s exactly what I needed.</p><p>So I headed over to the Kafka Connect UI and started to create a new instance. I was presented with the template that just needed a few slight amendments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*NUOui7KMuyDodJQmlZ2XgQ.png" /></figure><p>I completed the template with some help from the above link to the Landoop site as the template is missing a couple of important properties.</p><pre>{<br> “connector.class”: “com.datamountaineer.streamreactor.connect.elastic6.ElasticSinkConnector”,<br> “tasks.max”: “1”,<br> “topics”: “nyc_driver_loc_data”,<br> “connect.elastic.url”: “A_URL”,<br> “connect.elastic.kcql”: “INSERT INTO index_1 SELECT * FROM nyc_driver_loc_data”,<br> “connect.elastic.cluster.name”: “A_CLUSTER”,<br> “connect.progress.enabled”: “true”<br>}</pre><p>Nothing really special going on, I’ve just specified the name of the topic that contains the juicy JSON and the KCQL command to run which is basically taking all the messages from the topic and inserting it into an Index named index_1.</p><p>Once I’d hit that create button I was presented with the first error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*GPlSX93lCvqwW_wpM31ImQ.png" /></figure><p>As soon as I saw this message I knew what I had missed from the configuration. This…</p><pre>value.converter=org.apache.kafka.connect.json.JsonConverter</pre><p>This errors a bit of a tricky one but in a nutshell the Avro Converter is trying to convert the message and this is done by default if a specific converter isn&#39;t added to the configuration…but we’re not using Avro, so I added the above to the configuration and did a restart and then got another message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*nnl1pJpNUmzXPxbju-ReZw.png" /></figure><p>Now this one was a bit easier to understand. So what we need to do is override the default schema as our JSON doesn’t include a schema or payload field which the toConnectorData() method is looking for. So as the message says, we need to set schemas.enable to false as shown below.</p><pre>value.converter.schemas.enable=false</pre><p>After I changed and saved this last setting data started flowing from my Kafka Topic into ElasticSearch and it looked pretty good.</p><p>I hope this helps anyone who comes across this issue and needs some assistance.</p><p>If anyone wants any more information on Kafka, Kafka-Connect or ElasticSearch drop me comment or hit me up on Twitter.</p><p>Cheers,</p><p>Danny</p><p><a href="https://twitter.com/danieljameskay">https://twitter.com/danieljameskay</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8d77ca174be" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>