<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Suhothayan Sriskandarajah on Medium]]></title>
        <description><![CDATA[Stories by Suhothayan Sriskandarajah on Medium]]></description>
        <link>https://medium.com/@suhothayan?source=rss-256afc94c5d3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*nffybkukWC6cTjR9.</url>
            <title>Stories by Suhothayan Sriskandarajah on Medium</title>
            <link>https://medium.com/@suhothayan?source=rss-256afc94c5d3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:12:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@suhothayan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What’s New in Siddhi CNSP 5.1.0?]]></title>
            <link>https://medium.com/siddhi-io/whats-new-in-siddhi-cnsp-5-1-0-331b7e185d24?source=rss-256afc94c5d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/331b7e185d24</guid>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[siddhi]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Suhothayan Sriskandarajah]]></dc:creator>
            <pubDate>Thu, 17 Oct 2019 05:34:18 GMT</pubDate>
            <atom:updated>2019-10-17T05:34:18.210Z</atom:updated>
            <content:encoded><![CDATA[<p>After a continuous effort for the past three months and sleepless nights, finally, we have released Siddhi Cloud-Native Stream Processor 5.1.0 on the 8th of October 2019. It is not a simple task; it was a marathon effort to release the Siddhi Stream Processor on time.</p><p>Before going into much detail about the release, we would like to share some words about the journey of Siddhi. As many of you already know, the Siddhi engine (library) is already being used in various open-source and proprietary frameworks, and tools such as Apache Flink, Apache Eagle, Uber, and Punch platform. Especially, Siddhi has been a core component of various WSO2 products such as WSO2 Complex Event Processor, WSO2 Data Analytics Server, WSO2 Stream Processor, WSO2 API Manager (for throttling purposes) and WSO2 Identity Server (for adaptive authentication); and served a lot of customers and their needs. All these years, Siddhi has been pitched as an on-premise Complex Event Processing and Stream Processing engine.</p><p>As you are aware, nowadays a lot of companies evaluating and revisiting their enterprise architecture and move applications to the cloud. As the initial step, they have started converting their services and apps into microservices. Since stream processing and event-driven data processing are some of the key components in the enterprise architecture; we were able to see them struggling to port legacy big data solutions to the cloud, or trying to build microservices using on-premise solutions such as Siddhi library, in order to fulfill the serious gap for a stream processing systems in the Cloud.</p><p>For a stream processor to play in the cloud-native space it should have characteristics such as being lightweight, loosely coupled and adhering to the agile dev-ops process. However, most of the traditional stream processors are heavy and depend on bulky monolithic technologies which makes them harder moving to the cloud. Therefore, with the feedback of the users, we have implemented Siddhi Stream Processor (<a href="https://siddhi.io/">https://siddhi.io</a>) as a 100% open source stream processor under Apache License v2, to solve the event-driven data processing and stream processing use cases natively in the Cloud. Now users can use Siddhi to collect, inject, process, analyze, integrate (with services and databases), and send messages to other systems, all in one tool both natively on cloud and on-premise.</p><p>In the following sections, we will highlight some of the key features of Siddhi Stream Processor 5.1.0 and provide necessary details on those.</p><h3>Native support to run Siddhi Streaming Apps in K8s</h3><p>One of the major features that we have embedded in this release is to natively deploy and run Siddhi apps in Kubernetes. To fulfill the native Kubernetes support we have developed a Kubernetes <a href="https://github.com/siddhi-io/siddhi-operator">operator for Siddhi</a>. The Siddhi operator has the responsibility to deploy, run, and reconcile Siddhi app deployments in a Kubernetes cluster. The users can now deploy Siddhi apps directly in a Kubernetes cluster using the Siddhi operator <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Custom Resource Definition(CRD)</a> namely SiddhiProcess by simply using kubectl commands. Siddhi supports two types of deployments via the Siddhi operator, and they are :</p><ol><li>Simple Deployment</li><li>Distributed Deployment</li></ol><p>The simple deployment deploys selected Siddhi Applications in a single pod in Kubernetes, whereas the distributed deployment breaks the Siddhi App into stateless and stateful parts, integrate them using NATS (and NATS streaming) messaging system, and deploy them in two different pods as shown below, in such a way they can automatically achieve high availability and zero event loss. Find out more details about the Siddhi Kubernetes deployments from the <a href="https://siddhi.io/en/v5.1/docs/siddhi-as-a-kubernetes-microservice/#deploy-and-run-siddhi-app-in-distributed-mode">official documentation.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t2sS1jjEKpsElYGO" /><figcaption>Distributed Siddhi deployment in K8s supporting High Availability</figcaption></figure><p>You can also try out the deployment in <a href="https://www.katacoda.com/siddhi/courses/siddhi-deployment">Katacoda</a>.</p><h3>Tooling support to generate Docker and K8s artifacts</h3><p>Previously, to deploy Siddhi apps in Docker or Kubernetes environments, the users are required to write Docker files and Kubernetes YAML files on their own. Writing these files required a reasonable amount of knowledge and experience in Docker and Kubernetes. At times, users have to write the same type of files again and again, because all those files follow common standards, which makes the process time-consuming and error-prone. This release introduces tooling support to automate the generation of Docker and Kubernetes artifacts for the Siddhi apps.</p><p>Now, users can implement and test Siddhi apps using the Siddhi tooling editor and then generate Docker and K8s artifacts with few button clicks. Apart from downloading the artifacts, the tooling editor also supports to building and pushing of Docker images directly to the docker registry, making the overall Siddhi app development lifecycle more convenient and aligning with the CI/CD process.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fe7xo2pO0DXg%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3De7xo2pO0DXg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fe7xo2pO0DXg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bbad7574c456c2a90894e25188562e15/href">https://medium.com/media/bbad7574c456c2a90894e25188562e15/href</a></iframe><h3>Proven CI/CD reference implementation for Siddhi</h3><p>CI/CD has become an essential part of every organization&#39;s effective software development and delivery process. It contains two separate but complementary parts that enable users to continuously integrate the Siddhi Applications using automated testing, and continuously deliver the apps by verifying that they are ready for production. With Siddhi 5.1.0 we support the entire CI/CD pipeline from the development to the deployment, as shown in the reference implementation below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aYdmn2KdGp60PnTC" /><figcaption>CI/CD reference implementation for Siddhi</figcaption></figure><p>With the new Siddhi test framework, users can now create Sandbox Siddhi runtimes and perform unit tests independently on Siddhi Apps without connecting to any external systems, then they can perform integration tests on those Siddhi Apps by integrating with external systems running as Docker containers, and finally perform the same integration tests against an actual deployment environment as black-box tests to verify the deployment before moving into production. These streamline CI/CD and enable users to move to production much faster. Refer to the blog <a href="https://medium.com/siddhi-io/building-an-efficient-ci-cd-pipeline-for-siddhi-c33150721b5d">“Building an efficient CI/CD pipeline for Siddhi”</a> for detailed information on using the Siddhi test framework.</p><h3>Caching and query optimizations when integrating with external databases</h3><p>Database integration is an integral part of any stream processing use case as it helps to enrich event data. However, traditional databases can be slow, and they can introduce high latency to the event processing pipeline. With this release, Siddhi introduces <strong>caching support for all external databases</strong> to reduce data retrieval latency via caching policies such as FIFO, LRU and LFU and preloading data when every possible. For more info refer to the blog on <a href="https://medium.com/siddhi-io/database-caching-for-event-processing-siddhi-5c02e125dff4">“In-memory Database Caching for Stream Processing — Siddhi”</a>.</p><p>Further, Siddhi has also been improved to<strong> reduce RDBMS database query latency up to 3 times by directly executing most of the execution in the database itself.</strong> For more information see refer the blog <a href="https://medium.com/siddhi-io/siddhi-5-1-1-reduces-database-query-latency-by-60-36632d507374">“Siddhi 5.1 reduces Database Query Latency up to 3 times”</a>.</p><h3>gRPC Connector</h3><p>gRPC is a modern high-performance RPC framework. Due to its high performance, it has become widely used in cloud environments. In this release, we have released an <a href="https://siddhi-io.github.io/siddhi-io-grpc/">IO extension for Siddhi to perform gRPC</a> communications, and a <a href="https://siddhi-io.github.io/siddhi-map-protobuf/">mapper to support any Protobuf messages</a> via gRPC. Here, the gRPC IO extension supports the following features.</p><ol><li><strong>gRPC sink</strong><br>gRPC sink enables to publish events to a gRPC server. It uses the fire-and-forget approach to send events. In other words, it just sends events in a streaming manner and does not wait for a response.</li><li><strong>gRPC call sink &amp; gRPC call-response source</strong><br>gRPC call sink can be used to send events to a gRPC server, and here gRPC call response source will be used to receive responses from the server. Note that these two cannot be used in isolation.</li><li><strong>gRPC source</strong><br>This gRPC source listens to gRPC requests that are sent by gRPC clients in a streaming manner and passes them to Siddhi for processing. Note, here the clients do not expect any response from the server.</li><li><strong>gRPC service source &amp; gRPC service response sink</strong><br>gRPC service source listens to gRPC requests from clients, and synchronously sends responses using the gRPC service response sink to the clients. Note that these two cannot be used in isolation.</li></ol><p>In all these cases input and output messages can be formatted as JSON, TEXT, XML messages or mapped to a Protobuf message using respective mappers.</p><h3>Support for complex List &amp; Map operations</h3><p>List and Map data types are necessary elements for Siddhi Apps when performing complex message transformations. Therefore these functionalities are now supported using the <a href="https://siddhi-io.github.io/siddhi-execution-list">siddhi-execution-list</a> and <a href="https://siddhi-io.github.io/siddhi-execution-map">siddhi-execution-map</a> extensions. These provide useful functionalities to process list and map data structures, using functions such as <em>create()</em>,<em> isMap()/isList(), size()</em>, <em>getKeys(), getValues()</em>, etc. Furthermore, Siddhi also supports Scatter &amp; Gather mode of processing by splitting each item in the list using <em>tokenize()</em>, process them in isolation, and then combine them back using <em>collect()</em> as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ee2jQ-zaoCQ8sRyiGFPlZQ.png" /><figcaption>Scatter-gather data pipelining</figcaption></figure><h3>Pattern query improvements</h3><p>When counting <a href="https://siddhi.io/en/v5.1/docs/query-guide/#pattern">patterns</a> are used such as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*vrdxatyl4iqSZKTJK59Yzg.png" /><figcaption>Simple Pattern Query in Siddhi</figcaption></figure><p>Siddhi now allows retrieving all the events matched against the counting condition as a list. Since the list is not native to Siddhi, it is returned with type ‘object’, but this can be accessed and processed with the newly introduced <a href="https://siddhi-io.github.io/siddhi-execution-list">siddhi-execution-list</a> extension.</p><h3>Conclusion</h3><p>You can download the Siddhi 5.1.0 from <a href="https://siddhi.io/en/v5.1/download/">siddhi.io site</a> try out the <a href="https://siddhi.io/en/v5.1/docs/guides/alerts-for-thresholds/guide/">various use cases</a> which are newly introduced each covering a complete end to end use case in sections such as :</p><ul><li><a href="https://siddhi.io/en/v5.1/docs/guides/alerts-for-thresholds/guide/">Alerts Based on Thresholds</a></li><li><a href="https://siddhi.io/en/v5.1/docs/guides/fault-tolerance/guide/">Fault Tolerance &amp; Error Handling</a></li><li><a href="https://siddhi.io/en/v5.1/docs/guides/patterns-and-trends/guide/">Patterns &amp; Trends Over Time</a></li><li><a href="https://siddhi.io/en/v5.1/docs/guides/database-static-rule-processing/guide/">Static Rule Processing using Database Rules</a></li><li><a href="https://siddhi.io/en/v5.1/docs/guides/integrate-various-enterprise-systems/guide/">Integrate Various Enterprise Systems</a></li><li><a href="https://siddhi.io/en/v5.1/docs/guides/realtime-movie-recommendation/guide/">Realtime predictions with pre-trained ML models</a></li></ul><p>For more information about writing Siddhi queries refer to <a href="https://siddhi.io/en/v5.1/docs/query-guide/">Siddhi Query Guide</a> and refer the <a href="https://siddhi.io/en/v5.1/docs/">documentation</a> to deploy Siddhi in Java, Python, Docker, and Kubernetes.</p><p>For questions and support please reach out to the development team via <a href="https://siddhi.io/community/#community-collaboration">Mailing lists, StackOverflow or Slack</a>. You can also participate in the <a href="https://siddhi.io/community/#weekly-sync-up-meeting">weekly sync up meetings</a> and start collaborating and contributing to the project.</p><p>Thank You</p><p>Siddhi Team</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=331b7e185d24" width="1" height="1" alt=""><hr><p><a href="https://medium.com/siddhi-io/whats-new-in-siddhi-cnsp-5-1-0-331b7e185d24">What’s New in Siddhi CNSP 5.1.0?</a> was originally published in <a href="https://medium.com/siddhi-io">siddhi-io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating Event-Driven Systems with WSO2 Stream Processor and Siddhi]]></title>
            <link>https://medium.com/@suhothayan/integrating-event-driven-systems-with-wso2-stream-processor-and-siddhi-22fe2ef848fb?source=rss-256afc94c5d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/22fe2ef848fb</guid>
            <category><![CDATA[stream-processing]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[data-integration]]></category>
            <dc:creator><![CDATA[Suhothayan Sriskandarajah]]></dc:creator>
            <pubDate>Wed, 12 Dec 2018 12:45:02 GMT</pubDate>
            <atom:updated>2018-12-13T12:30:13.638Z</atom:updated>
            <content:encoded><![CDATA[<h3>Integrating Event-Driven Systems using WSO2 Stream Processor and Siddhi</h3><p>Enterprise systems communicate with each other using APIs, events, and data streams. APIs allow systems to communicate in a synchronous manner where they call other services to retrieve data or to perform tasks. At the same time, systems also asynchronously publish events or stream data to other systems, as a mean of informing other systems without expecting any response from the receiver. Notification, surveillance, and monitoring systems are some that fall into this category. These systems are built using the enterprise architecture, which is famously known as the <a href="https://en.wikipedia.org/wiki/Event-driven_architecture">event-driven architecture (EDA)</a>, where all the systems communicate with each other by passing messages in an asynchronous manner.</p><p>This post will be an insight into how we can efficiently integrate events and data streams to build an effective enterprise system based on event-driven architecture.</p><h3>Components of Event-Driven Architecture (EDA)</h3><p>Although the systems in the event-driven architecture can directly send messages to each other, in most of the cases, message passing is done, through a centralized message queue such as <a href="https://kafka.apache.org/">Kafka</a>, or a JMS broker like <a href="http://activemq.apache.org/">ActiveMQ</a>, <a href="https://www.ibm.com/products/mq">IBM MQ</a>, etc. These message queues become very handy as they provide guaranteed message delivery by allowing the consumer to be offline when the publisher publishes the message and lets you scale the system by publishing the message to one or more consumers.</p><p>To integrate systems using event-driven architecture only adding a message queue to your solution will not solve the problem. This is because the systems that consume and publish the message from the message queues should also be written in an efficient way to get the complete benefit of the overall solution. This is where the stream processors become very useful since they eliminate the need to write complex codes, to publish &amp; consume messages from event queues, process, transform and integrate them with various other systems.</p><h3>Stream Processors</h3><p>With lightweight and feature-rich stream processing systems you can:</p><ul><li>Collect events from various streaming sources, log files, and data stores using techniques like Change Data Capture (CDC).</li><li>Transform events to &amp; from JSON, XML, text and other formats.</li><li>Enrich data from databases &amp; services.</li><li>Perform filtering, aggregations and other complex event processing operations like pattern matching, and anomaly detection.</li><li>Do real-time prediction using machine learning.</li><li>Send events as notifications to external systems or load the data to various data stores.</li></ul><p>Unfortunately, all stream processing systems are not lightweight or have the capability to performable all the above-mentioned operations by default. Therefore, to demonstrate the necessary capabilities we chose <a href="https://wso2.com/analytics-and-stream-processing/">WSO2 Stream Processor</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zcnWENbv5QredgIkPSGzkw.png" /><figcaption>WSO2 Stream Processor.</figcaption></figure><p><a href="https://wso2.com/analytics-and-stream-processing/">WSO2 Stream Processor</a> is a cloud-native, lightweight, highly scalable system. It’s built using the high-performance <a href="http://siddhi.io/">Siddhi</a> streaming engine providing rich features for building stream processing logic. It also provides great developer experience with its graphical and Streaming SQL-based query editor allowing developers to build, test, debug, and simulate their use cases in the editor.</p><h3>Sample Event-Driven Data Integration Scenario</h3><p>To demonstrate how we can integrate event-driven systems, let’s take a notification management system as a sample use case. Here, suppose we have to send notifications to the buyers when the items are shipped from the store, acknowledging that their items are shipped, and keeping them informed on when to expect the delivery. Here, the shipment information will be consumed from a message queue/broker, the buyer details are enriched by querying a database and the expected delivery date is added by calling an external service, before sending the notification as an email.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*VI8Cqfeqsus9DxMh1PdmRw.png" /><figcaption>Sample integration scenario.</figcaption></figure><p>For this example, we will be using Kafka as the message broker. <a href="https://avro.apache.org/">Avro</a> as the message format used by Kafka, MySQL database to store buyer information, a mock Siddhi HTTP service to retrieve the expected delivery date, JSON as the message format to communicate with the HTTP service, and finally Gmail for email notifications.</p><h3>Implementation</h3><p>This can be easily implemented and visualized through the WSO2 Stream Processor Studio as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJYDm6KpxVFcCNp9lNcHKg.png" /><figcaption>Design view of the WSO2 Stream Processor Editor.</figcaption></figure><p>The 46 lines of Siddhi Streaming SQL query for building the sample notification scenario is given below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34c8333ac9d23e8a2dcf46e31d381b97/href">https://medium.com/media/34c8333ac9d23e8a2dcf46e31d381b97/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBpqI6p57Jo2RoR7VfFrVA.png" /><figcaption>Code view of the WSO2 Stream Processor Editor.</figcaption></figure><p>As shown in lines 4~9, Siddhi application consumes messages having Avro messaging format from a Kafka topic called ‘<em>shipment-info</em>’ and passes that to a stream called <em>‘ShipmentInfoStream’</em>. As of lines 42~46, this ‘ShipmentInfoStream’ joins with an RDBMS backed ‘<em>BuyerInfoTable</em>’ (defined in lines 38~40) to enrich the event stream by retrieving username, address, and email fields from the database. The concatenated result is then fed into ‘<em>ShipmentAndBuyerInfoStream</em>’ which converts the events to JSON messages and calls an HTTP service running on URL <a href="http://localhost:8080/delivery-info/">http://localhost:8080/delivery-info/</a> through the http-request sink given in lines 11~15. As this http-request sink and http-response source defined in lines 18~23 are correlated by the same sink.id, ‘<em>delivery-info</em>’, the JSON response coming from the HTTP service call is retrieved by the http-response source, converted into an event and pushed to ‘<em>DeliveryInfoResponseStream</em>’ along with the other event properties present while calling the HTTP service. Since log sink and email sink are registered to the ‘<em>DeliveryInfoResponseStream</em>’ they immediately get the message from the stream. Here the log sink logs the message to the console while the email sink defined in line 25~33 sends the constructed email payload to the buyer’s email address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*WVia0cQ8PZTeXIVoHcNt7g.png" /><figcaption>Email notification</figcaption></figure><p>In a nutshell, integrating event-driven systems and processing streaming data has now become seamless with graphical editing and a powerful streaming SQL language like <a href="http://siddhi.io/">Siddhi</a>. To implement the above scenario download the latest stable version of <a href="https://wso2.com/analytics-and-stream-processing/">WSO2 Stream Processor</a> and follow the step by step guide given <a href="https://github.com/suhothayan/wso2sp-demo/tree/master/SimpleNotificationManagement">here</a>. You can get more details about the WSO2 Stream Processor by visiting its <a href="https://wso2.com/analytics-and-stream-processing/features/">official website</a>. For further assistance refer to the <a href="https://docs.wso2.com/display/SP430">documentation</a> and the <a href="https://wso2.github.io/siddhi/documentation/siddhi-quckstart-4.0/">Siddhi Query Guide</a>.</p><p>Happy streaming integration!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22fe2ef848fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Perform Realtime and Periodic ETL with WSO2 Stream Processor]]></title>
            <link>https://medium.com/stream-processing/perform-realtime-and-periodic-etl-with-wso2-stream-processor-358680689935?source=rss-256afc94c5d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/358680689935</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[cdc]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-integration]]></category>
            <category><![CDATA[realtime]]></category>
            <dc:creator><![CDATA[Suhothayan Sriskandarajah]]></dc:creator>
            <pubDate>Sat, 21 Jul 2018 05:26:22 GMT</pubDate>
            <atom:updated>2019-04-02T13:19:50.668Z</atom:updated>
            <content:encoded><![CDATA[<p>Data integration is an integral part of an integrated enterprise. One of the key aspects of data integration is having the capability of loading or moving the data from one location to another location to be processed or stored for future reference. This process is commonly referred to as extract, transform, and load (<a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL</a>). This dates back to the time where we started using databases. In early days the ETL job is done as batch processing where we extract, transform, and load a huge amount of data in very sparse intervals for archiving or analysis purposes. But in the current day’s enterprises demand for the results instantly have propelled ETL functions to run in real-time. In this blog we’ll look at the advantages of real-time ETL functions, how both in real-time and periodically data can be extracted, how they can be transformed, and how they can be loaded to the final system using <a href="https://wso2.com/analytics/">WSO2 Stream Processor</a> (WSO2 SP) as an example.</p><p>Instead of updating the stakeholders once every day with the traditional ETL systems, real-time ETL functions let you keep your stakeholders and systems up to date using real-time data integration as and when the data is ready for extraction. Here, when the data is available it is immediately extracted, transformed according to your requirement and loaded to the system for processing. This is very useful when we have a data source that can push data or inform the systems as and when data is available for processing. But some data sources are passive where we need to periodically poll the system to check for new data or updates and perform ETL functions when they are available. Both of these approaches are commonly adopted in the modern enterprises. These ETL processes have now become the central part of the information backbone of most of the organizations.</p><h3><a href="https://wso2.com/analytics/">WSO2 Stream Processor</a> (WSO2 SP)</h3><p><a href="https://wso2.com/analytics/">WSO2 Stream Processor</a> (WSO2 SP) is an open source, and a cloud-native product optimized for processing data in real-time for agile digital businesses. It has the capability of processing <a href="https://docs.wso2.com/display/SP410/Performance+Analysis+Results">100k events per second</a> with just <a href="https://dzone.com/articles/is-your-stream-processor-obese">two nodes</a> and scaling more with its distributed deployment. As shown below it can consume events from various data sources, process the data in real-time, and produce instantaneous actionable insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fkchp9MlpD-RqF_CIpW65w.png" /><figcaption>Figure 1 : WSO2 Stream Processor Overview</figcaption></figure><p>In the following sections let’s see how we can achieve ETL functions using WSO2 Stream Processor.</p><h3>Data Extraction</h3><p>WSO2 SP has pre-built connectors to consume data from Kafka, JMS &amp; MQTT Message Brokers, RDBMS databases, MongoDB, Solr, Cassandra, HBase, ElasticSearch, REST APIs, HTTP, TCP, Email, File, and many <a href="https://docs.wso2.com/display/SP4xx/Collecting+Events">others</a>.</p><p>WSO2 SP use multiple techniques to extract data from the sources</p><ul><li><strong>Subscription:</strong> Systems like Kafka, JMS &amp; MQTT Message Brokers are push-based systems from which WSO2 SP consume messages by subscribing to them.</li><li><strong>Exposing Services:</strong> Some clients publish data using HTTP, or TCP transports in such cases a service will be exposed by WSO2 SP to consume the messages.</li><li><strong>Change data capture:</strong> For systems like RDBMS databases and files, DB logs and file system triggers are monitored to catch the changes to extract the relevant data.</li><li><strong>Polling:</strong> For other passive systems like MongoDB, Solr, Cassandra, HBase, ElasticSearch and REST APIs, periodic polling is triggered to extract data from the systems.</li></ul><h3>Data Transformation</h3><p>Since organizational systems are heterogeneous the data they produce also comes in multiple formats, the ETL systems should have the capability to transform the data to a format that they can use for storing or further processing. WSO2 SP has default support for transforming data that comes in JSON, XML, Text, WSO2 Event, CSV, and key-value formats, and allow users to provide a mapping to convert the incoming data to a format that they prefer. In WSO2 SP the collected data are mapped into one or more streams, which have a logical series of events having the same set of attributes or schema. When data is mapped into streams WSO2 SP allows you to write <a href="https://wso2.com/library/articles/2018/02/stream-processing-101-from-sql-to-streaming-sql-in-ten-minutes/">Streaming SQL</a> queries to manipulate the data, these manipulations can be very simple like replacing null values with a sensible default or it can be even running a machine learning or natural language processing operation.</p><p>WSO2 SP support following forms of data transformations.</p><ul><li>Data filtering and cleansing</li><li>Splitting and concatenation attributes in a single event or recode</li><li>Adding default values</li><li>Updating values upon a condition</li><li>Performing mathematical and logical functions based on the data</li><li>Enriching the data by correlating with other data streams</li><li>Enriching the data by correlating with stored data from RDBMS and NoSQL data sources</li><li>Enriching the data by calling external rest APIs</li><li>JSON transformations</li><li>Handling late and out-of-order data arrival</li><li>Correlating data based on arrival order using pattern and sequence queries</li><li>Aggregating events over time</li><li>Running machine learning models over the data</li><li>Performing Natural Language Processing on the data</li><li>Doing Geographical Processing with the data</li><li>… and <a href="https://docs.wso2.com/display/SP4xx/Developing+Streaming+Applications">many others</a></li></ul><p>With the above-mentioned techniques, the incoming data can be transformed and if needed further processed according to the business use cases. WSO2 SP provides a <a href="https://docs.wso2.com/display/SP4xx/Understanding+the+Development+Environment">drag-and-drop graphical editor</a> to build the transformation and processing pipeline, the same can also be done by scripting using the easy to use <a href="https://wso2.github.io/siddhi/documentation/siddhi-4.0/">Siddhi Streaming SQL</a> language. Since WSO2 SP uses streams as its intermediate data format the above operations will consume events from one or more streams, do the necessary transformation or processing, and produce new streams which can be then consumed by another operator for processing or loaded into the end system.</p><h3>Data Loading</h3><p>When it comes to loading data to the end systems, like data sources the data sinks also require the data to be in a message format that they prefer, and at times they only consume messages via a particular messaging protocol. WSO2 SP supports loading data to Kafka, JMS &amp; MQTT Message Brokers, RDBMS databases, MongoDB, Solr, Cassandra, HBase, ElasticSearch, HTTP REST services, TCP endpoints, Email &amp; Files systems, and to many others. It also by default supports pushing messages in JSON, XML, Text, WSO2 Event, CSV, and key-value formats, and the users can customize the messages before they load or send to the end systems. To achieve performance Stream Processor can be confirmed to load data in batches and using multiple threads.</p><p>WSO2 SP using the following techniques to load the data into external systems:</p><ul><li><strong>Data publishers:</strong> To publish data for pub/sub systems like Kafka, JMS &amp; MQTT Message Brokers</li><li><strong>Database clients:</strong> To write or modify RDBMS databases, MongoDB, Cassandra, HBase data stores</li><li><strong>HTTP and TCP clients:</strong> To send data to HTTP REST services, TCP endpoints, Solr, and systems like ElasticSearch</li><li><strong>Email or VFS clients:</strong> for send Email, or write to the files systems</li><li>… and <a href="https://docs.wso2.com/display/SP4xx/Publishing+Events">more</a></li></ul><h3>Guaranteed ETL with Zero Data Loss or Downtime</h3><p>When working with heterogeneous systems there is always an uncertainty of endpoints becoming unreliable and node going down. This can cause data loss and system downtimes. To mitigate endpoints unavailability scenarios WSO2 SP automatically retries to its endpoints in an exponentially decaying fashion until same or an alternative endpoint is available to load the data. Further, it can also be <a href="https://docs.wso2.com/display/SP4xx/Minimum+High+Availability+%28HA%29+Deployment">deployed in a highly available manner</a> such that SP node failure will not incur any data loss or system downtime. WSO2 SP employs a periodic incremental state persistence and an event syncing mechanism among its nodes, that allows it to achieve zero data loss or zero system downtime processing up to 100K events per second with just two of its nodes even when it was doing a stateful data transformation.</p><h3>Simple ETL Examples with WSO2 SP</h3><p>Let’s consider a couple of scenarios where we retrieve transaction data which need to be transformed and loaded into another system for processing.</p><h4>Scenario 1: Extract data from JMS, perform a stateless transformation, and load to a database.</h4><p>Let’s consider in the first scenario the data comes from a JMS topic in JSON format, which we need to clean the data by checking for null values and using defaults, enrich the data by joining with an RDBMS table and load the results to another RDBMS table.</p><p>Let’s assume the input data format will be like:</p><pre>{“userId”:“2345”, “transactionAmount”:456, “location”:”CA, USA”}</pre><p>Let’s say we need to transform this data to have userId, userName, transactionAmount, and locations fields. Where if the location is not provided in the incoming message we have to use the word “UNKNOWN”.</p><p>Finally, this data should be stored to a MySQL table, having columns userId, userName, transactionAmount, and transactionLocation.</p><p>A sample Siddhi Application for this scenario is listed here:</p><pre>@App:name(&quot;JMS-to-DB-ETL&quot;)</pre><pre>@App:description(&quot;Extract data from JMS, perform stateless transformation, and load to database&quot;)</pre><pre><em>-- </em><strong><em>Sample input message</em></strong><em>: {“userId”:2345, <br>--                        “transactionAmount”:456.0, <br>--                        “location”: &quot;CA, USA”}<br></em>@source(type = &#39;jms&#39; , destination = &#39;transactions&#39;, <br>   factory.initial = &#39;...&#39;, provider.url = &#39;...&#39;,<br>   @map(type = &#39;json&#39;,<br>      @attributes(&#39;$.userId&#39;, &#39;$.transactionAmount&#39;, &#39;$.location&#39;)))</pre><pre>define stream TrasanctionStream (userId long, <br>                        transactionAmount double, location string);</pre><pre>-- <strong>Table used to enrich data</strong><br>@store(type = &#39;rdbms&#39;, datasource = &#39;TRANSACTION_DATA_SOURCE&#39;)<br>define table UserTable (userId long, firstName string, <br>                        lastName string);</pre><pre>-- <strong>Final table to load the data</strong><br>@store(type = &#39;rdbms&#39; , datasource = &#39;TRANSACTION_DATA_SOURCE&#39;)<br>define stream TrasanctionTable (userId long, <br>                        transactionAmount double, location string);</pre><pre>@info(name = &#39;<strong>CleaningData</strong>&#39;)<br>from TrasanctionStream<br>select userId, <br>       transactionAmount, <br>       ifThenElse(location is null, &quot;UNKNOWN&quot;, location) as location<br>insert into CleanedTrasanctionStream;</pre><pre>@info(name = &#39;<strong>EnrichData</strong>&#39;)<br>from CleanedTrasanctionStream as c join UserTable as u<br>   on c.userId == u.userId<br>select c.userId, <br>       str:concat( u.firstName, &quot; &quot;, u.lastName) as userName, <br>       transactionAmount, <br>       location<br>insert into EnrichedTrasanctionStream;</pre><pre>@info(name = &#39;<strong>LoadData</strong>&#39;)<br>from EnrichedTrasanctionStream<br>insert into TransactionTable;</pre><p>The Siddhi Application’s graphical design view for this scenario in the Stream Processor Studio is represented in Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VjIU2AQl_i0hGLdj" /><figcaption>Figure 2: Design view of Scenario 1 in the WSO2 SP Studio</figcaption></figure><h4>Scenario 2: Extract data from a database, perform a stateful transformation, and load data to Kafka.</h4><p>Let’s consider we are going to poll the database every 5 min, performs aggregations on the data, and load that into Kafka. Let’s assume the input table will have the following columns: transactionId, userId, transactionAmount, and transactionLocation. Where we need to aggregate this data every 5 minutes to get total and average for each transactionLocation and load the result to Kafka in using default XML format supported by WSO2 SP.</p><p>A sample Siddhi Application for this scenario is presented here:</p><pre>@App:name(&#39;DB-to-Kafka-ETL&#39;)</pre><pre>@App:description(&#39;Extract data from database, perform stateful transformation, and load data to kafka&#39;)</pre><pre>define trigger TriggerStream at every 5 min;</pre><pre>-- <strong>Sink to load the data<br></strong>@sink(type=&#39;kafka&#39;, bootstrap.servers=&#39;...&#39;, <br>   topic=&#39;...&#39;,is.binary.message=&#39;...&#39;, <br>   @map(type=&#39;xml&#39;))<br>define stream TranformedDataStream(transactionLocation string, <br>      totalTransactionAmount double, avgTransactionAmount double, <br>      transactionId long);</pre><pre>-- <strong>Store table</strong><br>@store(type = &#39;rdbms&#39; , datasource = &#39;TRANSACTION_DATA_SOURCE&#39;)<br>define table TransactionTable(transactionId long, userId long, <br>              transactionAmount double, transactionLocation string);</pre><pre>-- <strong>In-memory Table to keep last processed transaction ID<br></strong>@primaryKey(&#39;key&#39;)<br>define table TreaggerStateTable (key string, lastProcessedId long);</pre><pre>@info(name = &#39;<strong>CollectLastProcessedId</strong>&#39;)<br>from TriggerStream as s right outer join TreaggerStateTable as t<br>select ifThenElse(t.lastProcessedId is null, 0l, t.lastProcessedId ) <br>             as lastProcessedId<br>insert into DataRetrivalStream;</pre><pre>@info(name = &#39;<strong>ExtractData</strong>&#39;)<br>from DataRetrivalStream as s join TransactionTable as t<br>   on s.lastProcessedId &lt; t.transactionId<br>select t.transactionId, t.transactionAmount, t.transactionLocation<br>insert into ExtreactedDataStream;</pre><pre>@info(name=&#39;<strong>UpdateLastProcessedId</strong>&#39;)<br>from ExtreactedDataStream<br>select &quot;lastProcessedId&quot; as key, transactionId as lastProcessedId<br>update or insert into TreaggerStateTable<br>   on TreaggerStateTable.key == &quot;lastProcessedId&quot;;</pre><pre>@info(name = &#39;StatefulAggregation&#39;)<br>from ExtreactedDataStream#window.timeBatch(5 min)<br>select transactionLocation, <br>       sum(transactionAmount) as totalTransactionAmount, <br>       avg(transactionAmount) as avgTransactionAmount, <br>       transactionId<br>group by transactionLocation<br>insert into TranformedDataStream;</pre><p>The Siddhi Application’s graphical design view for this scenario in the Stream Processor Studio is represented in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wbItcd-fiiHZZYHb278xSw.png" /><figcaption>Figure 3: Sample design view of Scenario 2 in the WSO2 SP Studio</figcaption></figure><h3>Enabling ETLs with WSO2 SP</h3><p>As shown in the above example scenarios WSO2 SP allows you to write ETL processor without writing a single Java code. What you need to do is to open the Stream Processor Studio/Editor that comes with WSO2 SP and simply type in the scripts using Streaming SQL or use the Graphical Editor to drag and drop components and build ETL flows. After creating the Siddhi Applications they can be deployed in WSO2 SP. The status of the ETL process and the performance of WSO2 SP nodes can be monitored via its Status Dashboard to identify performance and bottlenecks if there are any. Furthermore, such ETL processes can also be templated and enabled to be deployed by non-technical users with a click of a button using WSO2 SP’s Business Rules Feature.</p><p>In conclusion, Real-time ETLs have now become inevitable for any organizations and with products like WSO2 Stream Processor building such ETL process has become a very simple task. <a href="https://wso2.com/analytics">Download</a> and try out the latest stable version of WSO2 Stream Processor to implement the scenarios. You can get more details about the product by visiting its <a href="https://wso2.com/analytics">official website</a>, for further help refer to the <a href="https://docs.wso2.com/display/SP4xx">documentation</a> and the <a href="https://wso2.github.io/siddhi/">Siddhi Query Guide</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=358680689935" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stream-processing/perform-realtime-and-periodic-etl-with-wso2-stream-processor-358680689935">Perform Realtime and Periodic ETL with WSO2 Stream Processor</a> was originally published in <a href="https://medium.com/stream-processing">Stream Processing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Synchronous Request-Response based Real-time Processing with WSO2 Stream Processor]]></title>
            <link>https://medium.com/@suhothayan/synchronous-request-response-based-real-time-processing-with-wso2-stream-processor-38613d32a1ec?source=rss-256afc94c5d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/38613d32a1ec</guid>
            <category><![CDATA[http-request]]></category>
            <category><![CDATA[stream-processing]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[rule-engine]]></category>
            <category><![CDATA[request-response]]></category>
            <dc:creator><![CDATA[Suhothayan Sriskandarajah]]></dc:creator>
            <pubDate>Wed, 04 Jul 2018 17:56:25 GMT</pubDate>
            <atom:updated>2018-07-13T12:58:13.684Z</atom:updated>
            <content:encoded><![CDATA[<p>Current <a href="https://www.kdnuggets.com/2018/05/event-processing-important-open-problems.html">stream processing systems</a> are generally designed to process data asynchronously. They consume events from various input sources, process via data processing pipelines, and asynchronously publish outputs to relevant event sinks. This type of processing is useful for building systems like <a href="https://medium.com/@suhothayan/stream-processor-as-a-notification-manager-7eab163b4770">monitoring and alerting systems</a>, for example, a system that monitor servers and alert potential system failures to relevant stakeholders. In these cases, for each incoming message, a corresponding response is only produced at a later period of time. Further, in most cases, these responses are published to a different system other than the original event producer. Therefore, the requesting system returns immediately without knowing the outcome of the message.</p><p>But still, there are a whole different set of use cases that require real-time processing with an immediate synchronous response for each request. These contain use cases such as adaptive authentication, <a href="https://wso2.com/library/articles/2016/09/article-introducing-wso2-api-managers-throttling-implementation-architecture/">real-time usage throttling</a>, and real-time resource allocation systems. In these cases, the system that provides services will synchronously request the real-time control system (in this use case the stream processing system) to get an immediate response such that it can decide on how it could proceed. In this post, let’s see how we can achieve synchronous request-response decision making using <a href="https://wso2.com/analytics">WSO2 Stream Processor</a>.</p><h3>General Flow of Event Stream Processing</h3><p>Typically WSO2 Stream Processor consumes events from <a href="https://docs.wso2.com/display/SP4xx/Collecting+Events">various event sources such as Kafka, HTTP, JMS, MQTT, etc</a>, and it runs Siddhi stream processing queries to detect abnormal incidents among them. These identified incidents are sent to various endpoints as events via <a href="https://docs.wso2.com/display/SP4xx/Publishing+Events">protocols such as HTTP, Kafka, e-mail, JMS, etc</a>. Here, the sources that send events will be different to the ones that receive the notifications, and therefore asynchronous processing the incoming events makes lots of sense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eAIC9FABqR0kNZnmjA6zHg.png" /><figcaption>Figure 1: Asynchronous data pipeline processing in WSO2 Stream Processor</figcaption></figure><p>Based on this approach, even if the source and sink are from the same entity, the response delivery will be done asynchronously. Therefore, this mode of processing is not ideal to implement rule-based systems which needs an immediate synchronous response for each of their requests.</p><h3>Synchronous Processing of WSO2 Stream Processor</h3><p>The requirement of responding with an immediate synchronous response for each of requests and be achieved using the following two concepts of WSO2 Stream Processor.</p><ol><li>Store Queries</li><li>Synchronous HTTP Request-Response</li></ol><p>Let’s see how each of these can be used to achieve real-time data processing for synchronous request-response.</p><h3>Store Queries</h3><p>With store queries, users are able to synchronously query runtime data stores using a REST APIs. The runtime data stores that support querying in WSO2 Stream Processor are Siddhi tables, windows, and aggregations. These runtime data stores can contain both static or dynamic data. For example, it can be an RDBMS database backed Siddhi table or a dynamically updated in-memory Siddh time window, or it can even be a hourly based long-running Siddhi aggregation of sales data that’s updated both in-memory and in a database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q94Awwun0Ul8tM-DjUuW-A.png" /><figcaption>Figure 2: Synchronous store querying in WSO2 Stream Processor</figcaption></figure><p>As depicted in the above diagram, WSO2 Stream Processor allows users to query tables, windows, and aggregations defined in Siddhi applications. A POST request issued to the API context “/stores/query” of WSO2 Stream Processor will allow users to query these data stores. Following represents a curl template for synchronously querying data stores.</p><pre>curl -X POST <br>https://&lt;WSO2 Stream Processor Host&gt;:&lt;Port&gt;/stores/query <br>-H “content-type:application/json” <br>-u “&lt;User Name&gt;:&lt;Password&gt;” <br>-d ‘{“appName” : “&lt;Siddhi App Name&gt;”, “query” : “&lt;Store Query&gt;” }’ <br>-k</pre><p>A sample POST request using curl for selecting all elements from the “StockTable” table defined in “StockProcessing” Siddhi application is as below.</p><pre>curl -X POST <a href="https://localhost:9443/stores/query">https://localhost:9443/stores/query</a> <br>-H &quot;content-type: application/json&quot; <br>-u &quot;admin:admin&quot; <br>-d &#39;{&quot;appName&quot; : &quot;StockProcessing&quot;, <br>     &quot;query&quot; : &quot;from StockTable select *&quot; }&#39; <br>-k</pre><p>With this approach users can issue SELECT queries as following on tables, windows, and aggregations:</p><pre><strong>from</strong> StockTable<br><strong>select</strong> symbol, avg(price) as avgPrice<br><strong>group</strong> <strong>by</strong> symbol<br><strong>order by</strong> avgPrice<br><strong>limit</strong> 10</pre><p>When such requests are made the system will instantaneously process aggregation, group by, order by and limits clauses and produces the results. These queries will even work while other real-time operators such as streams are modifying the queried data stores.</p><p>Like issuing SELECT queries, from WSO2 Stream Processor 4.2 and higher, the users can also issue DELETE, UPDATE, INSERT and UPDATE OR INSERT queries to Siddhi tables. These can manipulate both in-memory, and databases backed Siddhi tables. For more information on the queries refer <a href="https://wso2.github.io/siddhi/documentation/siddhi-4.0/#store-query">the store query section on Siddhi Query Guide</a>.</p><h3>Synchronous HTTP Request-Response</h3><p>Apart from querying the stores, WSO2 Stream Processor allows users to send a request, and synchronously get it processed and responded using the data processing pipeline. As depicted in the below diagram, when a user posts a request message, it is converted to an event and synchronously processed through the data processing pipeline. Once the processing is finished, the processed event is responded back to the original request. By this, the users can now use any stream processing constructs such as filter, pattern, join and others in a synchronous manner. This feature is implemented as part of <a href="https://wso2-extensions.github.io/siddhi-io-http/api/latest/#http-response-sink">HTTP IO</a> and available from WSO2 Stream Processor 4.2 and above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bM8dY_lfFwxmvHlWGEkj1g.png" /><figcaption>Synchronous HTTP request-response with WSO2 Stream Processor</figcaption></figure><p>This synchronous behavior is achieved by using the HTTP IO’s <a href="https://wso2-extensions.github.io/siddhi-io-http/api/latest/#http-request-source">http-request source</a> and <a href="https://wso2-extensions.github.io/siddhi-io-http/api/latest/#http-response-sink">http-response sink</a> together. Here the http-request source should be configured to consume the incoming request by providing an API context, message mapping, timeout period, and an unique ID. The message mapping will be used to convert the incoming XML or JSON message into Siddhi event format, the timeout period will determine the connection timeout, and the ID will be used to uniquely identify the particular http-request source. This http-request source should be added to the binding stream to which the events need to be pushed.</p><p>A sample http-request source that can consume a JSON message on API context “/endpoints/stock” can be defined as below:</p><pre>@source(type=&#39;http-request&#39;, <strong>source.id=&#39;StockStreamSource&#39;</strong>,  <br>        receiver.url=&#39;http://localhost:9055/endpoints/stock&#39;,    <br>        connection.timeout=&#39;150000&#39;, <br>        @map(type=&#39;json,<br>             @attributes(<strong>messageId=&#39;trp:messageId&#39;</strong>,          <br>                         symbol=&#39;$.events.event.symbol&#39;,  <br>                         price=&#39;$.events.event.price&#39;,  <br>                         volume=&#39;$.events.event.volume&#39;)))<br>define stream StockStream (messageId string, symbol string, <br>                           price float, volume long);</pre><p>Here the source.id=‘StockStreamSource’ uniquely identifies the source and ‘trp:messageId’ mapping property is used to assign a unique message ID to correlate each incoming message.</p><p>After the http-request source, the event enters a stream. Through the stream, it goes through various parts of the data processing pipeline and eventually reaches an http-response sink. There the processed event will be mapped as output message and delivered as a response to the user.</p><p>A sample http-response sink that corresponds to the above-defined source can be defined as below:</p><pre>@sink(type=’http-response’, <strong>source.id=&#39;StockStreamSource&#39;</strong>,  <br>      <strong>message.id=’{{messageId}}’</strong>, <br>      @map(type=’json’, @payload(‘{{results}}’))<br>define stream ProcessedStockStream (messageId string, <br>                                    results String);</pre><p>In order to correlate the request message form http-request source and the response message to the http-response sink the ‘source.id’ and ‘message.id’ properties are used. Through the ‘source.id’ the linked http-request source and http-response sink are identified and using the ‘message.id’ the corresponding instance of the request is mapped to response.</p><p>Further, a single http-request source can also be linked with multiple http-response sinks. In the synchronous HTTP request-response process, when the request arrives via the REST API to the source it gets blocked and a corresponding event with a unique message ID will be generated and sent to the processing pipeline for event processing. After all the processing are done the event will reach one of the http-response sinks. The sink that receives the message processed message first will extract the event’s message ID, find the associated blocked input request, convert the processed event according to output mapping, and send that as a response to the API call. As described when multiple sinks are available the sink that gets the event first will send the response, and if the event did not reach any sink within the connection timeout the http-request source will send a response with the request timeout response status code — 504. For further information on Synchronous HTTP Request-Response refer to the <a href="https://wso2-extensions.github.io/siddhi-io-http/api/latest/">HTTP IO documentation</a>.</p><p>In this post, we looked at how we can use WSO2 Stream Processor as a synchronous request-response system, in addition to generic asynchronous stream processing. This allows users to synchronously query its data stores and data pipelines, and get the response immediately in the request thread. One important aspect of the processing is that this will not block the treads and hence it will not introduce any additional performance degradation for the streaming applications that are processing data in realtime. You can download <a href="https://wso2.com/analytics">WSO2 Stream Processor 4.2 or higher</a> and try out these features. WSO2 Stream Processor can be download from <a href="https://github.com/wso2/product-sp/releases/download/v4.2.0-rc2/wso2sp-4.2.0.zip">here</a>. If you want to use the HTTP IO extension you can download that from <a href="https://store.wso2.com/store/assets/analyticsextension/list">WSO2 Store</a> and add that to the `<em>&lt;SP_HOME&gt;/libs</em>` directory replacing the existing <em>`siddhi-io-http-1.0.29.jar`</em>. For more information about WSO2 Stream Processor visit its <a href="https://wso2.com/analytics">official website</a> and <a href="https://docs.wso2.com/display/SP4xx/">documentation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38613d32a1ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use a Stream Processor as a Notification Manager?]]></title>
            <link>https://medium.com/stream-processing/stream-processor-as-a-notification-manager-7eab163b4770?source=rss-256afc94c5d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7eab163b4770</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[alert-management]]></category>
            <category><![CDATA[stream-processing]]></category>
            <category><![CDATA[wso2]]></category>
            <dc:creator><![CDATA[Suhothayan Sriskandarajah]]></dc:creator>
            <pubDate>Tue, 05 Jun 2018 06:05:46 GMT</pubDate>
            <atom:updated>2018-08-13T02:30:42.607Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GNoJTMErEsPPIsFhvI7ZZg.png" /><figcaption>Reference Architecture of Streaming Notification Management Platform</figcaption></figure><p>Streaming applications process events, identify relevant information or anomalies in them and create alerts. Although most of the stream processing systems support filtering, aggregation, joining, pattern matching, and alert generation, they don’t concentrate on how those alerts are managed. Managing alerts is not just a matter of detecting conditions and responding to them. The systems need to identify the relevant authorities to alert, notify them based on user’s preferred frequency and message formats, provide guaranteed delivery for mission-critical alerts, retry if the end user is not reachable, and finally escalate to other stakeholders when the allocated person has not attended to the alert on time. In most scenarios, the preference of the notification receiver is the key. The system should have the capability to let the end users manage the alert condition, when, and how they should get notified. Further, the system should also allow the users to change the configurations using alert templates and forms easily.</p><p>As a result, some organizations rely on an alternative system such as notification management systems for alerts. Stream processors can support this use case and often do a better job. In this post, let’s look at how we can use<a href="https://wso2.com/analytics"> WSO2 Stream Processor</a> to process streams while providing autonomy to the users for managing their notifications and alerts.</p><p>WSO2 Stream Processor (a.k.a. WSO2 SP) is an open source, cloud-native, and lightweight stream processing platform that understands streaming SQL queries in order to capture, analyze, process and act on events in real time. WSO2 SP facilitates real-time streaming analytics and streaming data integration. It has<a href="https://wso2.com/analytics/features"> many tools and components</a> to support different kinds of users from developers to business users to serve their requirements. For example, it includes built-in<a href="https://docs.wso2.com/display/SP4xx/Understanding+the+Development+Environment"> developer IDE</a> to build stream processing applications, a<a href="https://docs.wso2.com/display/SP4xx/Viewing+Statistics"> status dashboard</a> to provide observability, a<a href="https://docs.wso2.com/display/SP4xx/Visualizing+Data"> business user portal</a> to visualize stream processing results and aggregations, and<a href="https://medium.com/@suhothayan/making-stream-processing-accessible-to-business-users-54c57e566063"> business rules manager</a> to provide domain-specific stream processing configuration management. These components and tools are grouped together under<a href="https://docs.wso2.com/display/SP4xx/Running+the+Product"> different profiles</a> such as <em>Worker</em>, <em>Manager</em>, and <em>Dashboard </em>such that the can be deployed and scaled in an efficient way. Here, <em>Worker</em> profile executes stream processing logic, <em>Manager</em> profile deploys and manages stream processing apps on <em>Worker</em> profiles, and the <em>Dashboard</em> profile consists of the status dashboard, the business user dashboard and the<a href="https://medium.com/@suhothayan/making-stream-processing-accessible-to-business-users-54c57e566063"> business rules manager</a>. As we will discuss later, these profiles provide fine grain control over the deployment architecture.</p><h3>Scenario</h3><p>As an example, let’s consider a production pipeline of a manufacturing plant such as electric bikes and processor chips. We need to monitor the quality of the products, the rate of production, and notify necessary authorities when there are important production incidents such as defects in products, declined throughput of production output, and machine failures.</p><p>This use case has two parts. The system first needs to detect quality and production issues. Then it has to notify them to appropriate authorities. We can collect the status of the production machines by inspecting the products produced at various stages of the pipeline and publishing them to WSO2 Stream Processor (WSO2 SP). The streaming application running in WSO2 SP will identify anomalies and generate new notifications streams. Finally, we need to manage notifications, deliver them to relevant authorities, provide event format transformations, rate limiting, redelivery, and escalation as needed. Templated Siddhi streaming applications and <a href="https://medium.com/@suhothayan/making-stream-processing-accessible-to-business-users-54c57e566063">business rules</a> management component of WSO2 Stream Processor can handle this. The following section shows a reference architecture to achieve this.</p><h3>Reference Architecture</h3><p>Following reference, architecture shows how to use WSO2 SP to implement a notification management system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GNoJTMErEsPPIsFhvI7ZZg.png" /><figcaption>Reference Architecture of Streaming Notification Management Platform</figcaption></figure><p>The reference Architecture separates the management of alerts based on user preference from the general stream processing layer that is responsible for detecting alerts because any issue while pushing out the notifications should not hinder the general event processing flow. For example, unavailability of notification endpoints should not produce any back pressure on the processing environment and halt alert detection altogether.</p><p>WSO2 SP can consume events from<a href="https://docs.wso2.com/display/SP4xx/Collecting+Events"> various streaming data sources</a> such as Kafka, JMS, MQTT, HTTP, Databases and via other protocols in data formats such as JSON, XML, text, key-value, CSV and others. The stream processing layer will be the main processing section that processes all incoming events based on general requirements to identify interesting event occurrences that are worth notifying to the end users. These matched events comprising quality and production incidents are then passed to the alert management layer to filter out, rate limit, and to notify the relevant users’ based on their preferences. For example, these notifications will be delivered on a defined rate limits such as and when an incident has happened or one every hour, and they can also be based on users intended notification types and formats, such as e-mails and text messages with custom text content, API calls to endpoints with XML, JSON formats, or many others.</p><p>When the stream processing layer detects interesting events based on the defined Siddhi applications, it generates new notification streams containing the identified events. These notification streams need to be further processed, and the alerts should be only delivered to the intended audience based on their required frequency and message format. At times, if end users did not attend alerts on time, these alerts need to be escalated to other stakeholders.</p><p>Generating and managing such alerts can be done using the business rules component of WSO2 Stream Processor. This allows business users<a href="https://docs.wso2.com/display/SP4xx/Managing+Business+Rules"> to create, delete, edit, enable and disable</a> the alerts they receive in a self-service manner without issuing change requests and waiting for the requests to be serviced.</p><p>Let’s look at the business rules component in detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zk8Wym0h_fNHKQQxC7gg3A.png" /><figcaption>Alert management UI of the WSO2 Stream Processor’s business rules component</figcaption></figure><p>As the alert management UI of business rules speaks the domain language, it provides full authority for the business users to generate and manage their alerts by themselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bIQBjkfb-ckepfFJzH7sQQ.png" /></figure><p>The<a href="https://medium.com/@suhothayan/making-stream-processing-accessible-to-business-users-54c57e566063"> business rules component</a> plays a key role in this use case. It let the developers define a common stream processing logic with information such as the messaging protocols, the frequency of alert delivery, and escalation threshold. Then it let them template the logic and map them to a customized domain-specific input form as shown above image. As these mapped forms speak the domain of the end users it will be much easier for the end user to manage their alerts. Furthermore, WSO2 Stream Processor also allows the developers to add<a href="https://docs.wso2.com/display/SP4xx/Creating+a+Business+Rule+Template"> validations to the forms</a> such that they can validate the inputs and provide suggestions to the end users. It also allows the developers to<a href="https://docs.wso2.com/display/SP4xx/Creating+a+Business+Rule+Template"> combine the collected inputs using JavaScript</a> to generate the necessary content for the stream processing template. Through this, it provides better usability for both end users and developers, where the end users are only asked relevant information in a way they understand, and the developers are allowed to manipulate the input data in any form they prefer. In addition to these, these domain-specific business rules can also be accessed via<a href="https://docs.wso2.com/display/SP4xx/Stream+Processor+REST+API+Guide"> rest APIs</a>. Hence, they can be easily integrated with existing configuration system in the organization.</p><p>Consequently, the notification management system built using WSO2 Stream Processor allows end users to configure the alerts without worrying about the stream processing complexities while shortening the development lifecycle. For the details about how to use business rules and build domain-specific forms, go through the following screencast and for more information refer<a href="https://medium.com/@suhothayan/making-stream-processing-accessible-to-business-users-54c57e566063"> my post</a> and the<a href="https://docs.wso2.com/display/SP4xx/"> user documentation</a>.</p><p><a href="https://youtu.be/7WGJf4lxjqc">https://youtu.be/7WGJf4lxjqc</a></p><h3>Deployment</h3><p>figure shows how to deploy the discussed reference architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S_3FfbmCQ_sD4bJ2ZgVk4g.png" /></figure><p>Based on the diagram the stream processing layer will contain worker and manager profiles for stream processing, and the alert management layer will additionally contain the business rules component for managing notifications. Based on the scale of the system WSO2 Stream Processor can be deployed either in<a href="https://docs.wso2.com/display/SP4xx/Minimum+High+Availability+%28HA%29+Deployment"> High Available (HA)</a> or in the<a href="https://docs.wso2.com/display/SP4xx/Fully+Distributed+Deployment"> distributed deployment</a> model. Further, the stream processing layer and alert management layer can have two separate WSO2 Stream Processor clusters or share the same cluster based on the scalability and high availability needs. This is because the stream processing layer, and the alert management layer depicted in the above deployment diagram do not mandate a physical separation between them, but rather they only represent a logical separation in the overall stream processing flow.</p><h3>Conclusion</h3><p>In this post, we looked at a reference architecture to build a notification management system using WSO2 Stream Processor and its business rules component. We discuss how WSO2 SP can go beyond simple alert generator and act as a sophisticated alert manager supporting domain-specific notification rules building and management, validating business rules for correct user input while providing guaranteed message delivery for mission-critical data. To implement the above solution,<a href="https://wso2.com/analytics"> download</a> and try the latest stable version of WSO2 Stream Processor, and to get more details about the product by visiting <a href="https://wso2.com/analytics">WSO2 SP website</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7eab163b4770" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stream-processing/stream-processor-as-a-notification-manager-7eab163b4770">How to use a Stream Processor as a Notification Manager?</a> was originally published in <a href="https://medium.com/stream-processing">Stream Processing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Stream Processing Accessible to Business Users]]></title>
            <link>https://medium.com/@suhothayan/making-stream-processing-accessible-to-business-users-54c57e566063?source=rss-256afc94c5d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/54c57e566063</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[stream-processing]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[domain-specific-languages]]></category>
            <category><![CDATA[wso2-sp]]></category>
            <dc:creator><![CDATA[Suhothayan Sriskandarajah]]></dc:creator>
            <pubDate>Sun, 20 May 2018 04:54:32 GMT</pubDate>
            <atom:updated>2018-08-12T10:44:17.816Z</atom:updated>
            <content:encoded><![CDATA[<p>Current Big Data environment is complex with multiple options for processing data, such as batch, realtime, and machine learning. Nowadays organizations have understood the importance of time and started to focus more on realtime and machine learning technologies to achieve immediate diagnosis and future predictions. <a href="https://wso2.com/library/articles/2018/05/what-is-stream-processing">Stream Processing</a> is increasingly gaining a lot of popularity among developers as the way to do realtime processing in a faster, distributed and scalable manner. But business users are still reluctant to have a first hand experience in using this technology.</p><p>When compared to other data analytics approaches, Stream Processing is much more straightforward where it consumes incoming streaming data such as events coming from a Kafka Topic, and executes predefined rules on them. Some Stream Processing systems also have the capability of accessing historical data and doing predictions in the realtime. Further, it also does not need an intervention of a Data Scientist who need to get involved in continuous monitoring and configurations, because if we setup once, then it can continue to run and check for anomalies by itself.</p><p>Yet, as we all know neither the market or the business does not stay still, and their requirements always change. With this constant changes, without an agile development environment, the organizations will fail to meet their deadlines in keeping up their realtime systems uptodate. Also, more often these change requests are simple tweaks such as changing the KPIs or threshold values.</p><p>Overtime, Stream processing vendors have realized this requirement, and now there is an increased momentum on building drag-and-drop and building form based query builders to get business and non technical folks to use Stream Processing systems. Companies such as Hortonworks, and Tibco Streambase have followed this path. Though this give some advantages to a novice technical user, based on my experience with the customers it does not give a lot of advantage to non-technical, business and even to highly technical user. This is because a highly skilled users find these as inefficient way to implementing streaming applications, where they usually prefer typing the rules compared to meddling with complicated UIs. For this reason, some systems like Spark, Flink, &amp; Storm have continued to stick with SQL-Like or general purpose programing language to implement stream processing applications. Also, when it comes to business and non-technical users, they find this very complex both in understanding streaming terminologies and concepts such as windows, partitions, patterns, etc. Therefore, these users need more domain specific configurations to manage such systems.</p><p>I’m my opinion, the best way forward is to build a <strong>domain specific configuration UIs</strong> such that it allows business users to directly interact with the system. But, generally building such UI require immense effort, where it takes a lot of development time and it can also be error prone. At the same time we can’t also provide prebuilt UIs, because it might work for one specific use case but not for the other, and further it also should confess the users domain, be customizable as per the organization, and support validations.</p><p>With modern technology and innovation, building domain specific UIs have become much more feasible. New systems like <a href="https://wso2.com/analytics/features">WSO2 Stream Processor 4.2</a> takes an innovative approach of templating configurations and enabling developers to build a domain specific UIs for business users in just few minutes. This reduces the complexity of the processing logic and give only the required probes to the business users to manage the system.</p><p><strong>The following steps explains the process of creating and managing templated domain UIs in </strong><a href="https://wso2.com/analytics/features"><strong>WSO2 Stream Processor 4.2</strong></a>, including steps done by developers and business users:</p><p><strong>The developer:</strong></p><blockquote><strong>Step 1:</strong> Builds a standard stream processing application.<br>In this case it’s a SiddhiApp file, that can be generated via the <a href="https://docs.wso2.com/display/SP410/Understanding+the+Development+Environment">Siddhi Editor</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rr7gVAtZtUrWbkQf." /><figcaption>Siddhi Application written in Stream Processor Console</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nbMarRA7zFoskWWO_-QHIw.png" /><figcaption>Design view of the Siddhi Application created.</figcaption></figure><blockquote><strong>Step 2:</strong> Template the application with business information and necessary validation such that it confess the domain of the end user.<br>Though this is quite straightforward to manually generate templates, WSO2 Stream Processor provides a <a href="https://docs.wso2.com/display/SP410/Creating+a+Business+Rule+Template">template builder tool</a> that makes building templates effortless.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*17DZwcxHtq8YHmU1." /><figcaption>Templating Siddhi Application via the Template Editor</figcaption></figure><blockquote><strong>Step 3:</strong> Deploy templates in production as business rules, such that the business users can manage those rules.</blockquote><p><strong>The business user:</strong></p><blockquote><strong>Step 1: </strong>Access and <a href="https://docs.wso2.com/display/SP410/Creating+Business+Rules">configure</a> the domain specific business rules from the management UI.<br>Here with a single click the system will automatically deploy the rules as Siddhi App with appropriate configurations in to relevant nodes. This also supports deploying the application in a distributed and scalable manner.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*31XD0U-2d0Xhydfr." /><figcaption>Create rules in Business Rules Manager based on the deployed template.</figcaption></figure><blockquote><strong>Step 2: </strong>As and when the business requirement changes, the business user <a href="https://docs.wso2.com/display/SP410/Managing+Business+Rules">modifies, disable or enable</a> the rule via the management UI.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3BlNzlGmquz1YC0V." /><figcaption>Manage created rules by view, edit, delete and redeploy options.</figcaption></figure><p>As always there can be exceptions, if the requirement has drastically changed, then the business users will not be able to handle this by themselves. At this point, such request has to go through the traditional change request process. Nevertheless, the chances of this circumstance is minimum. Even if it happens quite often, we are not losing anything, because the effort for building templated rules are insignificant, and hence templated rule can be regenerated by the developer.</p><p>Following screencast demonstrate how we can use WSO2 Stream Processor to build and manage Business User friendly templated rule management that we have discussed so far.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7WGJf4lxjqc%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7WGJf4lxjqc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F7WGJf4lxjqc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6397f591b98dd7486cd28bede53f529a/href">https://medium.com/media/6397f591b98dd7486cd28bede53f529a/href</a></iframe><p>The major advantage of this approach is, its flexibility and autonomy that it brings to the business user, and letting them control the system they manage with almost no addition effort from the developers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54c57e566063" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>