<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ajanthy Jayarajan on Medium]]></title>
        <description><![CDATA[Stories by Ajanthy Jayarajan on Medium]]></description>
        <link>https://medium.com/@ajanthy?source=rss-4b55be40f8f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Ioh2EsMMTPHyDTYlgvjWBg.jpeg</url>
            <title>Stories by Ajanthy Jayarajan on Medium</title>
            <link>https://medium.com/@ajanthy?source=rss-4b55be40f8f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:52:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ajanthy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Cron Support for Siddhi file connector]]></title>
            <link>https://medium.com/@ajanthy/cron-support-for-siddhi-file-connector-c0c08eb19ad2?source=rss-4b55be40f8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0c08eb19ad2</guid>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[siddhi]]></category>
            <category><![CDATA[schedular]]></category>
            <category><![CDATA[streaming-integrator]]></category>
            <dc:creator><![CDATA[Ajanthy Jayarajan]]></dc:creator>
            <pubDate>Mon, 29 Jun 2020 11:42:54 GMT</pubDate>
            <atom:updated>2020-06-29T11:42:54.336Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I’m going to discuss the cron support for file connector (siddhi-io-file) used in WSO2 Streaming Integrator. For this purpose I’m using siddhi-io-file source extension.</p><p><a href="https://wso2.com/integration/streaming-integrator/"><strong>WSO2 Streaming Integrator(SI)</strong></a> is a streaming data processing server that integrates streaming data and takes action based on streaming data. The streaming integration capabilities of EI are delivered via this runtime.<strong> File Source</strong> provides the functionality for the user to feed data to Siddhi from files. Both text and binary files are supported by file source. Here I’m going to define the new feature Cron Support for <a href="https://siddhi-io.github.io/siddhi-io-file/"><em>Siddhi-io-file</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/0*9rtvcLunM8UMtuZ0.png" /></figure><p>Let’s get into the topic. Some of you might be wondering what is cron expression. To get a clear idea on Cron Expression go through my <a href="https://medium.com/@ajanthy/cron-expression-5a68526ffb23">previous article</a>.</p><h3>Why do we need this?</h3><p>You might be wondering why we need this cron support in siddhi-io-file. Initially we had a traditional system where it dumps all the data to the file system, then the new system turned up to analyze the data via database. Now we are using WSO2 Streaming Integrator to load data from file and insert into a database. But there is an issue on heavy load in the database and sometimes we might miss the records of processed files, to overcome these two issues we have scheduled time using cron expression to start the processing of file or files.</p><p>Let’s say if the time is scheduled as every night 12 am, All the file or files in the given directory or given file will be processed and inserted into the database at the specific time.</p><p>Let’s see how Siddhi-io-File works with cron Expression. For this functionality a new parameter ‘cron.expression’ is introduced. <strong>‘cron.expression’</strong> : This is used to specify a timestamp in cron expression. The file or files in the given dir.uri or file.uri will be processed when the given expression satisfied by the system time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*GssoA9UTs5Ex2TJLeZy02Q.png" /><figcaption>Implementation of Cron Support</figcaption></figure><h3>Prerequisites :</h3><p>Running instance of WSO2 Streaming Integrator Tooling server. <a href="https://wso2.com/integration/streaming-integrator/">Click here</a> for instructions on how to download and start Streaming Integrator Tooling server.</p><h3>Sample Siddhi Application</h3><pre>@App:name(“SiddhiAppCronSchedule”)<br>@App:description(“Processing a file at given cron Expression”)</pre><pre>@source(type=’file’, file.uri=’file:/path/to/file’, action.after.process=’MOVE’, move.after.process=’file:/path/to/be/moved’, cron.expression=’0 */1 * * * ?’, tailing=’false’, read.only.header=&#39;false&#39;, @map(type=’csv’))<br>define stream InputStream (id int, name string, amount double);</pre><pre>@sink(type=’log’)<br>define stream OutputStream (id int, name string, amount double);<br> <br>from InputStream<br>select *<br>insert into OutputStream;</pre><p>This application demonstrates how to configure WSO2 streaming integrator tooling to process file using the cron expression. Let’s go through this sample line by line.</p><pre>@App:name(“SiddhiAppFile”)@App:description(“Processing a file at given cron Expression”)</pre><p>Here <strong>@App:name</strong> gives the name of the Siddhi Application and <strong>@App:description</strong> provide option to define the application and it is optional.</p><pre>@source(type=’file’, file.uri=’file:/path/to/file’, action.after.process=’MOVE’, move.after.process=’file:/path/to/be/moved’, cron.expression=’0 */1 * * * ?’, tailing=’false’, read.only.header=&#39;false&#39;, @map(type=’csv’))<br>define stream InputStream (id int, name string, amount double);</pre><p><strong>Source</strong> is a contract that consumes data from external sources in the form of events, then converts each event (which can be in XML, JSON, binary, etc. format) to a Siddhi event, and passes that to a Stream for processing. In this scenario source type is <strong>file </strong>and <strong>cron.expression</strong> defines at every minute the cronjob will be scheduled. If the cron expression satisfies the system time file at in the <strong>file.uri</strong> is processed and after the process file will be moved to the filepath mentioned in the parameter called <strong>move.after.process</strong>. <strong>Tailing</strong> has to be set to <strong>false</strong>. There is an optional parameter specified as <strong>read.only.header</strong> which mentions if the parameter set to <strong>true</strong> it only reads the header of the file, If it set to <strong>false</strong> it reads the complete file with headers. And as default that is set to <strong>false</strong>. And also <strong>InputStream</strong> has 3 attributes such as id, name and amount.</p><pre>@sink(type=’log’)<br>define stream OutputStream (id int, name string, amount double);</pre><p><strong>Sink</strong> is a contract that takes events arriving at a stream, maps them to a predefined data format, and publishes them to external endpoints. In this scenario sink specified as <strong>log</strong> type that produces the event with the attributes id, name and amount. File will be processed and event will be thrown with these attributes.</p><pre>@info(name = ‘query’)<br>from insertSweetProductionStream<br>select name, amount<br>insert into logStream;</pre><p>This final segment of the code describes how events are sent from the input stream to the output stream. In this example you are selecting the <strong>name</strong> and <strong>amount</strong> from all the events from the<strong> insertSweetProductionStream</strong> and sending them to <strong>logStream </strong>without checking any conditions.</p><h3>How to run the Sample Application</h3><h3>Prerequisites</h3><ul><li>You can save this sample in SI tooling (<strong>File -&gt;Save</strong>) .</li><li>For this example, you are using the following sample CSV event set. Copy and paste the following text to a new CSV file and change the value of <strong>file.uri</strong> parameter to the location of the CSV file.</li></ul><pre> 1 WSO2 23.5 <br> 2 IBM 2.5</pre><ul><li>Edit the file.uri to ‘<strong>file:/path/to/file</strong>’ and <strong>‘file:/path/to/be/moved’ </strong>by<strong> </strong>replacing respectively with the absolute path of your file which has to be processed and where it has to be moved.</li><li>Save this sample.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uC6jEGdyau-6tHheA_0o4w.png" /></figure><h3>Executing the Sample</h3><ol><li>Start the Siddhi application by clicking on ‘<strong>Run</strong>’ .</li><li>If the Siddhi application starts successfully, the following message would be shown on the console<br><strong><em>* SiddhiAppCronSchedule.siddhi — Started Successfully!</em></strong></li></ol><h3>Viewing the Results:</h3><ul><li>According the content in the file, you will see events similar to the following when you start the Siddhi app.</li></ul><pre>[2020-04-27_21-54-01_003] INFO {io.siddhi.core.stream.output.sink.LogSink} - SiddhiApp : OutputStream : Event{timestamp=1588004641003, data=[1, WSO2, 23.5], isExpired=false} <br>[2020-04-27_21-54-01_005] INFO {io.siddhi.core.stream.output.sink.LogSink} - SiddhiApp : OutputStream : Event{timestamp=1588004641005, data=[2, IBM, 2.5], isExpired=false}</pre><p>If you change the cron.expression according to that, <strong>time</strong> when the file has to be processed will differ.</p><h3>Conclusion</h3><p>This article described how to use WSO2 Streaming Integrator to execute a task in siddhi-io-file at scheduled time using the cron expression. As mentioned in this example, you can check with any directory uri or file uri for any files which has to be processed.</p><h3>References</h3><ul><li><strong>Introduction on Cron Expression</strong> <br><a href="https://medium.com/@ajanthy/cron-expression-5a68526ffb23">https://medium.com/@ajanthy/cron-expression-5a68526ffb23</a></li><li><strong>Streaming Integrator Download</strong><br><a href="https://wso2.com/integration/streaming-integrator/">https://wso2.com/integration/streaming-integrator/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0c08eb19ad2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[File Listening Extension for WSO2 Streaming Integrator]]></title>
            <link>https://medium.com/@ajanthy/file-listening-extension-for-wso2-streaming-integrator-1c96c8cd9fa7?source=rss-4b55be40f8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c96c8cd9fa7</guid>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[streaming-integrator]]></category>
            <category><![CDATA[siddhi]]></category>
            <dc:creator><![CDATA[Ajanthy Jayarajan]]></dc:creator>
            <pubDate>Mon, 29 Jun 2020 11:42:05 GMT</pubDate>
            <atom:updated>2020-06-29T11:42:05.681Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I’m going to discuss the file listening extension used in WSO2 Streaming Integrator. For this purpose I’m using siddhi-io-file source extension.</p><p><a href="https://wso2.com/integration/streaming-integrator/"><strong>WSO2 Streaming Integrator</strong></a> is a streaming data processing server that allows you to integrate event streams and take action based on it. The streaming integration capabilities of EI are delivered via this runtime. File Source provides the functionality for the user to feed data to Siddhi from files. Both text and binary files are supported by file source. Here I’m going to define the new file extension added in <a href="https://siddhi-io.github.io/siddhi-io-file/"><em>Siddhi-io-file</em></a><em>.</em></p><h3>Why do we need this?</h3><p>Initially there was no extension which listens to the directory and send events when the content of the directory changes. This was a requirement for File Processing in ETL scenarios in streaming integrator. So to fulfill that requirement, a new source type was introduced in siddhi-io-file “<strong>fileeventlistener</strong>”. The ‘<strong>fileeventlistener</strong>’ component of the ‘<strong>siddhi-io-file</strong>’ extension allows you to get the details of files that have been created, modified or deleted during execution time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/0*XTI6KsL8aruCH0QW.jpeg" /></figure><p>Here source type is specified as <strong>‘fileeventlistener’</strong> with three main parameters such as<strong> dir.uri, monitoring.interval</strong> and <strong>file.name.list</strong>.</p><p><strong>dir.uri</strong> : During execution time, Siddhi by default processes all the files within this directory. However, if you have entered specific files to be processed via the ‘file.name.list’ parameter, only those files are processed. The URI specified must include the file handling protocol to be used for file processing.<br>E.g : If the file handling protocol to be used is ‘ftp’, the URI must be provided as ‘ftp://&lt;DIRECTORY_PATH&gt;&gt;’.</p><p><strong>monitoring.interval</strong> : The time duration (in milliseconds) for which the system must monitor changes to the files in the specified directory.</p><p><strong>file.name.list</strong> : If you want to carry out processing for only for one or more specific files in the the given directory URI, you can use this parameter to specify those files as a comma-separated list.</p><h3>Prerequisites :</h3><p>Running instance of WSO2 Streaming Integrator Tooling server. <a href="https://wso2.com/integration/streaming-integrator/">Click here</a> for instructions on how to download and start Streaming Integrator Tooling server.</p><h3>Sample Siddhi Application</h3><pre>@App:name(&#39;FileEventListenerSample&#39;)</pre><pre>@App:description(&#39;Identify whether file or folder has been created, modified or deleted in the given uri and throw events.&#39;)</pre><pre>@source(type = &#39;fileeventlistener&#39;, dir.uri = &quot;/path/to/sweet-factory-logs/&quot;, file.name.list = &quot;test.txt&quot;)<br>define stream FileListenerStream (filepath string, filename string, status string);</pre><pre>@sink(type = &#39;log&#39;)<br>define stream ResultStream (filepath string, filename string, status string);</pre><pre>from FileListenerStream<br>select *<br>insert into ResultStream;</pre><p>This application demonstrates how to configure WSO2 streaming integrator tooling to receive events via fileeventlistener on creation, modification or deletion of a file or directory. Let’s go through this sample line by line.</p><pre>@App:name(&#39;FileEventListenerSample&#39;)<br>@App:description(&#39;Identify whether file or folder has been created, modified or deleted in the given uri and throw events.&#39;)</pre><p>Here <strong>@App:name</strong> gives the name of the Siddhi Application and <strong>@App:description</strong> provide an option to define the application and it is optional.</p><pre>@source(type = &#39;fileeventlistener&#39;, dir.uri = &quot;/path/to/sweet-factory-logs/&quot;, file.name.list = &quot;test.txt&quot;)<br>define stream FileListenerStream (filepath string, filename string, status string);</pre><p><strong>Source</strong> is a contract that consumes data from external sources in the form of events, then converts each event (which can be in XML, JSON, binary, etc. format) to a Siddhi event, and passes that to a Stream for processing. In this scenario source type must be <strong>fileeventlistener</strong> and <strong>dir.uri </strong>can be given by the user. <strong>file.name.list</strong> can be null or a some specific files in that given directory path. If the <strong>file.name.list</strong> is null it will produce events for any actions on the directory. If it is given it only print the actions taken on that given file name. The event will be generated with <strong>filepath</strong> and <strong>filename</strong> and <strong>status</strong>. Here <strong>status</strong> includes 4 types of options such as</p><p><strong><em>STATUS_NEW : </em></strong>A new file or folder is created. “<strong>Created</strong>”</p><p><strong><em>STATUS_PROCESS : </em></strong>A new file or folder is modifying. (If it is a large file or folder) “<strong>Modifying</strong>”</p><p><strong><em>STATUS_DONE : </em></strong>A new file or folder is completely modified.<strong><em> </em></strong>“<strong>Modifying Completed</strong>”</p><p><strong><em>STATUS_REMOVE : </em></strong>A new file or folder is deleted from that folder. “<strong>Removed</strong>”</p><pre>@sink(type = &#39;log&#39;)<br>define stream ResultStream (filepath string, filename string, status string)</pre><p><strong>Sink</strong> is a contract that takes events arriving at a stream, maps them to a predefined data format, and publishes them to external endpoints. In this scenario sink specified as <strong>log</strong> type that produces the event with the attributes <strong>filepath</strong>, <strong>filename</strong> and <strong>status</strong>. According to the action you take on the specific directory.</p><pre>from FileListenerStream<br>select *<br>insert into ResultStream;</pre><p>This final segment of the code describes how events are sent from the input stream to the output stream. In this example, you are selecting all the events from the <strong>FileListenerStream</strong> and sending them to <strong>ResultStream </strong>without checking any conditions.</p><h3>How to run the Sample Application</h3><h3>Prerequisites</h3><ul><li>You can either save this sample in SI tooling (<strong>File -&gt;Save</strong>) or you can find the sample ‘<strong>FileEventListenerSample.siddhi</strong>’ from tooling and then save (<strong>File -&gt; Import Sample</strong>) it in your workspace.</li><li>Create a folder named ‘<strong>sweet-factory-logs</strong>’.</li><li>Add a file named ‘<strong>test.txt</strong>’ in the ‘sweet-factory-logs’ .</li><li>Edit the <strong>dir.uri</strong> to ‘<strong>/path/to/sweet-factory-logs/</strong>’ by replacing with the absolute path of your directory named sweet-factory-logs.</li><li>Save this sample.</li></ul><h3>Executing the Sample</h3><ol><li>Start the Siddhi application by clicking on ‘<strong>Run</strong>’ .</li><li>If the Siddhi application starts successfully, the following message would be shown on the console <br><strong><em>* FileEventListenerSample.siddhi — Started Successfully!</em></strong></li></ol><h3>Testing the Sample</h3><p>Perform the following actions in ‘sweet-factory-logs’ folder that you have already created. These actions will create events.</p><p><strong>I.</strong> Add new lines in the test.txt.<br><strong>II.</strong> Delete the file test.txt</p><h3>Viewing the Results:</h3><ul><li>You will see an event similar to the following when you start the Siddhi app.</li></ul><pre>INFO {io.siddhi.core.stream.output.sink.LogSink} — FileEventListenerSample : ResultStream : Event{timestamp=1584442060224, data=[/path/to/sweet-factory-logs/test.txt, test.txt, created], isExpired=false}</pre><ul><li>The above event shows that test.txt is created.</li><li>For each action performed in the folder <strong>(I, II)</strong> a new event will be thrown.</li></ul><p>The outputs on the console will be similar to the following.</p><pre>INFO {io.siddhi.core.stream.output.sink.LogSink} — FileEventListenerSample : ResultStream : Event{timestamp=1584442060224, data=[/path/to/sweet-factory-logs/test.txt, test.txt, modifyingCompleted], isExpired=false} INFO {io.siddhi.core.stream.output.sink.LogSink} — FileEventListenerSample : ResultStream : Event{timestamp=1584442060224, data=[/path/to/sweet-factory-logs/test.txt, test.txt, removed], isExpired=false}</pre><p>You can figure out how the file path, file name and status of a file or folder changes for every action (delete, update and create) you take in any files given in file.name.list.</p><h3>Conclusion</h3><p>This article described how to use WSO2 Streaming Integrator to get the details of files which have been created or modified or deleted in the execution time using a sample Siddhi application. As mentioned in this example, you can check with any directory uri with any files or folders mentioned in the file.name.list.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c96c8cd9fa7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cron Support for Change Data Capture in Siddhi]]></title>
            <link>https://medium.com/@ajanthy/cron-support-for-change-data-capture-in-siddhi-da157a35d1d2?source=rss-4b55be40f8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/da157a35d1d2</guid>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[siddhi]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[scheduler]]></category>
            <category><![CDATA[wso2]]></category>
            <dc:creator><![CDATA[Ajanthy Jayarajan]]></dc:creator>
            <pubDate>Mon, 29 Jun 2020 11:41:00 GMT</pubDate>
            <atom:updated>2020-06-29T11:41:00.256Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I’m going to discuss on the cron support for Siddhi-io-cdc used in WSO2 Streaming Integrator. For this purpose, I’m using siddhi-io-cdc source extension.</p><p><a href="https://wso2.com/integration/streaming-integrator"><strong>WSO2 Streaming Integrator(SI)</strong></a> is a streaming data processing server that integrates streaming data and takes action based on streaming data. The <strong>CDC source</strong> receives events when change events (i.e., INSERT, UPDATE, DELETE) are triggered for a database table. Events are received in the ‘key-value’ format.</p><p>For CDC, WSO2 Streaming Integrator utilizes a dedicated connector called “<a href="https://siddhi-io.github.io/siddhi-io-cdc/"><strong>siddhi-io-cdc</strong></a>”. This connector provides capabilities to listen to a particular database and emit events for data changes. The connector supports databases such as MySQL, PostgreSQL, Microsoft SQL Server, H2 and Oracle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1PiBERD4ioF5zclu" /></figure><p>There are two modes you could perform Change Data Capture (CDC) : <strong>Listening mode</strong> and <strong>Polling mode</strong>. In polling mode, the datasource is periodically polled for capturing the changes. The polling period can be configured. In polling mode, you can only capture INSERT and UPDATE changes. Here I’m going to define the new feature Cron Support for <a href="https://siddhi-io.github.io/siddhi-io-cdc/"><strong><em>Siddhi-io-cdc</em></strong></a><strong><em>.</em></strong></p><h3>Why do we need this?</h3><p>Traditionally the events which have been inserted in the database will be converted to event as soon as it get inserted in the database. This leads to heavy load and and sometimes we might miss the records, to solve these issues we have scheduled time using cron expression. Cron supports the polling mode which means the events which have been inserted or updated in the database between the time when the app gets starts and the time which satisfies the cron expression, will be polled. This is the use case of siddhi-io-cdc with cron support. To get a clear idea on how cron works go through my<a href="https://medium.com/@ajanthy/cron-expression-5a68526ffb23"> previous article</a>. To get complete idea on how CDC works <a href="https://medium.com/@lasanthas/change-data-capturing-with-wso2-streaming-integrator-45b40fd15c09">Click here.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*PN2V0dWGozy-K1SU8k4aHA.png" /><figcaption>Implementation of Cron Support</figcaption></figure><p>Let’s see how Siddhi-io-cdc works with cron Expression on WSO2 Streaming Integrator Tooling.</p><h3>Prerequisites :</h3><p>Running instance of WSO2 Streaming Integrator Tooling server. <a href="https://wso2.com/integration/streaming-integrator/#">Click here</a> for instructions on how to download and start Streaming Integrator Tooling server.</p><h3>Sample Siddhi Application</h3><pre>@App:name(“CDCWithPollingModeAndCron”)</pre><pre>@App:description(“Capture MySQL Inserts and Updates using cdc source polling mode.”)</pre><pre>@source(type = ‘cdc’,<br> url = ‘jdbc:mysql://localhost:3306/production?useSSL=false’,<br> mode = ‘polling’,<br> jdbc.driver.name = ‘com.mysql.jdbc.Driver’,<br> polling.column = ‘last_update’,<br> polling.interval = ‘1’,<br> username = ‘’,<br> password = ‘’,<br> cron.expression = ‘0 */2 * * * ?’,<br> table.name = ‘SweetProductionTable’,<br> @map(type = ‘keyvalue’ ))<br>define stream insertSweetProductionStream (name string, amount double);</pre><pre>@sink(type = ‘log’)<br>define stream logStream (name string, amount double);</pre><pre>@info(name =‘query’)<br>from insertSweetProductionStream<br>select name, amount<br>insert into logStream;</pre><p>This application demonstrates how to configure WSO2 streaming integrator tooling to publish events using cdc polling mode at given cron expression time. A MySQL table’s <strong>INSERTS</strong> and <strong>UPDATES</strong> are captured by changing the database type, the following databases will also be supported.<br> * Oracle<br> * H2<br> * MS SQL Server<br> * PostgreSQL</p><p>Let’s go through this sample line by line.</p><pre>@App:name(“CDCWithPollingModeAndCron”)<br>@App:description(“Capture MySQL Inserts and Updates using cdc source polling mode.”)</pre><p>Here <strong>@App:name</strong> gives the name of the Siddhi Application and <strong>@App:description</strong> provide option to define the application and it is optional.</p><pre>@source(type = ‘cdc’,<br> url = ‘jdbc:mysql://localhost:3306/production?useSSL=false’,<br> mode = ‘polling’,<br> jdbc.driver.name = ‘com.mysql.jdbc.Driver’,<br> polling.column = ‘last_update’,<br> polling.interval = ‘1’,<br> username = ‘’,<br> password = ‘’,<br> cron.expression = ‘0 */2 * * * ?’,<br> table.name = ‘SweetProductionTable’,<br> @map(type = ‘keyvalue’ ))<br>define stream insertSweetProductionStream (name string, amount double);</pre><p><strong>Source</strong> is a contract that consumes data from external sources in the form of events, then converts each event (which can be in XML, JSON, binary, etc. format) to a Siddhi event, and passes that to a Stream for processing. In this scenario source type is <strong>cdc</strong>. <strong>‘mode’</strong> must set to <strong>‘polling’</strong>. Cron expression</p><pre>cron.expression = ‘0 */2 * * * ?’</pre><p>defines at every even minute the cronjob will be scheduled. If the cron expression satisfies the system time, events with inserted or updated data will be generated according to the insertion or modification happened between the time app get started and the cron satisfied time. In this scenario <strong>last_update</strong></p><pre>polling.column = ‘last_update’,</pre><p>is a timestamp value so it checks for the new event with the change in last_update. If the new value gets inserted or updated there will be a change in timestamp from that it will separate the new event and print the events which have been inserted or updated when cron time matches the system time. The stream <strong>insertSweetProductionStream</strong> has 2 attributes name and amount.</p><pre>@sink(type = ‘log’)<br>define stream logStream (name string, amount double);</pre><p><strong>Sink</strong> is a contract that takes events arriving at a stream, maps them to a predefined data format, and publishes them to external endpoints. In this scenario sink specified as <strong>log</strong> type that produces the event with the attributes name and amount.</p><pre>@info(name = ‘query’)<br>from insertSweetProductionStream<br>select name, amount<br>insert into logStream;</pre><p>This final segment of the code describes how events are sent from the input stream to the output stream. In this example you are selecting all the events from the <strong>insertSweetProductionStream</strong> and sending them to <strong>logStream </strong>without checking any conditions.</p><h3>How to run the Sample Application</h3><h3>Prerequisites</h3><ul><li>Ensure that MySQL is installed on your computer.</li><li>Add the MySQL JDBC driver into {WSO2_SI_HOME}/lib, Install the required jars using the Extensions Installer tool which is available in the editor or navigate to Tools/Extension Installer on Editor menu and search for the `Change Data Capture — MySQL` from the search panel and install.</li><li>Create a database “production”.</li><li>Create table “SweetProductionTable (last_update TIMESTAMP, name VARCHAR(20),amount double(10,2));</li><li>Input relevant values for ‘username’ and ‘password’ in the ‘insertSweetProductionStream’ query.</li><li>Save this sample.</li></ul><h3>Executing the Sample</h3><ul><li>Start the Siddhi application by clicking on ‘Run’.</li><li>If the Siddhi application starts successfully, the following message is shown on the console<br><strong><em> *</em></strong> <strong><em>CDCWithPollingModeAndCron.siddhi — Started Successfully!</em></strong></li></ul><h3>Testing the Sample</h3><ul><li>Insert data to the created table.</li></ul><blockquote><em>insert into SweetProductionTable(name,amount) values(‘chocolate’,100.0);</em></blockquote><blockquote><em>insert into SweetProductionTable(name,amount) values(‘cake’,33.0);</em></blockquote><h3>Viewing the results</h3><ul><li>Observe the inserted data is logged on the console at the time which satisfies the cron expression.</li></ul><pre>[2020–05–19_14–30–00_046] INFO {io.siddhi.core.stream.output.sink.LogSink} — CDCWithPollingMode : logStream : Event{timestamp=1589878800039, data=[chocolate, 100.0], isExpired=false} <br>[2020–05–19_14–32–00_003] INFO {io.siddhi.core.stream.output.sink.LogSink} — CDCWithPollingMode : logStream : Event{timestamp=1589878920003, data=[cake, 33.0], isExpired=false}</pre><p>Optional: Update existing row and obverse the change data events. And observe time matches the cron expression. Polling mode is can also be used with Oracle, MS-SQL server, PostgreSQL, H2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/0*XqybrpB3ifQSQiKQ.jpg" /></figure><h3>Conclusion</h3><p>This article described how to use WSO2 Streaming Integrator to execute a task in siddhi-io-cdc at schedued time using the cron expression. As mentioned in this example, you can check with any datasources and with any tables.</p><h3>References</h3><ul><li><strong>Change Data Capturing with WSO2 Streaming Integrator </strong><a href="https://medium.com/@lasanthas/change-data-capturing-with-wso2-streaming-integrator-45b40fd15c09">https://medium.com/@lasanthas/change-data-capturing-with-wso2-streaming-integrator-45b40fd15c09</a></li><li><strong>Introduction on Cron Expression</strong> <br><a href="https://medium.com/@ajanthy/cron-expression-5a68526ffb23">https://medium.com/@ajanthy/cron-expression-5a68526ffb23</a></li><li><strong>Streaming Integrator Download</strong><br><a href="https://wso2.com/integration/streaming-integrator/">https://wso2.com/integration/streaming-integrator/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da157a35d1d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cron Expression]]></title>
            <link>https://medium.com/@ajanthy/cron-expression-5a68526ffb23?source=rss-4b55be40f8f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a68526ffb23</guid>
            <category><![CDATA[scheduler]]></category>
            <category><![CDATA[cron]]></category>
            <dc:creator><![CDATA[Ajanthy Jayarajan]]></dc:creator>
            <pubDate>Tue, 09 Jun 2020 16:56:42 GMT</pubDate>
            <atom:updated>2020-06-11T05:42:44.455Z</atom:updated>
            <content:encoded><![CDATA[<p>Cron is the name of program that enables Unix users to execute commands or scripts automatically at a specified time/date. This allows users to create a Cron Trigger that can schedule job at time specified in the expression.</p><p>It enables users to schedule tasks to run periodically at a specified date/time. And, it’s naturally a great tool for automating lots of process runs which otherwise would require human intervention.</p><p>Cron-Expressions are strings that are actually made up of seven sub-expressions, that describe individual details of the schedule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*6-dp-3cSKVGez-Tq.png" /></figure><p>With integer values cron expression uses some special characters to represent the scheduling time.</p><p>Special Characters used in cron Expression</p><ul><li><strong>* (all) : </strong>It is used to specify that event should happen for every time unit.</li><li><strong>? (any) : </strong>It is used in the &lt;<em>day-of-month&gt;</em> and &lt;<em>day-of -week&gt; </em>fields to denote the arbitrary value — neglect the field value. Any value can be added to that field.</li><li><strong>– (range) : </strong>It is used to determine range of values.</li><li><strong>/ (increments) : </strong>It is used to specify the incremental values.</li><li><strong>L (last): </strong>It has different meanings when used in various fields. In day-of-month it specifies the last day of month and In the day-of-week, it specifies the last day of week.</li><li><strong>W (weekday) : </strong>It is used to specify the weekday (Monday to Friday) nearest to a given day of the month.</li></ul><p><strong>Seconds</strong>, <strong>Minutes</strong> :</p><ul><li>0–59 can be given as values.</li><li><strong>‘*’</strong> , ‘<strong>,’ , ‘/’ , ‘-’ </strong>can be used as special characters.</li></ul><p><strong>Hours :</strong></p><ul><li>0–23 can be given as values.</li><li><strong>‘*’</strong> , ‘<strong>,’ , ‘/’ , ‘-’ </strong>can be used as special characters.</li></ul><p><strong>Day (Month) :</strong></p><ul><li>1–31 can be given as values.</li><li><strong>‘*’ , ‘,’ , ‘/’ , ‘-’, ‘L’ , ‘W’ </strong>can be used as special characters.</li></ul><p><strong>Month :</strong></p><ul><li>1–12 and JAN-DEC can be allowed as values.</li><li><strong>‘*’ , ‘,’ , ‘/’ , ‘-’ </strong>can be used as special characters.</li></ul><p><strong>Day (Week) :</strong></p><ul><li>0–6 and SUN-SAT can be allowed as values.</li><li><strong>‘*’ , ‘,’ , ‘/’ , ‘-’ , ‘L’ </strong>can be used as special characters.</li></ul><p><strong>Examples :</strong></p><ul><li><strong>0 0 12 * * ?</strong> : Every day at 12.00 PM</li><li><strong>0 */2 * ? * * : </strong>Every even minute</li><li><strong>0 0 12 * * SUN : </strong>Every Sunday at noon</li><li><strong>0 0 12 LW * ? : </strong>Every month on the last weekday at noon</li><li><strong>0 0 12 ? DEC * : </strong>Every day at noon in December</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*N1XX2DqA_jQ10Gxt.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a68526ffb23" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>