<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Niveathika Rajendran on Medium]]></title>
        <description><![CDATA[Stories by Niveathika Rajendran on Medium]]></description>
        <link>https://medium.com/@niveathika?source=rss-6a9fa9c7a519------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Y7vP1o4QlbnYXMTT.jpg</url>
            <title>Stories by Niveathika Rajendran on Medium</title>
            <link>https://medium.com/@niveathika?source=rss-6a9fa9c7a519------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:23:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@niveathika/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[SFMC Internals: A Developer’s Guide — Part 3: Journey Builder 101]]></title>
            <link>https://medium.com/@niveathika/sfmc-internals-a-developers-guide-part-3-journey-builder-101-34720894abb3?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/34720894abb3</guid>
            <category><![CDATA[sfmc]]></category>
            <category><![CDATA[marketing-cloud]]></category>
            <category><![CDATA[salesforce]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Sat, 12 Jul 2025 17:31:03 GMT</pubDate>
            <atom:updated>2025-07-12T17:31:03.724Z</atom:updated>
            <content:encoded><![CDATA[<h3>SFMC Internals: A Developer’s Guide</h3><h4>Part 3: Journey Builder 101 — Orchestrating Engagement</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rr1uUOtk36qEkQeWUgF2iw.jpeg" /></figure><h3>Introduction: Turning Triggers into Journeys</h3><p>So far in this series, we’ve covered the building blocks:</p><ul><li>In <a href="https://medium.com/@niveathika/sfmc-internals-a-developers-guide-part-1-contacts-and-data-extensions-explained-36dc57365b11">Part 1</a>, we explored <strong>Contacts</strong> and <strong>Data Extensions</strong> — how SFMC stores customer data.</li><li>In <a href="https://medium.com/@niveathika/sfmc-internals-a-developers-guide-part-2-entry-events-and-assets-a196fa42f49e">Part 2</a>, we looked at <strong>Entry Events</strong> and <strong>Assets</strong> — how data triggers automation and what content is delivered.</li></ul><p>Now it’s time to bring it all together in <strong>Journey Builder</strong> — the tool that lets you design <strong>multi-step, personalised marketing flows</strong> based on that data.</p><p>In this post, you’ll learn:</p><ul><li>What Journey Builder does</li><li>Key activities you can use inside a journey</li></ul><h3>What is Journey Builder?</h3><p><strong>Journey Builder</strong> is a drag-and-drop tool in SFMC that allows you to create automated workflows — known as <strong>Journeys</strong> — that guide customers through a series of steps based on their behaviour, preferences, or data.</p><p>Each journey can:</p><ul><li>Start when a customer meets a condition (entry event)</li><li>Wait or branch based on time or behavior</li><li>Send messages via email, SMS, or push</li><li>Update or modify contact data</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DynzM6--xR5ee0HAjkKr_g.png" /></figure><h3>Key Components of a Journey</h3><p>A journey is made up of <strong>activities</strong>, which you arrange on a canvas. Here are the most commonly used ones:</p><h4>✅ Entry Source</h4><ul><li>Where the contact comes from (e.g., a Data Extension, API event).</li><li>Configured when you set up the journey.</li></ul><h4>✅ Send Email / SMS / Push</h4><ul><li>Sends a message to the contact using a predefined <strong>asset</strong>.</li><li>You can personalise content using fields from the <strong>Entry Data</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x5ukCr0g0SWZsFxvs1Qaow.png" /></figure><h4>✅ Wait</h4><p>Delays the next step by:</p><ul><li>A fixed amount of time (e.g., wait 2 days)</li><li>Until a specific date (e.g., send on birthday)</li></ul><h4>✅ Decision Split</h4><p>Branch the journey based on data or behaviour.</p><p>Examples:</p><ul><li>If contact’s location = India → Send campaign A</li><li>If opened last email → Send follow-up B</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*-G_dN2n0U7oUKSn-VPrGSQ.png" /></figure><h4>✅ Update Contact / Set Attribute</h4><ul><li>Changes or updates data in a DE or contact record.</li><li>Useful for tagging status like “Welcome Email Sent”</li></ul><h4>✅ Goal &amp; Exit Criteria</h4><ul><li>Define what success looks like (e.g., a click or purchase)</li><li>Exit criteria can stop the journey early for certain contacts</li></ul><h3>Real-World Example: Welcome Journey</h3><p>Let’s walk through a basic welcome journey:</p><ol><li>Contact fills out a signup form on your website.</li><li>Info is saved to a <strong>NewSubscribers</strong> DE.</li><li>A <strong>Data Extension Entry Event</strong> starts the journey.</li><li>SFMC sends a <strong>Welcome Email</strong> (using a personalised asset).</li><li>The journey <strong>waits 2 days</strong>.</li><li>If the user <strong>clicks the email</strong>, they get an onboarding offer.<br> If not, they get a reminder email.</li></ol><h3>Summary</h3><p>With Journey Builder, you can:</p><ul><li>Trigger journeys using <strong>Entry Events</strong></li><li>Deliver content using <strong>Assets</strong></li><li>Use conditions and data to personalise the flow</li><li>Create a scalable, automated experience for every contact</li></ul><p>Together with Contacts, Data Extensions, and Assets, <strong>Journey Builder completes the foundation</strong> of how Salesforce Marketing Cloud automates communication at scale.</p><p>This brings us to the end of the <strong>SFMC Internals: A Developer’s Guide</strong> series.<br> Whether you’re just getting started or brushing up on the basics, we hope this gave you a clear, practical understanding of how SFMC works under the hood.</p><p><strong><em>Thanks for reading!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34720894abb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SFMC Internals: A Developer’s Guide Part 2: Entry Events and Assets]]></title>
            <link>https://medium.com/@niveathika/sfmc-internals-a-developers-guide-part-2-entry-events-and-assets-a196fa42f49e?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/a196fa42f49e</guid>
            <category><![CDATA[salesforcemarketingcloud]]></category>
            <category><![CDATA[marketing-cloud]]></category>
            <category><![CDATA[salesforce]]></category>
            <category><![CDATA[sfmc]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Sat, 12 Jul 2025 16:02:55 GMT</pubDate>
            <atom:updated>2025-07-14T05:46:55.010Z</atom:updated>
            <content:encoded><![CDATA[<h3>SFMC Internals: A Developer’s Guide</h3><h4>Part 2: Entry Events and Assets — Starting the Conversation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*fqSde8AJXv28VYMW7pEboA.png" /></figure><h3>Introduction: From Data to Action</h3><p>In <a href="https://medium.com/@niveathika/sfmc-internals-a-developers-guide-part-1-contacts-and-data-extensions-explained-36dc57365b11">Part 1 of this series</a>, we explored how Salesforce Marketing Cloud (SFMC) stores information about your contacts using <strong>Data Extensions (DEs)</strong>. Now that your data is in place, the next step is putting it to use by actually sending messages to your customers.</p><p>But how does SFMC know <strong>when</strong> to send a message? Or <strong>which contact</strong> should it send it to?</p><p>The answer lies in two important concepts:</p><ul><li><strong>Entry Events</strong>, which trigger a contact’s journey</li><li><strong>Assets</strong>, which are the actual content sent during that journey</li></ul><p>In this post, we’ll explain both — with simple examples and visuals — so you can see how SFMC turns raw data into action.</p><h3>What is an Entry Event?</h3><p>An <strong>Entry Event</strong> is the starting point of a <strong>Journey</strong> in SFMC. It tells the system:</p><ul><li>Who should enter the journey</li><li>When should they enter</li><li>What data should they bring with them</li></ul><p>Think of it like a gate that opens when a certain condition is met, such as when a new contact is added to a Data Extension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*jUejQP6zcEmal9gTqqbO6w.png" /></figure><h3>Common Entry Event Types</h3><p>SFMC supports several types of entry events, but here are the most common:</p><ol><li><strong>Data Extension Entry Event</strong></li></ol><ul><li>Triggered when a new row is added to a specific DE</li><li>Ideal for: signups, newsletter opt-ins, event registrations</li></ul><p><strong>2. API Entry Event</strong></p><ul><li>Triggered via API call from an external system</li><li>Ideal for: real-time actions like purchases or transactions</li></ul><p><strong>3. CloudPages Form Submission</strong></p><ul><li>Starts a journey when a user fills and submits a web form</li></ul><p><strong>4. Salesforce Data Entry</strong></p><ul><li>Triggers from changes in Salesforce CRM data (if integrated)</li></ul><p>📌 For most beginners, <strong>Data Extension Entry Events</strong> are the easiest and most commonly used.</p><h3>Entry Source vs Entry Data</h3><p>This is an important concept:</p><ul><li><strong>Entry Source</strong> is <strong>where</strong> the contact is coming from — usually a specific Data Extension.</li><li><strong>Entry Data</strong> is <strong>what information</strong> from that source that will travel into the journey.</li></ul><p>Why does this matter? Because only the fields in the <strong>entry data</strong> can be used inside the journey, for personalisation or decision-making.</p><h3>Understanding Re-entry Settings</h3><p>By default, a contact can only enter a journey once, but you can customise this behaviour.</p><p>There are three <strong>re-entry options</strong>:</p><ol><li><strong>No re-entry — </strong>Contact enters only once, ever.</li></ol><p><strong>2. Re-entry any time — </strong>Contact can re-enter multiple times, even immediately.</p><p><strong>3. Re-entry only after exiting — </strong>Contact can enter again, but only after their previous journey finishes.</p><p>📌 Example:<br> If someone signs up twice in one day, re-entry settings determine whether they’ll get two welcome emails or just one.</p><h3>What Are Assets in SFMC?</h3><p><strong>Assets</strong> are the content elements used inside your emails, SMS, and other messages. They’re created and managed using <strong>Content Builder</strong>.</p><p>Types of assets include:</p><ul><li><strong>Email templates</strong></li><li><strong>Text blocks</strong></li><li><strong>Images and logos</strong></li><li><strong>Buttons, footers, and headers</strong></li><li><strong>AMPscript blocks</strong> for personalisation</li></ul><p>You build assets once and reuse them across journeys, automations, and campaigns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6cZIMBky_M9DAOkGD7MStA.jpeg" /></figure><h3>Using Assets Inside a Journey</h3><p>When you design a journey in <strong>Journey Builder</strong>, you use activities like <strong>Send Email</strong> to communicate with your contacts. Each of these activities uses one of your <strong>Assets</strong>.</p><p>For example:</p><ul><li>A “Welcome Email” activity will use an email asset that includes a subject line, body content, and personalisation fields like “Hi {{FirstName}}”.</li></ul><p>Assets can be static or dynamic, and they help you maintain consistency across your messaging.</p><h3>Real-World Example: Welcome Journey Flow</h3><p>Let’s walk through a simple scenario:</p><ol><li>A user signs up for your newsletter on your website.</li><li>Their information is saved in a Data Extension called NewSubscribers.</li><li>A <strong>Data Extension Entry Event</strong> is listening to that DE.</li><li>When a new row appears, the <strong>Welcome Journey</strong> starts.</li><li>The journey sends a personalised email using a <strong>Welcome Email Asset</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/219/1*X9iWzlVdvoRmtOn0so8JAg.png" /></figure><h3>Summary</h3><p>Let’s recap what you’ve learned:</p><ul><li><strong>Entry Events</strong> are how contacts <strong>start</strong> their journeys in SFMC.</li><li>They monitor your data and kick off automation when specific conditions are met.</li><li><strong>Entry Data</strong> determines which fields are available in the journey.</li><li><strong>Re-entry settings</strong> control how often someone can go through the journey.</li><li><strong>Assets</strong> are the <strong>content</strong> (emails, images, templates) used in messages sent through journeys.</li></ul><p>By combining data, triggers, and content, SFMC helps you send the right message to the right person at the right time.</p><h3>What’s Next: Orchestrating Journeys</h3><p>Now that you understand <strong>how journeys start</strong> and <strong>what gets sent</strong>, it’s time to explore how to <strong>design the journey itself</strong>.</p><p>In<a href="https://medium.com/@niveathika/sfmc-internals-a-developers-guide-part-3-journey-builder-101-34720894abb3"> Part 3 of the series</a>, we’ll walk through <strong>Journey Builder</strong> — the tool you’ll use to build multi-step, automated flows that guide each customer through a personalised experience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a196fa42f49e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SFMC Internals: A Developer’s Guide — Part 1 Contacts and Data Extensions Explained]]></title>
            <link>https://medium.com/@niveathika/sfmc-internals-a-developers-guide-part-1-contacts-and-data-extensions-explained-36dc57365b11?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/36dc57365b11</guid>
            <category><![CDATA[sfmc]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[salesforce]]></category>
            <category><![CDATA[marketing-cloud]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Fri, 11 Jul 2025 17:17:45 GMT</pubDate>
            <atom:updated>2025-07-14T05:45:41.756Z</atom:updated>
            <content:encoded><![CDATA[<h3>SFMC Internals: A Developer’s Guide</h3><h4>Part 1: Contacts and Data Extensions Explained</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NSl7unKB_tdtLCAlLc-vQ.png" /></figure><h3>Introduction: What is Salesforce Marketing Cloud?</h3><p>Salesforce Marketing Cloud (SFMC) is a platform that enables businesses to send <strong>personalised messages</strong> to their customers across various channels, including email, SMS, push notifications, and more. Whether it’s a welcome email, a birthday coupon, or a follow-up after a purchase, SFMC makes it possible to send the right message at the right time.</p><p>But before you can send anything, <strong>SFMC needs to know who your customers are</strong>, what they’ve done, and what messages they should receive. That’s where the concepts of <strong>Contacts</strong> and <strong>Data Extensions</strong> come in.</p><p>In this first part of the series, we’ll look at:</p><ol><li>What is a <strong>Contact</strong> in SFMC?</li><li>What is a <strong>Data Extension</strong>?</li><li>How do they work together to power messaging and automation?</li></ol><h3>What is a Contact?</h3><p>A <strong>Contact</strong> in SFMC represents a single person you may want to communicate with. It could be someone who signed up for your newsletter, a customer who placed an order, or a lead from your website.</p><p>Each contact is tracked using a <strong>Contact Key</strong> — a unique identifier that helps SFMC recognise the person across different channels (email, SMS, push, etc.).</p><p>📌 <strong>You choose what the Contact Key is</strong> — it might be an email address, a customer ID, or even a randomly generated UUID. The important thing is that it’s unique and consistent.</p><p>You might also hear the term <strong>Subscriber Key</strong> — this is used in Email Studio and is often the same as the Contact Key, though they technically serve different roles under the hood.</p><p>The following diagram showcases the internal structure of a Contact in Salesforce Marketing Cloud, highlighting how attributes, attribute sets, and attribute groups are organised.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rHK8fCuRC4JaKu-G" /></figure><h3>What is a Data Extension (DE)?</h3><p>A <strong>Data Extension</strong> is a table that stores data about your contacts, like a spreadsheet or database table inside SFMC.</p><p>You define the structure of a DE: what columns it has, what data types they use, and which column is used as the Contact Key. This gives you full flexibility to store the exact data you need.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/0*9ZIOJHJD-ZwClpF6" /></figure><p>A simple table layout diagram showing a DE with fields: Contact Key, Email, First Name, and Signup Date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EDNogMsxmyYtxKkMlL6zCg.png" /></figure><p>This kind of data structure allows SFMC to send personalised messages (e.g., “Hi Alex”) and to automate actions based on dates or behaviour.</p><h3>Types of Data Extensions</h3><p>There are different types of DEs depending on how they’re used:</p><ul><li><strong>Sendable DEs</strong>: Used to send emails or SMS. These must include a <strong>Contact Key</strong> and an <strong>email address or phone number</strong>.</li><li><strong>Non-Sendable DEs</strong>: Used to store extra info like purchase history or preferences — not directly used to send messages.</li></ul><h3>Data Extensions vs Lists</h3><p>You might also come across <strong>Lists</strong>, which are simpler versions of contact storage in SFMC. Lists work well for basic email sends and small audiences, but they have limitations.</p><p><strong>Data Extensions offer more flexibility and control</strong>, especially when you want to:</p><ul><li>Store custom fields (like age, region, and signup source)</li><li>Trigger automations</li><li>Create multiple DEs for different journeys</li></ul><p>📌 <strong>Most modern SFMC implementations use Data Extensions, not Lists.</strong></p><h3>How DEs and Contacts Connect in SFMC</h3><p>So far, we’ve talked about <strong>Contacts</strong> and <strong>Data Extensions</strong> separately. Now let’s connect them.</p><p>In SFMC, your contact data lives in Data Extensions, but how SFMC understands those relationships is managed in a tool called <strong>Contact Builder</strong>.</p><p>In Contact Builder, you can define <strong>Attribute Groups</strong>, which help organise and link different Data Extensions back to the main contact record.</p><p>This lets you:</p><ul><li>Store a single contact once</li><li>Enrich their profile with data from multiple sources</li><li>Use that data for personalisation and targeting</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKfHlr1W34lvcla3TXcyJA.png" /></figure><h3>Real-World Example: A Welcome Journey</h3><p>Let’s say someone signs up on your website. You capture their details in a Data Extension called NewSubscribers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EDNogMsxmyYtxKkMlL6zCg.png" /></figure><p>That DE is <strong>sendable</strong>, which means SFMC can use it to:</p><ol><li>Start an <strong>automated journey</strong> (like sending a welcome email).</li><li>Personalise that email with their name.</li><li>Track engagement (open, click, etc.).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*-upwsFzV3NqzXVNKGspUkw.png" /></figure><h3>Summary</h3><ul><li>A <strong>Contact</strong> is a unique person in SFMC, identified by a <strong>Contact Key</strong></li><li>A <strong>Data Extension</strong> is a flexible table where you store data about your contacts</li><li>Together, they form the foundation for personalised and automated messaging</li><li>Understanding this relationship is the first step to using SFMC effectively</li></ul><h3>A Peek Ahead: Entry Events and Journeys</h3><p>Now that you understand Contacts and Data Extensions, you’re ready for the next step: <strong>Journeys</strong>.</p><p>In SFMC, Journeys are automated workflows that begin with an Entry Event, which triggers the journey to start (e.g., when someone is added to a DE).</p><p>In <a href="https://medium.com/@niveathika/sfmc-internals-a-developers-guide-part-2-entry-events-and-assets-a196fa42f49e">Part 2 of the series</a>, we’ll explore:</p><ul><li>What <strong>Entry Events</strong> are</li><li>How they use Data Extensions</li><li>And what <strong>Assets</strong> (like email templates) are needed to build a full journey</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36dc57365b11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Salesforce to S/4 HANA Integration with Ballerina: Part 2]]></title>
            <link>https://medium.com/ballerina-techblog/salesforce-to-s-4-hana-integration-with-ballerina-part-2-1da0054db9de?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/1da0054db9de</guid>
            <category><![CDATA[ballerinalang]]></category>
            <category><![CDATA[salesforce]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[s4-hana]]></category>
            <category><![CDATA[sap]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Tue, 16 Jul 2024 04:46:21 GMT</pubDate>
            <atom:updated>2024-07-22T04:05:25.318Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zn_n51ULg3BlUfGppzPYBw.png" /></figure><p>This is part of a series on implementing Salesforce to S/4HANA integration in <a href="http://ballerina.io/?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=devrel&amp;utm_term=salesforces/4hana">Ballerina</a>. To start from the beginning, go to <a href="https://medium.com/@niveathika/salesforce-to-s-4-hana-integration-with-ballerina-part-1-cd2d8a20e196">Part 1</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vfgXkryjjAduFduy" /></figure><p>In this part, we’ll use the data gathered from Salesforce to create a Sales Order in S/4HANA.</p><h3>S/4HANA API</h3><p>We’ll be using the Sales Order (A2X) ODATA API to create the Sales Order. The <a href="https://central.ballerina.io/ballerinax/sap/latest#setup-guide">SAP Setup Guide</a> in the Ballerina documentation provides detailed instructions for setting up this API in S/4HANA.</p><h3>Ballerina S/4HANA Sales Order Connector</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lOFRmEgG33UEXy8K" /></figure><p>When looking up the API in the Business Accelerator Hub, note the service name ‘API_SALES_ORDER_SRV’ in the URL. This unique name identifies the API within SAP.</p><p>Use the service name to look up the ballerina package for the Sales Order API in <a href="https://central.ballerina.io/search?q=api_sales_order_srv&amp;sort=relevance%2CDESC&amp;page=1&amp;m=packages">Ballerina Central</a>,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ko596oMP23ja5a_S" /></figure><h3>Step 1: Import and Initialise the SAP S/4HANA Sales Order Client</h3><p>First, import the ‘sap.s4hana.api_sales_order_srv’ connector and initialise the client.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5677ce3f0ad7a8d0ac4067c2ac0c1d40/href">https://medium.com/media/5677ce3f0ad7a8d0ac4067c2ac0c1d40/href</a></iframe><h3>Step 2: Create a Sales Order</h3><p>We will use the ‘createA_SalesOrder()’ function from the ‘sap.s4hana.api_sales_order_srv’ package.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93a0b968ae9837b8c01bc230656ea4cb/href">https://medium.com/media/93a0b968ae9837b8c01bc230656ea4cb/href</a></iframe><h3>Step 3: Test the Integration</h3><p>You can find the complete code in <a href="https://github.com/ballerina-platform/module-ballerinax-sap.s4hana.sales/tree/main/examples">module-ballerinax-sap.s4hana.sales</a> repository.</p><p>Now it’s time to test the code!</p><ol><li>Add the credentials to ‘Config.toml’.</li><li>Run the example:</li></ol><pre>bal run</pre><p>When you close an opportunity on Salesforce, you should see the logs in the terminal where you are running the Ballerina code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*3SZHNbvJbx5G4GXdiVR-qA.png" /></figure><p>Similarly, in S/4HANA, you should see the Sales Order created with the specified ID.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ibQWznxi1mfjSPGv" /></figure><h3>Conclusion</h3><p>You’ve successfully completed the integration journey from Salesforce to S/4HANA using Ballerina. We hope you found this guide useful. Stay tuned for more tutorials and examples!</p><p>Feel free to leave your comments and questions below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1da0054db9de" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ballerina-techblog/salesforce-to-s-4-hana-integration-with-ballerina-part-2-1da0054db9de">Salesforce to S/4 HANA Integration with Ballerina: Part 2</a> was originally published in <a href="https://medium.com/ballerina-techblog">Ballerina Swan Lake Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Salesforce to S/4 HANA Integration with Ballerina: Part 1]]></title>
            <link>https://medium.com/ballerina-techblog/salesforce-to-s-4-hana-integration-with-ballerina-part-1-cd2d8a20e196?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd2d8a20e196</guid>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[salesforce]]></category>
            <category><![CDATA[ballerinalang]]></category>
            <category><![CDATA[sap]]></category>
            <category><![CDATA[s4-hana]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Tue, 16 Jul 2024 04:19:21 GMT</pubDate>
            <atom:updated>2024-09-14T06:52:31.600Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zn_n51ULg3BlUfGppzPYBw.png" /></figure><p>This article was written using Ballerina Swan Lake Update 9.0 (2201.9.0).</p><h3>Why Ballerina for Integration?</h3><p>Ballerina is a programming language designed specifically for integration. It features unique abstractions that incorporate network aspects directly into the language, making network services, endpoints, and actions first-class citizens. This makes it easy to write integration scenarios in Ballerina.</p><p>We will automate the creation of sales orders in S/4HANA when an opportunity is closed with Won state in Salesforce.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FknHxDfcAdjjwOlk9fFb0w.png" /></figure><p>In this first part of our two-part series, we’ll explore how to build a Salesforce listener using Ballerina. By the end of this post, you’ll have a functional Salesforce listener that captures opportunity changes and extracts data to identify events with the “Closed Won” state.</p><h3>Prerequisites</h3><p>Before we dive into the code, make sure you have the following prerequisites:</p><ol><li><strong>Ballerina</strong>: Installed on your system. You can download it from <a href="http://ballerina.io/?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=devrel&amp;utm_term=salesforces/4hana">ballerina.io.</a></li><li><strong>Salesforce Account</strong>: Access to Salesforce with necessary API permissions.</li><li><a href="https://central.ballerina.io/ballerinax/trigger.salesforce/0.10.0#prerequisites"><strong>Subscribe to Salesforce Change Capture Event</strong></a></li></ol><h3>Step 1: Initialise a Ballerina Project</h3><p>Create a new Ballerina project using the following commands:</p><pre>bal new salesforce-to-sap<br>cd salesforce-to-sap</pre><p><strong>Note</strong>: The ‘main.bal’ will contain a sample “Hello World” program. You can delete it and get started on the project.</p><h3>Step 2: Import Salesforce Module and Initialize the Listener</h3><p>We will use the ballerinax/salesforce package to implement the listener. As the first step, initialize the listener.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60b23963d45c2604c45b65757b6446ec/href">https://medium.com/media/60b23963d45c2604c45b65757b6446ec/href</a></iframe><h4>Optional: Store Credentials Securely</h4><p>In Ballerina, credentials can be stored in ‘Config.toml’ for security. Ballerina will load the credentials at runtime.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ff026aa1619754cd490bb99aaa6dbef/href">https://medium.com/media/7ff026aa1619754cd490bb99aaa6dbef/href</a></iframe><h4>Step 3: Attach salesforce Service to Salesforce Listener which listens to OpportunityChangeEvent</h4><p>Now that we have initialized the listener, we need to access the changed event. Attach a “/data/OpportunityChangeEvent” to the listener. A service represents a collection of remotely accessible methods attached to a particular listener. Here, the Service has functions designed for Salesforce events.</p><p>The channel name for capturing opportunity object changes is “/data/OpportunityChangeEvent”. This <a href="https://developer.salesforce.com/docs/atlas.en-us.platform_events.meta/platform_events/platform_events_objects_change_data_capture.htm">Salesforce documentation</a> elaborates more on how these channel names are derived.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/546227c633fa6ad27afa43e3d4de36f0/href">https://medium.com/media/546227c633fa6ad27afa43e3d4de36f0/href</a></iframe><h3>Step 4: Implement the onUpdate Function to Filter Opportunity Close Won Events</h3><p>For this integration, we are only interested in the onUpdate function. Let’s see how it can be implemented to filter only opportunity won events.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0077e609f79eef9e7b86bcdab73d990d/href">https://medium.com/media/0077e609f79eef9e7b86bcdab73d990d/href</a></iframe><h3>Step 5: Retrieve Opportunity Items</h3><p>So far, we have captured specific events. However, this event does not include all the items the opportunity contains. These are not included because this will be finalized beforehand. The OpportunityLineItem table in Salesforce keeps track of all products added to an opportunity.</p><p>Use the following SOQL query to retrieve all product details in the opportunity that was closed:</p><pre>SELECT ProductCode, Name, Quantity <br>FROM OpportunityLineItem <br>WHERE OpportunityId=&lt;id&gt;;</pre><p>Ballerina’s ‘ballerinax/salesforce’ package provides the client to run the above query.</p><h3>Step 6: Initialise Salesforce Client</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15269b135ee6de177a93fceca5765fbc/href">https://medium.com/media/15269b135ee6de177a93fceca5765fbc/href</a></iframe><p>The <a href="https://central.ballerina.io/ballerinax/salesforce/8.0.2#setup-guide">Salesforce Setup Guide</a> in Ballerina documentation provides detailed steps to obtain the required tokens.</p><h3>Step 7: Execute the SOQL Query</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c26b342a2ce8f81cdc1467a4cf19288b/href">https://medium.com/media/c26b342a2ce8f81cdc1467a4cf19288b/href</a></iframe><p>Results will be bound to the SfOpportunityItem Ballerina record for easy manipulation.</p><h3>Full Code So Far</h3><p>Here’s the complete code up to now,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0bdc47e557249640856cd33d0692582b/href">https://medium.com/media/0bdc47e557249640856cd33d0692582b/href</a></iframe><h3>Conclusion</h3><p>In this part, we’ve set up a Salesforce listener using Ballerina that listens for opportunity change events and retrieves the necessary opportunity items. We’ve covered how to configure the listener and client, handle events, and query Salesforce for data.</p><p>In <a href="https://medium.com/@niveathika/salesforce-to-s-4-hana-integration-with-ballerina-part-2-1da0054db9de">part 2,</a> we’ll explore how to use the Ballerina S/4 HANA Sales Order Connector to create a sales order from the extracted details.</p><p>Feel free to leave your comments and questions below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd2d8a20e196" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ballerina-techblog/salesforce-to-s-4-hana-integration-with-ballerina-part-1-cd2d8a20e196">Salesforce to S/4 HANA Integration with Ballerina: Part 1</a> was originally published in <a href="https://medium.com/ballerina-techblog">Ballerina Swan Lake Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Adventure into S/4 HANA Cloud: A Beginner’s Perspective Part 1]]></title>
            <link>https://medium.com/@niveathika/my-adventure-into-s-4-hana-cloud-a-beginners-perspective-part-1-52a20db3ac11?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/52a20db3ac11</guid>
            <category><![CDATA[s4-hana]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[sap]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Sun, 04 Feb 2024 11:38:39 GMT</pubDate>
            <atom:updated>2024-02-06T09:14:41.203Z</atom:updated>
            <content:encoded><![CDATA[<h3>My Adventure into S/4 HANA Cloud: A Beginner’s Perspective</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oq0ef45d2Cy9NLOpWUnFoQ.jpeg" /></figure><p>Ever dived into something completely out of your depth? That was me, plunging headfirst into the realm of S/4 HANA ERP systems last month. Yes, I expected some challenges, even some learning curve. What awaited me was a nightmare, filled with long daunting YouTube tutorials that assumed I had a base of knowledge I simply did not possess. This blog is a live diary of my exploration through the intricacies of ERP systems, capturing both the hurdles and breakthroughs.</p><p>Welcome to my journey through the world of S/4 HANA.</p><h3>What is S/4 HANA?</h3><p>ChatGPT will say something like this,</p><blockquote><strong>SAP S/4HANA is an enterprise resource planning (ERP) software suite developed by SAP (Systems, Applications, and Products in Data Processing).</strong></blockquote><p>Let’s break this down further.</p><h3>ERP Software — A Practical Example</h3><p>Picture this: My homeland, Sri Lanka, is renowned for exporting apparel to leading EU brands. “SL Apparels” has a sales order to fulfill, 1,000 jeans by the end of the month. Now, multiply this demand by a thousand. The sheer volume of requests becomes unmanageable through conventional means like email. This is where “ERP” systems comes into play. This software lets the production manager to plan the company’s resource — be it the production line, workforce, or procurement of materials. This example, focused to the manufacturing sector, illustrates the ERP’s pivotal role across various industries.</p><p>SAP’s flagship, S/4 HANA, is not just any ERP solution. It’s a powerhouse capable of processing vast data volumes at lightning speed, offering real-time insights that are a game-changer for businesses across sectors. Whether it’s streamlining operations in healthcare, revolutionizing retail experiences, or enhancing efficiency in the manufacturing sector, S/4 HANA stands as a versatile tool tailored to meet diverse industry challenges.</p><p>In the retail industry, for instance, consider a multinational chain with hundreds of outlets globally. S/4 HANA’s capability to integrate data from all stores, manage inventory instantaneously, and unlock consumer behavior insights empowers retailers to customize their offerings, streamline supply chains, and elevate customer satisfaction.</p><h4>Understanding the Basics</h4><p>Before we leap into the nitty-gritty of S/4 HANA, getting acquainted with some foundational concepts is crucial:</p><ul><li><strong>UI Home screen</strong>: Your gateway to the world of S/4 HANA.</li><li><strong>Modules and Submodules:</strong> The building blocks of your ERP solution.</li><li><strong>Organizational Structure</strong>: The blueprint of your business within S/4 HANA.</li><li><strong>Master Data</strong>: The core information driving your processes.</li><li><strong>Communication Scenarios and APIs</strong>: The bridges connecting S/4 HANA to the world.</li></ul><p>As my exploration progresses, I aim to delve into the specific functionalities, hurdles, and victories of employing S/4 HANA across different industries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52a20db3ac11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Ballerina SQL Clients: Your Go-To Guide ]]></title>
            <link>https://medium.com/ballerina-techblog/mastering-ballerina-sql-clients-your-go-to-guide-77411de18b37?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/77411de18b37</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[ballerina]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Fri, 01 Dec 2023 13:21:10 GMT</pubDate>
            <atom:updated>2024-01-05T06:11:51.096Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yc1Xkqjw0Ez4kgd6Kl0_xQ.jpeg" /></figure><p><em>This article was written using Ballerina Swan Lake Update 8 (2201.8.0).</em></p><p>Hey there, coding maestros! Let’s dive into the secrets of making the most out of your <a href="http://ballerina.io/?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=devrel&amp;utm_term=sqlclients">Ballerina</a> SQL clients — whether it’s the classic `mysql:Client`, the sassy `mssql:Client`, the trusty `postgresql:Client`, the mighty `oracledb:Client`, or the evergreen `jdbc:Client`. Here’s the lowdown on leveling up your SQL game with these savvy recommendations. 🎩💻</p><p><strong>1. Stick with One `sql:Client` Throughout</strong></p><p>Employing a single SQL client throughout the application execution is pivotal for efficient connection management. The ballerina/sql module utilizes connection pooling, optimizing the opening and closing of database connections. This practice minimizes the time spent on these operations and conserves network resources. 🕺🏽</p><p><strong>2. Don’t Ghost the Stream</strong></p><p>Ensure thorough consumption of fetch data or promptly close the stream following processing from the query method. The connection to the database persists within the stream until fully iterated or in the event of an iteration error. Timely closure of the database connection after query execution prevents resource loss over time, maintaining a consistent connection pool.</p><p>3. <strong>Close Those sql:ProcedureCallResult</strong></p><p>Unlike streams, the sql module can’t read minds. If you’re processing with `sql:ProcedureCallResult`, remember to send out the “party’s over” memo by calling ProcedureCallResult.close().</p><p>4. <strong>Utilize sql:ParameterizedQuery for Dynamic Queries</strong></p><p>Use sql:ParameterizedQuery to dodge those pesky SQL injection moves. For an in-depth guide, check out <a href="https://medium.com/@niveathika/guarding-against-sql-injection-in-ballerina-applications-%EF%B8%8F-a20e642f0e49">Guarding Against SQL Injection in Ballerina Applications</a>.</p><p>5. <strong>Configure the </strong><a href="https://central.ballerina.io/ballerina/sql/latest#ConnectionPool"><strong>Connection Pool</strong></a><strong> of the Client</strong></p><p>Last but not least, let’s talk about connection pool management. Configure the connection pool settings and tailor them to your app’s personality. Detailed insights into pool sizing configurations can be found in <a href="https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing">About Pool Sizing</a>.</p><p>So, there you have it — your backstage pass to rocking Ballerina SQL clients like a coding rockstar. Happy coding! 🚀✨</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77411de18b37" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ballerina-techblog/mastering-ballerina-sql-clients-your-go-to-guide-77411de18b37">Mastering Ballerina SQL Clients: Your Go-To Guide 🚀</a> was originally published in <a href="https://medium.com/ballerina-techblog">Ballerina Swan Lake Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Guarding Against SQL Injection in Ballerina Applications ️]]></title>
            <link>https://medium.com/ballerina-techblog/guarding-against-sql-injection-in-ballerina-applications-%EF%B8%8F-a20e642f0e49?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/a20e642f0e49</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[ballerina]]></category>
            <category><![CDATA[sql-injection]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Tue, 28 Nov 2023 06:35:02 GMT</pubDate>
            <atom:updated>2024-01-05T05:54:10.125Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Database security. Photo by &lt;a href=”https://unsplash.com/@growtika?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash&quot;&gt;Growtika&lt;/a&gt; on &lt;a href=”https://unsplash.com/photos/a-blue-and-white-logo-ahgsuFHlIFo?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash&quot;&gt;Unsplash&lt;/a&gt;" src="https://cdn-images-1.medium.com/max/1024/1*-6M-eZigllj0_jtYpS_DVg.jpeg" /></figure><p><em>This article was written using Ballerina Swan Lake Update 8 (2201.8.0).</em></p><p>In the realm of database-backed programs, one of the foremost security concerns is SQL Injection. This perilous scenario unfolds when a dynamic SQL query incorporates user inputs.</p><p>To shield against SQL Injections, <a href="https://ballerina.io/?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=devrel&amp;utm_term=sqlinjectionblog">Ballerina</a> employs the Prepared Statement method.</p><p>Check this out:</p><pre>sql:ParameterizedQuery = `SELECT * FROM users WHERE user_name=${username};`</pre><blockquote><em>Tip: User inputs are seamlessly inserted into the query with </em><em>${...}.</em></blockquote><p>Now, let’s break down how this query is executed:</p><ol><li><strong><em>Template the Query:</em> </strong>The query starts as SELECT * FROM users WHERE user_name=${username}. The sql module swaps the &quot;${}&quot; placeholders with “?”. All insertions are treated as user inputs – no surprise attacks here!</li><li><strong><em>Compile the Query: </em></strong>The database double-checks the SQL query for syntax and semantics, making sure everything is legit. The server gives it a thumbs-up for execution.</li><li><strong><em>Placeholder Replacement: </em></strong>User data values are sent to the database, swapping placeholders with real values. Since the query is pre-compiled, no matter what values are added, it won’t turn a SELECT into a DELETE. Take that, SQL Injection!</li><li><strong><em>Query Execution: </em></strong>Boom! The processed query runs, and we get the results we want.</li></ol><p>Now, in the Ballerina universe, some devs ask:</p><pre>Why can&#39;t we template the table name in the query? 🤔<br>Why not add order by key as placeholder? 🤔</pre><p>Hold up! Table names and order by keys are like VIPs in SQL queries. They gotta be there in Step 2 for a smooth compilation. Thats why, sql:ParameterizedQuery only allows parameters in the query; the rest gotta stay put.</p><p>But hey, does this mean we can’t jazz up a query dynamically in Ballerina? Nah, we can, but it’s not a walk in the park. Our advice? Play it safe with an allow list when you’re crafting dynamic queries in code.</p><p>SQL identifiers you toss into the query better be on the allow list. Any user input going off-script? Boom! Throw an error. Check this Ballerina snippet:</p><pre>sql:ParameterizedQuery orderUsersPartialQuery = `SELECT * FROM users ORDER BY `;<br>sql:ParameterizedQuery orderUsers = ``;<br>string orderByKey = &quot;&quot;;<br>match orderByKey {<br>    &quot;username&quot; =&gt; {<br>        orderUsers = sql:queryConcat(orderUsersPartialQuery, `username`);<br>    }<br>    &quot;age&quot; =&gt; {<br>        orderUsers = sql:queryConcat(orderUsersPartialQuery, `age`);<br>    }<br>    _ =&gt; {<br>        return error(&quot;Invalid order by key&quot;);<br>    }<br>}</pre><p>Keep rocking those queries, and say no to SQL Injection! 🎸✨</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a20e642f0e49" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ballerina-techblog/guarding-against-sql-injection-in-ballerina-applications-%EF%B8%8F-a20e642f0e49">Guarding Against SQL Injection in Ballerina Applications 🛡️</a> was originally published in <a href="https://medium.com/ballerina-techblog">Ballerina Swan Lake Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Siddhi YAML Config Manager]]></title>
            <link>https://medium.com/siddhi-io/introducing-siddhi-yaml-config-manager-5499625a99e?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/5499625a99e</guid>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[siddhi]]></category>
            <category><![CDATA[yaml]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Fri, 18 Oct 2019 13:03:30 GMT</pubDate>
            <atom:updated>2019-10-25T16:03:06.599Z</atom:updated>
            <content:encoded><![CDATA[<p>Siddhi can run as an embedded Java or Python library, microservice on bare-metal, VM, or Docker, and natively in Kubernetes. With Siddhi 5.1 release a common YAML based configuration can be used to pass system and extension configurations across all types of deployment. In this article, we’ll see how the new common YAML based configuration can be applied to Siddhi running as an embedded Java library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*ARkRrUks-2_LrrQhZWdHVQ.png" /></figure><p>These system and extension configurations that are relevant to the execution environment can be passed at runtime allowing the same Siddhi App to work on various environments without any changes. These configurations can be passed to SiddhiManager via SiddhiConfigManagers at initialization.</p><p>In the next sections, we’ll see the configurations that can be passed to Siddhi Apps at runtime, and how we can write Java API based on the configurations with the new YAML based approach.</p><h3><strong>Siddhi Configuration Types</strong></h3><p>Siddhi Manager allows setting of three types of siddhi configurations through Config Manager as follows,</p><h4>Extension System Parameters</h4><p>Some extension properties that are common across all instances of the extension can be set as <strong>extension system parameters. </strong>See the below siddhi definition as an example,</p><pre>@Store(type=&quot;rdbms&quot;, jdbc.url=&quot;jdbc:mysql://localhost:3306/foo&quot;,  <br>       username=&quot;root&quot;, password=&quot;root&quot;,  <br>       jdbc.driver.name=&quot;com.mysql.jdbc.Driver&quot;)<br>define table FooTable(name string, foo string);</pre><p>This defines a FooTable table that uses MySQL as the backend store. Here, the RDBMS store uses some default properties such as connection timeout which is set to 60000 sec. Let’s say we need to override this to 80000 sec, due to the network condition of the environment, we can simply set this as an extension system parameter such that the property does not need to be defined in each Siddhi store RDBMS annotation.</p><h4>References</h4><p>The references allow some sections of source, sink, and store configuration to be passed at runtime. This helps to pass different configuration on various environments such as using MySQL database on one environment and using Oracle on another, without changing the Siddhi Application logic.</p><p>Consider the same table example, this specific store configuration can be defined as a reference and used in Siddhi Apps as follows,</p><pre>@Store(ref=&#39;fooDBStore&#39;)<br>define table FooTable(name string, foo string);</pre><h4>Siddhi Properties</h4><p>Lastly, Siddhi <a href="https://siddhi.io/en/v5.1/docs/query-guide/#named-aggregation%5C">Named aggregation</a> expects some properties related to Distributed Aggregations such as shardId and partitionById that should be passed at runtime.</p><p>Let’s say that we need to set 2 extension system parameters for RDBMS store, 1 reference, and 2 Siddhi properties. In the following sections, we’ll see how we can pass them via InMemoryConfigManager and YAMLConfigManager.</p><h3>Usage of existing InMemoryConfigManager</h3><p>Up to Siddhi 5.1, the only way of passing configurations is via the InMemoryConfigManger. As it only accepts Maps, we can initialize 3 maps and pass these three types of configurations as follows,</p><pre>Map&lt;String, String&gt; extensionConfigs = new HashMap&lt;&gt;();<br>extensionConfigs.put(&quot;store.rdbms.mysql.batchEnable&quot;, &quot;true&quot;);<br>extensionConfigs.put(&quot;store.rdbms.mysql.batchSize&quot;, &quot;1000&quot;);</pre><pre>Map&lt;String, String&gt; references = new HashMap&lt;&gt;();<br>references.put(&quot;fooDBStore.jdbc.url&quot;,     <br>                                 &quot;jdbc:mysql://localhost:3306/foo&quot;);<br>references.put(&quot;fooDBStore.username&quot;, &quot;root&quot;);<br>references.put(&quot;fooDBStore.password&quot;, &quot;root&quot;);<br>references.put(&quot;fooDBStore.jdbc.driver.name&quot;, <br>                                  &quot;com.mysql.jdbc.Driver&quot;);</pre><pre>Map&lt;String, String&gt; properties = new HashMap&lt;&gt;();<br>properties.put(&quot;partitionById&quot;, &quot;true&quot;);<br>properties.put(&quot;shardId&quot;, &quot;shard1&quot;);</pre><pre>InMemoryConfigManager inMemoryConfigManager = new <br>    InMemoryConfigManager(extensionConfigs, references, properties);</pre><pre>SiddhiManager siddhiManager = new SiddhiManager();<br>siddhiManager.setConfigManager(inMemoryConfigManager);</pre><p>As seen above, this needs repetitive coding to initiate the Config Manager, and this also needs code change when deploying in different environments.</p><p>Now let’s check how the same properties can be passed via YAML Config Manager.</p><h3>Usage of YAMLConfigManager</h3><p>From Siddhi 5.1 onwards, the YAMLConfigManager can be used to pass the same configuration via YAML as follows,</p><pre># Siddhi Extensions system Parameters<br>extensions:<br>  - extension:<br>      name: rdbms<br>      namespace: store<br>      properties:<br>        mysql.batchEnable: true<br>        mysql.batchSize: 1000</pre><pre># Siddhi references<br>refs:<br>  - ref:<br>      name: fooDBStore<br>      type: rdbms<br>      properties:<br>        jdbc.url: jdbc:mysql://localhost:3306/foo<br>        username: root<br>        password: root<br>        jdbc.driver.name: com.mysql.jdbc.Driver</pre><pre>properties:<br>  partitionById: TRUE<br>  shardId: shard1</pre><p>Let’s say this file is saved as siddhi.yaml.</p><p>This can be initialized, configured and passed to SiddhiManager via YAMLConfigManager with only 4 lines of code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/363c5b9a13ee712e50e736c18003a9e0/href">https://medium.com/media/363c5b9a13ee712e50e736c18003a9e0/href</a></iframe><p>In conclusion, YAML Config Manager comes as a convenient and efficient way to set Siddhi Configurations to Siddhi Java library and enables Siddhi Apps to efficiently migrate among various environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5499625a99e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/siddhi-io/introducing-siddhi-yaml-config-manager-5499625a99e">Introducing Siddhi YAML Config Manager</a> was originally published in <a href="https://medium.com/siddhi-io">siddhi-io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Siddhi 5.1.1 reduces Database Query Latency upto 3 times]]></title>
            <link>https://medium.com/siddhi-io/siddhi-5-1-1-reduces-database-query-latency-by-60-36632d507374?source=rss-6a9fa9c7a519------2</link>
            <guid isPermaLink="false">https://medium.com/p/36632d507374</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[siddhi]]></category>
            <category><![CDATA[complex-event-processing]]></category>
            <dc:creator><![CDATA[Niveathika Rajendran]]></dc:creator>
            <pubDate>Mon, 29 Jul 2019 12:28:06 GMT</pubDate>
            <atom:updated>2019-08-03T18:13:45.043Z</atom:updated>
            <content:encoded><![CDATA[<h3>Siddhi 5.1 reduces Database Query Latency up to 3 times</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*3lcY86n-pgO0vbJ9YEfhVw.png" /></figure><p>Stream processing use cases have a need to process millions of events per second. However, not all data can be stored in memory or attached as stream attributes. Hence, the need for getting additional data from databases introduces to the use case of executing traditional database queries in the stream processing pipeline.</p><p>Siddhi allows working with stored data through its feature called Tables.</p><pre>define table Purchase (purchaseId string, dateOfPurchase long, <br>                       customerId string, country string, <br>                       totalAmount float);</pre><p>Here, the tables can be in-memory or backed with traditional databases. Siddhi gives support to connect with traditional DBs such as MySQL, Postgres, MSSQL and Oracle, as well as NoSQL DBs such as MongoDB. However, the drawback of table query is the latency introduced to the processing pipeline especially when the tables are backed by traditional databases. <strong>Even though this cannot be avoided, from Siddhi Core 5.1.1 this has been tweaked to perform at the lowest possible latency along with efficient memory management.</strong></p><p>This blog talks about the architecture behind the Siddhi table join, the changes introduced, and the performance evaluation.</p><h3>Sample Usecase</h3><p>Before delving into the architecture, let’s see the sample use case for the explanation. Consider that we have the following DB table in the system which records customer transaction, and we have a use case to query for the top 10 customers in terms of their total purchases originated from different countries for the month of July.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*ws7HOG0yEdy7hEBbOL9EqQ.png" /><figcaption>Purchase Table</figcaption></figure><p>This can be represented in Siddhi as follows,</p><pre>@PrimaryKey(“purchaseId”)<br>define table Purchase (purchaseId string, dateOfPurchase long,  <br>                       customerId string, country string,   <br>                       totalAmount float);</pre><p>By default, this will be an empty in-memory table. Here, with the addition of “@store” annotation, this table can be configured to mirror a traditional DB. For instance, the following definition mirrors a MySQL DB.</p><pre>@Store(type=&quot;rdbms&quot;, jdbc.url=&quot;jdbc:mysql://localhost:3306/stocks&quot;,<br>       username=&quot;root&quot;, password=&quot;root&quot;,  <br>       jdbc.driver.name=&quot;com.mysql.jdbc.Driver&quot;)<br>@PrimaryKey(&quot;purchaseId&quot;)<br>define table Purchase (purchaseId string, dateOfPurchase long, <br>                       customerId string, country string, <br>                       totalAmount float);</pre><p>As we have a use case to query for the top 10 customers in terms of their total purchases originated from different countries for the month of July, we use TriggerStream that is configured with an HTTP source via “@source” annotation to trigger the processing.</p><pre>@source(type=&#39;http&#39;, receiver.url=&#39;http://localhost:5005/trigger&#39;,         <br>        @map(type = &#39;json&#39;))<br>define stream TriggerStream (startTimeStamp long, endTimestamp long,<br>                             country string);</pre><p>The query used to get top 10 customers will be as follows,</p><pre>from TriggerStream as T join Purchase as P<br>     on P.country == T.country and <br>        P.dateOfPurchase &gt; T.startTimestamp and <br>        P.dateOfPurchase &lt; T.endTimestamp<br>select P.customerId as customer, sum(P.totalAmount) as totalAmount <br>group by P.customerId <br>order by totalAmount desc <br>limit 10 <br>insert into OutputStream;</pre><p>Here, the on condition on line 2 represents the filters, i.e country in the table should match the country value sent through HTTP request and dateOfPurchase must be between startTimestamp and endTimestamp sent through each HTTP request. The clauses select, group by, order by and limit are similar to DB SQL queries.</p><p>The final Siddhi app will be as follows,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aed264ce7db148d85ea557dda6aeaecb/href">https://medium.com/media/aed264ce7db148d85ea557dda6aeaecb/href</a></iframe><h3>Table Join Architecture</h3><p>The internal architecture for Table joins queries as seen below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uq-Nu9NkSybtVg6AkBGVSA.png" /></figure><p>Here, Table Processor is the interface used to run the find query against the table. All Siddhi tables have to be implemented using <strong>AbstractRecordTable </strong>which has the function find() through which the on condition (where clause) can be passed to underlying DB.</p><p>The DB SQL Query for the above use case will be as follows, this is compiled once during the runtime and reused upon each event arrival.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f2fc700c96843b3524d53fab91eea30/href">https://medium.com/media/0f2fc700c96843b3524d53fab91eea30/href</a></iframe><p>Then, Query Selector processes the queried events by performing group by, having, limit and offset as well as final stream attributes transformations in-memory.</p><p>Recently, a new table interface, <strong>AbstractQueryableRecordTable</strong> was introduced to Siddhi. This interface gives support to pass the selectors to the DB for processing. In essence, using its query() function we can pass the work of Query Selector to the DB itself and skip the Query Selector processing such as performing group by, having, limit, offset and attributes transformations in-memory. This optimized processing has been achieved through the following architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0zApyeZY5omziKmLyRWc0A.png" /></figure><p>The corresponding DB SQL Query will be,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/599ffb236a9d15672a929b08647689e6/href">https://medium.com/media/599ffb236a9d15672a929b08647689e6/href</a></iframe><p>With the new implementation, the Table Processor takes a decision to use the optimized flow over the normal flow when the following two conditions are satisfied,</p><ol><li>The underlying table(store) implements the interface <strong>AbstractQueryableRecordTable.</strong></li><li>The table(store) supports all the select functions used in the query.</li></ol><p>To understand the function support, let’s say in the query we have an ifThenElse function to rename the customer id to Unknown if it is equal to 0. The Siddhi select query having such ifThenElse function will be as follows,</p><pre>select <br> ifThenElse(P.customerId==&#39;0&#39;, &#39;Unknown&#39;, P.customerId) as customer,<br> sum(P.totalAmount) as totalAmount</pre><p>Here, when the selection is compiled (via compileSelection() method), the underlying implementation of the table(store) can throw an exception to indicate that the specific function cannot be executed natively in the DB violating the 2nd condition to use the optimization.</p><p>In such situations, the Table Processor will revert to the original flow to maintain backward compatibility. However, when both the conditions are satisfied, Siddhi passes the Query Selector’s work on to the physical database resulting in not loading a lot of data to the memory and reduces memory consumption. In the meantime performing group by and other operations on DB level enables the system to use the DB’s query optimizations and indexes and thus reduces the overall latency.</p><h3>Performance Evaluation</h3><p>The performance was evaluated by running the above discussed Siddhi query on a table with 100,000 records with a varying number of the unique keys combinations present in the records. I.e the Purchase table was tested with 100000 records, among which it has 10000 unique customer entries, then with 25000 unique customer entries and so on. The tests for a specific number of unique combinations was repeated 10 times to get average to reduce anomalies.</p><p>The tests were carried out in, 4 CPU, 16 Gb machine against MySQL 5.7.26</p><p>The latency measurement can be seen below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*2JmM5ChRvjxKdl32sIqm4g.png" /></figure><p>When group by is 1, the latency reduced from 227ms to 90 ms which is 2.5 times faster than before. The latency for 100,000 combinations reduced from 380 ms to 128 ms, giving a 66% latency improvement i.e time taken has reduced by almost 3 times. As seen above the latency for queries when group-bys increase was much steeper before the optimization, and after Siddhi Core 5.1.1 the latency hovered around 90–130 ms giving a 60–66 % improvement (2.5 to 3 times faster performance compared to previous versions). Further, the latency was reduced from the get-go and it is far more prominent when the DB contains a large number of unique customers i.e when group by has a large number of unique combinations.</p><h3><strong>Things to note</strong></h3><p>The optimization will produce an advantage when it can cut down a large amount of data that is being retrieved to perform the group by operation on the memory. Even though this behavior is executed by default, we still have to be careful when writing queries to ensure that the query can be passed to the underlying DB. For instance, for the query where we need to select top K users while renaming customer Ids having 0 to Unknown. Instead of writing the query as bellow,</p><pre>@info(name = ‘get_top_10_users’) <br>from TriggerStream as T join Purchase as P <br>on P.country == T.country and P.dateOfPurchase &gt; T.startTimestamp <br>                          and P.dateOfPurchase &lt; T.endTimestamp<br>select <br> ifThenElse(P.customerId==’0&#39;, ‘Unknown’, P.customerId) as customer,<br> sum(P.totalAmount) as totalAmount <br>group by P.customerId <br>order by totalAmount desc <br>limit 10 <br>insert into OutputStream;</pre><p>Where the optimization may fail because Siddhi’s ifThenElse function in select clause may not be supported by RDBMS to be processed by the DB level, and thus the system may revert to its standard behavior.</p><p>However, by rewriting the query to run the selection after results are taken from the DB as below, we will be able to get the advantage of the optimization.</p><pre>@info(name = ‘get_top_10_users’) <br>from TriggerStream as T join Purchase as P <br>    on P.country == T.country and <br>       P.dateOfPurchase &gt; T.startTimestamp and <br>       P.dateOfPurchase &lt; T.endTimestamp<br>select P.customerId as customer, sum(P.totalAmount) as totalAmount <br>group by P.customerId <br>order by totalAmount desc <br>limit 10 <br>insert into TempStream;</pre><pre>@(info=’CustomerId_Validation’)<br>From TempStream <br>Select <br> ifThenElse(P.customerId==’0&#39;, ‘Unknown’, P.customerId) as customer,<br> totalAmount <br>Insert into OutputStream;</pre><h3>Conclusion</h3><p>Siddhi Core 5.1.1 improves database query performance up to 3 times along with efficient query runtime memory usage. To utilize this performance gain underlying Siddhi store must be implemented with <strong>AbstractQueryableRecordTable</strong> and support all selection functions used in the query. If not, the Siddhi join query will revert to its standard(previous) behavior.</p><p>With this improvement, Siddhi stream processing will have lower latency and improved the overall throughput when having DB interactions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36632d507374" width="1" height="1" alt=""><hr><p><a href="https://medium.com/siddhi-io/siddhi-5-1-1-reduces-database-query-latency-by-60-36632d507374">Siddhi 5.1.1 reduces Database Query Latency upto 3 times</a> was originally published in <a href="https://medium.com/siddhi-io">siddhi-io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>