<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jayana_Gunaweera🙂 on Medium]]></title>
        <description><![CDATA[Stories by Jayana_Gunaweera🙂 on Medium]]></description>
        <link>https://medium.com/@Jayana_Gunaweera?source=rss-2c76dcf502f7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PvqHhcb_GcLw81t6ElHfJg.jpeg</url>
            <title>Stories by Jayana_Gunaweera🙂 on Medium</title>
            <link>https://medium.com/@Jayana_Gunaweera?source=rss-2c76dcf502f7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 06:31:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Jayana_Gunaweera/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unleashing the Power of Automation: How wso2 Identity Server Migration Tool Can Revolutionize…]]></title>
            <link>https://blog.stackademic.com/unleashing-the-power-of-automation-how-wso2-identity-server-migration-tool-can-revolutionize-206df35e0d3?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/206df35e0d3</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[wso2is]]></category>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Sun, 28 Dec 2025 06:25:47 GMT</pubDate>
            <atom:updated>2026-01-02T01:01:31.083Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Unleashing the Power of Automation: How wso2 Identity Server Migration Tool Can Revolutionize Product Migration Testing</strong></p><p>By: Jayana Gunaweera</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*gJf8FuDyV6NQTT26" /></figure><p><em>Image source: </em><strong><em>©</em></strong><em>Jayana Gunaweera</em></p><h3>Abstract</h3><h3>Product migration testing is an essential process that ensures the seamless transition of applications and data from one environment to another. However, this process can be time-consuming, error-prone, and resource-intensive when done manually. To address these challenges, this article proposes an approach to automate product migration testing in wso2 Identity Server. The proposed approach enables developers and testers to automate the entire product migration testing process, reducing time and costs while improving accuracy and efficiency.</h3><h3>Introduction</h3><p>Smooth sailing or rough waters ahead? Discover the importance of software product migration testing to ensure a seamless transition to a new environment without any data loss or corruption!</p><p>Let me paint a clear picture for you if you’re unsure about what software product migration testing entails in the context of wso2 Identity Server:</p><p>wso2 Identity Server is an open-source identity and access management product that allows organizations to manage user identities and access to resources. Let’s say that you have been using wso2 Identity Server version 5.11.0 on-premises for several years, and you’ve decided to migrate to version 6.0.0. However, you’re concerned about potential risks associated with the migration, such as data loss, security breaches, or compatibility issues.</p><p>To mitigate these risks, you decide to conduct software product migration testing. You start by creating a detailed testing plan that outlines the scope of the testing, the types of tests you will perform, and the expected outcomes. You then set up a test environment that replicates the production environment as closely as possible, using the same software, hardware, and network configurations.</p><p>Next, you begin the migration testing process for wso2 Identity Server. You test the migration process to ensure that all data, including user data, user roles, tenants,user stores,service providers and permissions, is migrated accurately and without any loss or corruption. You also test the functionality of wso2 Identity Server in the new environment to ensure that all features and functionalities are working as expected. For example, you might test authentication and authorization functionality, Single Sign-On (SSO), and Multi-Factor Authentication (MFA).</p><p>You also perform performance testing to ensure that the system can handle the same load as in the previous environment, and you conduct regression testing to ensure that the migration process has not introduced any new defects or issues.</p><p>Once you have completed the testing process, you analyze the results and make any necessary adjustments to the migration plan. You then migrate wso2 Identity Server version 5.11.0 to version 6.0.0, confident that the software product migration testing has ensured a smooth and successful transition without any negative impact on the system’s performance or security.</p><h4>Why is it important for organizations to upgrade to newer versions of wso2 Identity Server?</h4><p>Migrating from an older version of wso2 Identity Server to a new version is essential for several reasons:</p><p>1.Security: Newer versions of wso2 Identity Server often include security enhancements and bug fixes that address vulnerabilities found in older versions. Migrating to a new version ensures that the application is secure and that sensitive data is protected.</p><p>2.Improved Functionality: Newer versions of wso2 Identity Server include additional features, performance improvements, and bug fixes that enhance the functionality of the application. Migrating to a new version ensures that the application is up to date and that users have access to the latest features.</p><p>3.Support and Maintenance: wso2 provides support and maintenance for its software products, including Identity Server. Migrating to a new version ensures that the application is still supported and that any issues that arise can be addressed by the vendor.</p><p>4.Compatibility: As other software components, such as databases, operating systems, and third-party libraries change, compatibility issues may arise with older versions of wso2 Identity Server. Migrating to a new version ensures compatibility with these components and avoids any issues that may arise due to incompatibility.</p><p>5.Performance: Newer versions of wso2 Identity Server include performance improvements that can enhance the application’s overall performance. Migrating to a new version ensures that the application is optimized for performance and can handle increased traffic and load.</p><p>6.Compliance: Newer versions of wso2 Identity Server compliant with the latest industry standards and regulations. Migrating to a new version ensures that the application remains compliant with these standards and regulations.</p><p><strong>Why automate? Need of automating product migration testing in wso2 Identity Server</strong></p><p>Manual product migration testing is a complex process that requires careful planning, execution, and validation to ensure that the application is working as expected after migration. Testers need to be aware of the challenges involved and use a combination of manual and automated testing to achieve maximum test coverage and ensure the overall quality of the application.</p><p>Need of automating product migration testing can be understood by doing a manual migration.</p><p><strong>A detailed guide on how to do a manual migration in wso2 Identity Server</strong></p><p>Migrating to a newer version of wso2 Identity Server can be a complex process, but following these steps can help guide you through the process of doing a manual migration:</p><p>Assume that you want to manually migrate wso2 IS 5.11( your current wso2 Identity Server version) to wso2 IS 6.0.0 (a newer Identity Server version)</p><p>Pre-migration Planning:</p><p>Before starting the migration process, it is important to plan and prepare for the migration. This involves identifying the current system architecture, business requirements, and any constraints that may impact the migration. It is also important to assess the risks involved in the migration and define a migration strategy and timeline.</p><p>Install the new version:</p><p>Download and install the new version of wso2 Identity Server. Download the wso2 IS 6.0.0 and migration client zip files and extract them.In this scenario, this step involves downloading the wso2 IS 6.0.0 and migration client zip files from the wso2 website and extracting them to a local directory on your machine.Install it on the same machine or a different one, depending on your needs. Make sure to follow the installation instructions provided in the wso2 documentation.Be conscious about your OS type and install a compatible pack from:</p><p>wso2 website:<a href="https://wso2.com/identity-server/#">https://wso2.com/identity-server/#</a></p><p>Or else via</p><p>wso2 github account:<a href="https://github.com/wso2/product-is/releases">https://github.com/wso2/product-is/releases</a></p><p>Backup the configuration:</p><p>Before proceeding with the migration, it is important to backup the existing wso2 Identity Server configuration. This includes all configuration files, databases, and other relevant data. It is important to ensure that the backup is comprehensive and complete.This step involves taking a backup of the existing database to ensure that in case anything goes wrong during the migration process, you can always revert to the previous state.</p><p>Configure the new installation:</p><p>Once the new version of wso2 Identity Server is installed, configure it according to your requirements. This includes setting up the database, configuring the identity providers, and configuring other necessary components.</p><ul><li>Make changes to the deployment.toml file in wso2 IS 6.0.0 according to your requirements: The deployment.toml file in wso2 IS 6.0.0 contains various configuration settings for the server. In this step, you need to modify the file to suit your specific requirements.</li><li>Copy user stores, tenants, jar files, and .jks files from the old wso2 IS version to the new wso2 IS version: This step involves copying all the necessary files and folders, including user stores, tenants, jar files, and .jks files from the old wso2 IS version to the new wso2 IS version.</li><li>Copy the Migration-resources folder from the migration client to the root folder of the new wso2 IS: This step involves copying the Migration-resources folder from the migration client to the root folder of the new wso2 IS version.</li><li>Make necessary changes to the migration-config.yaml file located at /migration resources/migration-config.yaml: This step involves opening the migration-config.yaml file located in the migration client directory and making changes to the file based on your specific migration needs, such as the location of the old and new servers and the location of the database.</li><li>Copy the Jar files from the migration client to the dropins folder of the new wso2 IS: This step involves copying the Jar files from the migration client to the dropins folder of the new wso2 IS version.</li><li>Configure the connection pool to commit pending transactions on connection return. Navigate to either one of the following locations based on your OS. — On Linux/Mac</li></ul><p>OS: &lt;IS_HOME&gt;/bin/wso2server.sh/ -</p><p>On Windows: &lt;IS_HOME&gt;\bin\wso2server.bat</p><p>Add the following JVM option:</p><p>-Dndatasource.disable.rollbackOnReturn=true\</p><ul><li>Do necessary database configurations: you may need to execute some database scripts against your database according to your database type.</li></ul><p>Migrate the data:</p><p>Once the new installation is configured, migrate the data from the old installation to the new one. Run the migration client to initiate the migration process.This step involves running the migration client from the command line to start the migration process.</p><p>Navigate to the bin folder in the new wso2 IS and start the latest Identity Server by executing the following command if you are a linux/mac user.<br><strong>sh wso2server.sh -Dmigrate -Dcomponent=identity</strong></p><p>If you are a windows user, start the wso2server.bat file with the above command.</p><p>This includes migrating user data, groups, roles, and other relevant data. Depending on the size of your database, this may take some time.</p><p>If the migration executed successfully without errors in the console and got the below message, is it enough?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*P0xZLScx0K2Vkbgl" /></figure><p><em>Image source: </em><strong><em>©</em></strong><em>Jayana Gunaweera</em></p><p>Is the migration successful now? My dear curious reader, if you’re wondering whether the migration was successful, I must regretfully inform you that the answer is a resounding “no.” Alas, we cannot breathe a sigh of relief just yet. But do not lose heart! There is much more to uncover about this intricate and nuanced topic. So,I urge you to continue pursuing this article with fervor and curiosity, for there are still discoveries to be made.</p><p>Test the new installation and data validation:</p><p>Doing a migration without printing errors at the console is not enough.Once the migration is complete, it is important to thoroughly test the new wso2 Identity Server installation to ensure that it is fully functional and meets the business requirements. This includes testing all features and functionality, as well as validating the configuration and data.Start the migrated Identity Server and perform a database validation. This step involves performing a database validation to ensure Ex: You can generate an access token from a registered service provider in the Migrated Identity Server to ensure that the migration was successful and that all data has been migrated correctly and successfully.</p><p>Post-migration Monitoring:</p><p>After the deployment is complete, it is important to monitor the new wso2 Identity Server installation to ensure that it is functioning as expected. This involves monitoring system performance, security, and functionality over time to identify any issues or bugs that may arise.</p><p>Verify the migration by generating a report and checking the CPU health: This step involves generating a migration report to verify that the migration was successful and checking the CPU health to ensure that the new wso2 IS version is running smoothly.</p><p>So now we understand that manual product migration testing can be a complex and time-consuming process, especially when it comes to testing. It’s like trying to move all your furniture from one house to another and test, but instead of furniture, it’s code. And instead of a house, it’s a computer.But let’s face it, who wants to spend hours moving furniture when you can just hire movers, right? Well, that’s what we’re doing with our Automated Product Migration Testing. We’re the movers of the code world!</p><p><strong>Role of Migration Client Of wso2 Identity Server</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/0*rKcZf3iq3R_l9rWo.png" /></figure><p><em>Image source: </em><strong><em>©</em></strong><em>Jayana Gunaweera</em></p><p>wso2 Identity Server (IS) Migration Client is a tool designed to simplify and automate the migration of data and configurations from one instance of wso2 Identity Server to another. This client is specifically created for users who need to upgrade their Identity Server version or move their data to a different environment.</p><p>The Migration Client allows for seamless data transfer between the source and target environments, reducing the risk of errors and data loss during the migration process. It supports the migration of various Identity Server artifacts, such as users, roles, permissions, OAuth clients, and certificates.</p><p>The client operates on a command-line interface (CLI), which offers users greater flexibility and control over the migration process. Users can customize their migration operations by configuring a set of options and parameters that suit their specific requirements.</p><p>The Migration Client is built on top of the wso2 Carbon platform, which ensures that the migration process is reliable, scalable, and secure. Additionally, it provides robust error handling and logging capabilities, enabling users to troubleshoot any issues that may arise during the migration process.</p><p><strong>The potential role of Migration Client in facilitating the process of migration.</strong></p><p>In simple terms, migration is like moving from a cozy apartment to a fancy mansion. You would want to make sure all your stuff, like your collection of “World’s Best Child” mugs and your rolex watch collection, made it to the new place in one piece.</p><p>But instead of going through each box yourself, you could program a robot named “Miggy the Migrator” to do it for you.</p><p>Miggy would roll around the new mansion, checking each box and making sure everything is there, while you sit back and relax with a glass of wine. And, if Miggy found something missing, it would notify you right away so you can go and retrieve it.</p><p>In short, automating product migration testing is like having a trusty robot do the heavy lifting for you, so you can sit back, relax and enjoy the move (or software upgrade) with a nice drink. Instead of a robot, here we use our migration client built by wso2 to do the task.</p><p><strong>Need of automating product migration testing in wso2 Identity Server</strong></p><p>Considering above described cons of manual migration; automating product migration testing in wso2 Identity Server is essential for several reasons:</p><ol><li>Time and Cost Savings: Manual testing of product migration can be time-consuming and costly. Automating the testing process can save time and resources, reduce the risk of human error, and improve testing accuracy.</li><li>Improved Efficiency: Automating the product migration testing process using wso2 Identity Server can improve efficiency by enabling repetitive tests to be run automatically, freeing up resources to focus on other critical tasks.</li><li>Consistency: Automation ensures consistency in testing, which is crucial for identifying any issues that may arise during the migration process.</li><li>Better Test Coverage: Automated testing can cover more test scenarios than manual testing, which can help to identify issues that may not be detected through manual testing.</li><li>Scalability: Automated testing allows for scalability testing, which can help to identify any issues related to scalability that may arise during the migration process.</li><li>Regression Testing: Automated testing can also help to ensure that the migrated application continues to function as expected after migration by enabling regression testing.</li><li>Doing a migration testing using wso2 Identity Server can help to ensure that the migrated data is secure and that the access control policies are working as expected.</li></ol><p>For a seamless upgrade process and optimal experience, you can contact wso2 Support for assistance in upgrading wso2 Identity Server with minimal complications.Click below link for further details.</p><p><a href="https://is.docs.wso2.com/en/6.0.0/deploy/upgrade/upgrade-wso2-is/#get-started">https://is.docs.wso2.com/en/6.0.0/deploy/upgrade/upgrade-wso2-is/#get-started</a></p><p><strong>Story continues..</strong></p><p>You did it! You’ve now become a pro at manual product migration testing in wso2 Identity Server. But why spend your precious time and energy on manual labor when you can sit back, relax, and let automation do the heavy lifting?</p><p>In my next article, I’m excited to share with you a step-by-step guide on how to automate product migration testing in wso2 Identity Server using the wso2 Identity Server Migration Client. This guide will take you through the process of automating your migration testing and ensure a smooth transition to your new system.If you’re curious about how to make your migration process stress-free and seamless, then be sure to check out my upcoming article. It’s time to embrace the power of automation and let wso2 migration client do the work for you!</p><p>Let’s Say goodbye to the days of manually checking and hello to the convenience of automation.</p><p><strong>References:<br></strong><a href="https://is.docs.wso2.com/en/latest/">https://is.docs.wso2.com/en/latest/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=206df35e0d3" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/unleashing-the-power-of-automation-how-wso2-identity-server-migration-tool-can-revolutionize-206df35e0d3">Unleashing the Power of Automation: How wso2 Identity Server Migration Tool Can Revolutionize…</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ballerina CSV Whisper Agent️]]></title>
            <link>https://blog.stackademic.com/ballerina-csv-whisper-agent-%EF%B8%8F-6f89f357083b?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f89f357083b</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[ballerinalang]]></category>
            <category><![CDATA[ballerina]]></category>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Sat, 01 Nov 2025 01:49:19 GMT</pubDate>
            <atom:updated>2026-01-16T18:01:21.270Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How I built a Data Agent for CSV analysis using Ballerina and OpenAI</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4_lgpNb6DmXcAdXeBrEFmw.png" /><figcaption>Ballerina CSV Whisper Agent</figcaption></figure><h3>The Idea 💡</h3><p>Ever wanted to just <em>ask</em> your CSVs questions instead of writing complex formulas? That’s the power of the Ballerina CSV Whisper Agent.</p><p><strong>Instead of this:</strong></p><pre>SELECT AVG(salary) FROM employees WHERE department = &#39;Engineering&#39;</pre><p><strong>Do this. Chat with Your CSV Files</strong>📊</p><pre>&quot;What&#39;s the average salary in Engineering?&quot;</pre><h3>What It Does?</h3><ol><li><strong>Upload</strong> any CSV file</li><li><strong>Ask questions</strong> in plain English</li><li><strong>Get AI-powered answers</strong> with insights</li><li><strong>Chat</strong> with your data like it’s a colleague</li></ol><h3>The Tech Stack</h3><ul><li><strong>Backend</strong>: Ballerina (yes, that Ballerina!)</li><li><strong>AI Brain</strong>: OpenAI GPT-4o</li><li><strong>Frontend</strong>: Vanilla JavaScript (zero frameworks!)</li><li><strong>Architecture</strong>: RESTful microservices</li></ul><h3>Why Ballerina? 🤔</h3><p>Three reasons I fell in love with Ballerina for this project:</p><h3>1. Network-Native</h3><pre>resource function post upload(http:Request req) returns json {<br>    // That&#39;s it. REST endpoint defined!<br>}</pre><h3>2. Built-in Safety</h3><pre>lock {<br>    csvStore[fileId] = csvData;  // Thread-safe by default<br>}</pre><h3>3. OpenAI Integration</h3><pre>final chat:Client chatClient = check new ({<br>    auth: { token: openaiApiKey }<br>});<br>// One line. That&#39;s the whole setup.</pre><h3>The High Level Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*MbpRIuU19ROtsnSZu9Csew.jpeg" /><figcaption>High Level Architecture Diagram of Ballerina CSV Whisper</figcaption></figure><p>Simple. Effective. Fast.</p><h3>Code Highlights 💻</h3><h3>CSV Upload</h3><pre>resource function post upload(http:Request req) returns json {<br>    var bodyParts = req.getBodyParts();<br>    mime:Entity csvPart = getCsvPart(bodyParts);<br>    <br>    string csvContent = &#39;string:fromBytes(csvPart.getByteArray());<br>    string[] headers = re `,`.split(csvContent.split(&quot;\n&quot;)[0]);<br>    <br>    string fileId = uuid:createType1AsString();<br>    csvStore[fileId] = parseCSV(csvContent);<br>    <br>    return { fileId, headers };<br>}</pre><h3>Natural Language Query- Utilizes Ballerina’s natural expressions</h3><pre>resource function get query(string fileId, string question) returns json {<br>    CSVData data = csvStore.get(fileId);<br>    <br>    string prompt = buildPrompt(data, question);<br>    <br>    chat:CreateChatCompletionResponse result = <br>        chatClient-&gt;/chat/completions.post({<br>            model: &quot;gpt-4o&quot;,<br>            messages: [{ role: &quot;user&quot;, content: prompt }]<br>        });<br>    <br>    return parseResponse(result);<br>}</pre><h3>Prompt Engineering Magic 🧙‍♂️</h3><p>The secret sauce is in the prompts. Here’s the template:</p><pre>You are a data analyst. Analyze this CSV:</pre><pre>File: employees.csv<br>Rows: 100<br>Headers: name, age, salary, department</pre><pre>Sample Data:<br>John, 30, 75000, Engineering<br>Jane, 28, 65000, Marketing<br>...</pre><pre>Question: {user_question}</pre><pre>Return JSON:<br>{<br>  &quot;answer&quot;: &quot;detailed answer&quot;,<br>  &quot;insights&quot;: [&quot;insight 1&quot;, &quot;insight 2&quot;],<br>  &quot;visualization&quot;: { &quot;chartType&quot;: &quot;bar&quot;, ... }<br>}</pre><h3>Challenges Solved</h3><h3>1. CORS Hell</h3><p><strong>Problem:</strong> Frontend couldn’t talk to backend<br> <strong>Solution:</strong> Proper CORS config + web server requirement</p><h3>2. Token Limits</h3><p><strong>Problem:</strong> Large CSVs break GPT-4o<br> <strong>Solution:</strong> Smart sampling (first 5 rows for queries, 100 for insights)</p><h3>3. Response Structure</h3><p><strong>Problem:</strong> Inconsistent AI responses<br> <strong>Solution:</strong> Explicit JSON schema in prompts</p><h3>Quick Start: Playing with UI</h3><ol><li>Clone the repo</li></ol><pre>git clone https://github.com/JayanaGunaweera01/data-whisper.git</pre><p>2. Replace your OpenAI key in Config.toml</p><pre><br>openaiApiKey = &quot;sk-...&quot;</pre><p>3. Install Ballerina 2201.13.0-m3 (Swan Lake) or greater. Install Ballerina and use the bal dist pull command to pull the milestone version.</p><pre>$ bal dist pull 2201.13.0-m3</pre><p>4. From project root open a new terminal and start the ballerina service</p><pre>bal run --experimental</pre><p>5. Navigate to frontend folder and start another terminal.</p><pre>python -m http.server 8080</pre><p>6. Open Index.HTML</p><p>7. Go to localhost</p><pre>http://localhost:8080</pre><p>Now you’re ready🎉Upload your CSV file and query the Ballerina CSV Whisper Agent</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4vzNGm-O04DL8Au1r_hcfw.jpeg" /><figcaption>Ballerina CSV Whisper UI</figcaption></figure><h3>Quick Start: Playing with Terminal</h3><p>Complete the first 4 steps outlined in the ‘Playing with UI’ section</p><p>Start a terminal from the root</p><ol><li>Set the base URL</li></ol><pre>set BASE_URL=http://localhost:9090/csv-chat</pre><p>2. Health Check</p><pre>curl -X GET %BASE_URL%/health</pre><p>3. Upload File: Ex- test.csv</p><pre>curl -X POST -F &quot;file=@test.csv&quot; %BASE_URL%/upload</pre><p>4. Get the file ID (you can get this from the response of step 3 as well)</p><pre>curl -s -X POST -F &quot;file=@test.csv&quot; http://localhost:8080/csv-chat/upload | findstr &quot;fileId&quot;</pre><p>5. Ask Questions (*Change the file ID to your file ID)</p><pre>curl -X GET %BASE_URL%/query?fileId=01f0b6aa-321d-1e1e-a873-b66699866e4e^&amp;question=What is this csv about?</pre><p>6. Get Insights (* Use your file ID)</p><pre>curl -X GET &quot;%BASE_URL%/insights?fileId=abc123&quot;</pre><p>7. Chat (* Use your file ID)</p><pre>curl -X POST -H &quot;Content-Type: application/json&quot; -d &quot;{\&quot;fileId\&quot;:\&quot;abc123\&quot;,\&quot;message\&quot;:\&quot;Show trends of salary in this csv\&quot;}&quot; %BASE_URL%/chat</pre><p>You will get something like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQerLMcFadHxHiwyrM6Fxg.jpeg" /></figure><h3>Watch a demo video of the platform📽️</h3><p>Click below to see a demo of the platform.</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSVVQGRZKgWY&amp;type=text%2Fhtml&amp;schema=google&amp;display_name=YouTube&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSVVQGRZKgWY" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7af1d83a9e97c2f3f91b60901fc71813/href">https://medium.com/media/7af1d83a9e97c2f3f91b60901fc71813/href</a></iframe><h3>Try It Yourself 🚀</h3><p>The full code is open source! Check out the repo <a href="https://github.com/JayanaGunaweera01/data-whisper">https://github.com/JayanaGunaweera01/data-whisper</a></p><p>Start the conversation! Ballerina CSV Whisper Agent is now ready to answer🚀</p><h3>References</h3><ul><li><a href="https://ballerina.io/">The Ballerina programming language</a></li><li><a href="https://ballerina.io/learn/work-with-llms-using-natural-expressions">Work with Large Language Models (LLMs) using natural expressions - The Ballerina programming language</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f89f357083b" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/ballerina-csv-whisper-agent-%EF%B8%8F-6f89f357083b">Ballerina CSV Whisper Agent📊🕵️</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essentials Part 7-Cloud Design Patterns To Master! ☁️]]></title>
            <link>https://medium.com/@Jayana_Gunaweera/cloud-design-patterns-to-master-%EF%B8%8F-6b9c41bd241b?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b9c41bd241b</guid>
            <category><![CDATA[cloud-design-patterns]]></category>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Tue, 11 Mar 2025 03:39:58 GMT</pubDate>
            <atom:updated>2025-03-11T03:46:12.328Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xhDeqijHGRaXAzaEafpqsQ.gif" /></figure><p>Cloud computing has transformed the way applications are built, deployed and scaled. But to fully harness its power, understanding key cloud design patterns is crucial. These patterns help improve <strong>availability, scalability, performance and security</strong> in cloud-based applications. 🌍💡</p><p>Let’s explore some <strong>essential cloud design patterns</strong> every developer, architect and DevOps engineer should know! 👇</p><h3>📌 Availability Patterns 🟢</h3><p>Ensuring system uptime and resilience to failures.</p><h3>1️⃣ Circuit Breaker ⚡</h3><p>Prevents a service from repeatedly trying to execute a failing operation, reducing system overload.</p><h3>2️⃣ Retry 🔄</h3><p>Handles transient failures by automatically retrying failed operations, improving fault tolerance.</p><h3>3️⃣ Throttling 🚦</h3><p>Controls resource consumption by limiting the number of requests processed over time, preventing overloading.</p><h3>4️⃣ Health Endpoint Monitoring ❤️‍🩹</h3><p>Monitors system health by exposing diagnostic endpoints that help detect and address failures proactively.</p><h3>5️⃣ Queue-Based Load Leveling 📥📤</h3><p>Uses a message queue as a buffer between tasks and services, smoothing out workload spikes.</p><h3>6️⃣ Scheduler Agent Supervisor 🕰️</h3><p>Manages and coordinates tasks across distributed services to ensure reliable execution.</p><h3>📌 Data Management Patterns 🗄️</h3><p>Optimizing data storage, retrieval and consistency.</p><h3>7️⃣ Cache-Aside 🏎️</h3><p>Loads frequently accessed data into a cache, improving read performance and reducing database load.</p><h3>8️⃣ CQRS (Command and Query Responsibility Segregation) 🔄</h3><p>Separates read and write operations to optimize performance and scalability.</p><h3>9️⃣ Event Sourcing 📜</h3><p>Records all changes as an event log, maintaining a complete history of data modifications.</p><h3>🔟 Materialized View 🏗️</h3><p>Creates precomputed views of data to speed up queries and reduce computation time.</p><h3>1️⃣1️⃣ Sharding 🔀</h3><p>Splits a large database into smaller, more manageable partitions (shards) to improve scalability.</p><h3>1️⃣2️⃣ Valet Key 🔑</h3><p>Provides temporary, restricted access to specific resources using time-bound tokens.</p><h3>📌 Design &amp; Implementation Patterns 🏗️</h3><p>Enhancing modularity, scalability and maintainability.</p><h3>1️⃣3️⃣ Ambassador 📨</h3><p>Acts as a proxy service to handle network requests on behalf of client applications.</p><h3>1️⃣4️⃣ Anti-corruption Layer 🛡️</h3><p>Creates a facade to prevent legacy system dependencies from corrupting new application designs.</p><h3>1️⃣5️⃣ Backends for Frontends (BFF) 🖥️📲</h3><p>Customizes backend APIs for different front-end interfaces (e.g., web, mobile) to optimize performance.</p><h3>1️⃣6️⃣ Compute Resource Consolidation ⚙️</h3><p>Combines multiple workloads onto fewer compute instances, reducing costs and increasing efficiency.</p><h3>1️⃣7️⃣ External Configuration Store 📜</h3><p>Stores application configuration externally, allowing dynamic updates without redeployments.</p><h3>1️⃣8️⃣ Gatekeeper 🚪</h3><p>Protects backend services from unauthorized requests, enforcing security policies at the API gateway.</p><h3>1️⃣9️⃣ Sidecar 🏎️</h3><p>Deploys auxiliary services alongside the main application to provide <strong>logging, monitoring, or security</strong> enhancements.</p><h3>2️⃣0️⃣ Strangler Fig 🌿</h3><p>Gradually replaces a legacy system by migrating functionality piece by piece to a new architecture.</p><h3>📌 Messaging Patterns 📩</h3><p>Improving asynchronous communication and message processing.</p><h3>2️⃣1️⃣ Competing Consumers ⚖️</h3><p>Distributes messages across multiple consumers to balance load and increase parallel processing.</p><h3>2️⃣2️⃣ Priority Queue 🎯</h3><p>Processes high-priority messages first, ensuring critical tasks are handled before less urgent ones.</p><h3>🚀 Wrapping Up</h3><p>Mastering these <strong>cloud design patterns</strong> can help you build scalable, resilient and efficient cloud-native applications. 🌐💪 Whether you’re working with <strong>AWS, Azure, Google Cloud, or Kubernetes</strong>, applying these best practices can <strong>optimize performance, reduce costs and enhance security.</strong> 🔥</p><p>Which of these patterns have you used in your projects?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b9c41bd241b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essentials Part 6 - Top 10 Mistakes Developers Make in EF Core ⚠️]]></title>
            <link>https://medium.com/@Jayana_Gunaweera/essentials-part-6-top-10-mistakes-developers-make-in-ef-core-%EF%B8%8F-2f0c356f6e3b?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f0c356f6e3b</guid>
            <category><![CDATA[ef-core-performance]]></category>
            <category><![CDATA[ef-core]]></category>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Mon, 10 Feb 2025 08:35:39 GMT</pubDate>
            <atom:updated>2025-02-10T08:35:39.958Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QPqUSaK-gFbsrqWZKCePGg.gif" /></figure><p>Entity Framework Core (EF Core) is a powerful ORM, but misusing it can lead to performance issues, inefficient queries and potential data integrity problems. Here’s a breakdown of the most common mistakes developers make and how to avoid them with best practices and code examples.</p><h3>1️⃣ Not Using Indexes</h3><p>Failing to create indexes on frequently queried columns can lead to slow performance, especially on large datasets.</p><h3>✅ Solution:</h3><p>Define indexes for columns used in WHERE, JOIN, or ORDER BY clauses.</p><pre>public class User<br>{<br>    public int Id { get; set; }<br>    [Index]<br>    public string Email { get; set; }<br>}</pre><p>Alternatively, use migrations:</p><pre>modelBuilder.Entity&lt;User&gt;().HasIndex(u =&gt; u.Email);</pre><h3>2️⃣ Not Using Projections</h3><p>Fetching entire entities when only specific fields are needed increases memory usage and query execution time.</p><h3>✅ Solution:</h3><p>Use <strong>Select()</strong> to fetch only required fields.</p><pre>var users = context.Users.Select(u =&gt; new { u.Id, u.Email }).ToList();</pre><h3>3️⃣ Overfetching Data</h3><p>Loading too many records at once can cause memory bloat and slow performance.</p><h3>✅ Solution:</h3><p>Use <strong>pagination</strong> to retrieve data in chunks.</p><pre>int pageSize = 10, pageNumber = 1;<br>var users = context.Users.Skip((pageNumber - 1) * pageSize).Take(pageSize).ToList();</pre><h3>4️⃣ Not Using AsNoTracking</h3><p>EF Core tracks all retrieved entities by default, consuming unnecessary memory for read-only operations.</p><h3>✅ Solution:</h3><p>Use AsNoTracking() for queries where updates are not needed.</p><pre>var users = context.Users.AsNoTracking().ToList();</pre><h3>5️⃣ Unwise Eager Loading</h3><p>Including too many related entities can lead to heavy JOIN operations and performance overhead.</p><h3>✅ Solution:</h3><p>Use <strong>filtered Include</strong> when loading related data.</p><pre>var orders = context.Orders.Include(o =&gt; o.Items.Where(i =&gt; i.Price &gt; 100)).ToList();</pre><h3>6️⃣ Ignoring Transactions in Batch Operations</h3><p>Batch operations using ExecuteUpdate() or ExecuteDelete() are not tracked by EF Core, leading to inconsistent data if failures occur.</p><h3>✅ Solution:</h3><p>Wrap batch updates in a <strong>transaction</strong>.</p><pre>using var transaction = context.Database.BeginTransaction();<br>context.Users.Where(u =&gt; u.IsActive).ExecuteUpdate(u =&gt; u.SetProperty(u =&gt; u.LastLogin, DateTime.UtcNow));<br>transaction.Commit();</pre><h3>7️⃣ Ignoring Concurrency Handling</h3><p>Multiple users updating the same record can overwrite each other’s changes without warning.</p><h3>✅ Solution:</h3><p>Use a <strong>concurrency token</strong> and handle DbUpdateConcurrencyException.</p><pre>public class Product<br>{<br>    public int Id { get; set; }<br>    public string Name { get; set; }<br>    [ConcurrencyCheck]<br>    public int Stock { get; set; }<br>}</pre><p>Handling concurrency exception:</p><pre>try<br>{<br>    context.SaveChanges();<br>}<br>catch (DbUpdateConcurrencyException)<br>{<br>    // Handle concurrency conflict<br>}</pre><h3>8️⃣ Not Using Migrations</h3><p>Manually changing the database schema without migrations can cause inconsistencies.</p><h3>✅ Solution:</h3><p>Use EF Core migrations to manage schema changes.</p><pre>dotnet ef migrations add AddNewColumn<br>dotnet ef database update</pre><h3>9️⃣ Forgetting to Dispose DbContext</h3><p>Not disposing of manually created DbContext can lead to memory leaks.</p><h3>✅ Solution:</h3><p>Use a <strong>using statement</strong> to dispose of DbContext properly.</p><pre>using (var context = new AppDbContext())<br>{<br>    var users = context.Users.ToList();<br>}</pre><h3>🔟 Not Using Asynchronous Methods</h3><p>Using synchronous methods in database operations can block threads, reducing application responsiveness.</p><h3>✅ Solution:</h3><p>Use <strong>async/await</strong> for database calls.</p><pre>var users = await context.Users.ToListAsync();</pre><p>By avoiding these common EF Core mistakes, developers can significantly improve database efficiency, maintainability and performance. Implementing best practices like indexing, projections, pagination and proper concurrency handling ensures smooth database interactions in any application. 🚀</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f0c356f6e3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essentials Part 5 -Software Development Best Practices You Can’t Ignore]]></title>
            <link>https://medium.com/@Jayana_Gunaweera/hard-earned-wisdom-software-development-best-practices-you-cant-ignore-b07492755e8d?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b07492755e8d</guid>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Sat, 08 Feb 2025 03:37:25 GMT</pubDate>
            <atom:updated>2025-02-08T05:03:41.941Z</atom:updated>
            <content:encoded><![CDATA[<h3>💡 Hard-Earned Wisdom</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*77WbxBDd9qmy6c8nPUDuIA.gif" /></figure><p>Software development is full of challenges and learning the hard way is often part of the process. However, adopting best practices early can save time, reduce frustration and lead to more robust applications. Below are some critical lessons every developer should follow to write maintainable, efficient and secure software.</p><h3>1. Logging Isn’t Optional</h3><p>Logging helps diagnose issues, monitor system health and debug production failures.</p><p>✔ Log <strong>EVERY</strong> exception with a unique ID. <br>✔ Include request context in logs. <br>✔ Use <strong>structured logging</strong> instead of string concatenation. <br>✔ Rotate logs to prevent disk overflow.</p><blockquote><strong>Example: Basic Logging in Python for Error Handling and Debugging</strong></blockquote><pre>import logging<br>logging.basicConfig(level=logging.INFO, format=&#39;%(asctime)s - %(levelname)s - %(message)s&#39;)</pre><pre>def process_data(data):<br>    try:<br>        result = data / 10<br>        logging.info(&quot;Processed data: %s&quot;, result)<br>    except Exception as e:<br>        logging.error(&quot;Error processing data: %s&quot;, e, exc_info=True)</pre><h3>2. Never Trust User Input</h3><p>User input is a potential security risk and should always be validated and sanitized.</p><p>✔ Validate input at the <strong>API gateway</strong>. <br>✔ Use strong typing, never <strong>‘any’</strong>. <br>✔ Prevent <strong>SQL injection</strong> with prepared statements. <br>✔ Implement <strong>rate limiting</strong> on all APIs.</p><blockquote><strong>Example: Precautions against SQL injection</strong></blockquote><pre>import sqlite3</pre><pre>def get_user_data(user_id):<br>    conn = sqlite3.connect(&#39;database.db&#39;)<br>    cursor = conn.cursor()<br>    cursor.execute(&quot;SELECT * FROM users WHERE id = ?&quot;, (user_id,))  # Safe from SQL injection<br>    return cursor.fetchone()</pre><h3>3. Database Lessons</h3><p>A well-optimized database ensures fast and efficient data retrieval.</p><p>✔ Understand <strong>transaction isolation levels</strong>. <br>✔ Index queries <strong>before production</strong>. <br>✔ Never run <strong>migrations during peak hours</strong>. <br>✔ Keep a <strong>replica ready</strong>. <br>✔ Monitor <strong>query performance</strong> from day 1.</p><blockquote><strong>Example: How Index Improves Query Performance</strong></blockquote><pre>CREATE INDEX idx_user_email ON users(email);<br>EXPLAIN ANALYZE SELECT * FROM users WHERE email = &#39;user@example.com&#39;;</pre><h3>When to Use Indexing?</h3><p>✅ When searching frequently by a column (e.g., WHERE email = ...).<br>✅ When sorting (ORDER BY email).<br>✅ When enforcing <strong>unique constraints</strong>.</p><p>⚠️ <strong>Avoid excessive indexing</strong> since it <strong>slows down inserts and updates</strong> (index needs updating).</p><h3>4. Caching Gotchas</h3><p>Caching is a double-edged sword — if done right, it boosts performance; if done wrong, it creates chaos.</p><p>✔ Cache invalidation is <strong>harder than you think</strong>. <br>✔ Don’t cache <strong>everything</strong>. <br>✔ Set <strong>TTL (Time-To-Live)</strong> for cache entries. <br>✔ Choose the <strong>right layer</strong> (CDN, App, DB). <br>✔ Have a <strong>cache busting</strong> strategy.</p><blockquote><strong>Example: Use Redis as a Fast Cache for Database Queries</strong></blockquote><pre>import redis<br>cache = redis.Redis()</pre><pre>def get_data():<br>    data = cache.get(&#39;key&#39;)<br>    if not data:<br>        data = fetch_from_db()<br>        cache.setex(&#39;key&#39;, 3600, data)  # Cached for 1 hour<br>    return data</pre><h3>Advantages of Using Redis for Caching</h3><p>✅ <strong>Faster Performance:</strong> Reduces database load and speeds up responses.<br>✅ <strong>Reduces Repeated Queries:</strong> Database queries are expensive; caching helps avoid redundant requests.<br>✅ <strong>Auto-Expiration:</strong> The setex() method ensures data expires after a defined time (1 hour in this case).</p><h3>5. Error Handling</h3><p>Handling errors properly ensures a resilient system.</p><p>✔ Don’t <strong>silently swallow exceptions</strong>. <br>✔ Return proper <strong>HTTP status codes</strong>. <br>✔ Include <strong>error codes</strong> in responses. <br>✔ Have a <strong>fallback</strong> for critical flows.</p><blockquote><strong>Example: A global error-handling middleware in an Express.js application.</strong></blockquote><pre>app.use((err, req, res, next) =&gt; {<br>    console.error(err.stack);<br>    res.status(500).json({ error: &#39;Something went wrong&#39; });<br>});</pre><h3>6. Configuration Management</h3><p>Proper configuration management prevents deployment disasters.</p><p>✔ Never <strong>hardcode credentials</strong>. <br>✔ Use <strong>environment variables</strong>. <br>✔ Keep <strong>configs version-controlled</strong>. <br>✔ Maintain separate configs for <strong>dev/staging/prod</strong>. <br>✔ Document <strong>every config parameter</strong>.</p><blockquote><strong>Example: Managing Environment Variables in Bash for Secure Configuration</strong></blockquote><pre>export DB_HOST=localhost<br>export DB_USER=admin<br>export DB_PASSWORD=securepass</pre><p>⚠️ <strong>Avoid storing passwords in plain text. Instead:</strong> ✅ Use a .env file and load it securely:</p><pre>DB_PASSWORD=$(cat /path/to/secret_file)</pre><p>✅ Use a <strong>secrets manager</strong> like AWS Secrets Manager or HashiCorp Vault.<br>✅ Set restricted permissions:</p><pre>chmod 600 ~/.bashrc</pre><h3>7. Testing Wisdom</h3><p>Testing isn’t just about writing unit tests — real-world scenarios matter more.</p><p>✔ Unit tests <strong>aren’t enough</strong>. <br>✔ Integration tests <strong>catch real bugs</strong>. <br>✔ Test <strong>error scenarios</strong>. <br>✔ Mock <strong>external services</strong>. <br>✔ Test with <strong>realistic data volumes</strong>.</p><blockquote><strong>Example: Mocking API Calls for Reliable Python Tests</strong></blockquote><pre>def test_api_response():<br>    mock_response = {&quot;status&quot;: &quot;success&quot;}<br>    assert get_api_response() == mock_response</pre><h3>8. Deployment Sanity</h3><p>Deployments can go horribly wrong if not managed well.</p><p>✔ Feature flags <strong>&gt; direct deployments</strong>. <br>✔ Keep <strong>rollback scripts ready</strong>. <br>✔ Deploy during <strong>low-traffic hours</strong>. <br>✔ Monitor <strong>key metrics post-deployment</strong>. <br>✔ Document <strong>deployment steps</strong>.</p><blockquote><strong>Example: Git Branch Management and Docker Deployment</strong></blockquote><pre>git checkout release-branch<br>git pull origin main<br>docker-compose up -d --build</pre><h3>9. API Design</h3><p>A well-designed API ensures maintainability and scalability.</p><p>✔ Version APIs <strong>from day 1</strong>. <br>✔ Use <strong>DTOs</strong> for request/response. <br>✔ Keep <strong>payloads small</strong>. <br>✔ Follow <strong>REST conventions</strong>. <br>✔ Document <strong>API changes</strong>.</p><blockquote><strong>Example: Working with JSON in APIs - Parsing and Handling Data</strong></blockquote><pre>{<br>  &quot;id&quot;: 123,<br>  &quot;name&quot;: &quot;John Doe&quot;,<br>  &quot;email&quot;: &quot;john@example.com&quot;<br>}</pre><h3>10. Performance Optimization</h3><p>Optimizing performance prevents bottlenecks and improves user experience.</p><p>✔ Profile <strong>before optimizing</strong>. <br>✔ Avoid <strong>N+1 queries</strong>. <br>✔ Use <strong>connection pooling</strong>. <br>✔ Apply <strong>async operations</strong> where needed. <br>✔ Monitor <strong>memory leaks</strong>.</p><blockquote><strong>Example: Efficient Data Fetching in JavaScript - Parallel API Requests</strong></blockquote><pre>async function fetchData() {<br>    const [users, orders] = await Promise.all([<br>        fetch(&#39;/api/users&#39;).then(res =&gt; res.json()),<br>        fetch(&#39;/api/orders&#39;).then(res =&gt; res.json())<br>    ]);<br>    return { users, orders };<br>}</pre><h3>Why Use Promise.all()?</h3><p>✅ <strong>Better Performance</strong> → Both requests happen at the same time (concurrently).<br>✅ <strong>Code Simplicity</strong> → Uses destructuring to handle multiple responses cleanly.<br>✅ <strong>Error Handling</strong> → If <strong>one promise fails</strong>, the entire Promise.all() fails.</p><p>By following these best practices, developers can write robust, maintainable and high-performance software. Adopt these lessons early and you’ll save yourself from many headaches down the road! 🚀</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b07492755e8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essentials Part 4 -How to Improve Database Performance?]]></title>
            <link>https://medium.com/@Jayana_Gunaweera/how-to-improve-database-performance-00567d43ce07?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/00567d43ce07</guid>
            <category><![CDATA[database-performance]]></category>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Wed, 05 Feb 2025 06:35:46 GMT</pubDate>
            <atom:updated>2025-02-08T04:00:11.914Z</atom:updated>
            <content:encoded><![CDATA[<h3>🚀 How to Improve Database Performance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*78ERCffSB6RjrAbh2QCTOA.gif" /></figure><p>Optimizing database performance is essential to ensure fast and efficient data retrieval, reduce query execution times, and enhance overall system scalability. Whether handling small-scale applications or large enterprise databases, implementing the right strategies can significantly boost performance. This article explores key techniques to improve database efficiency.</p><h3>1. Indexing for Faster Data Retrieval</h3><p>Indexes help speed up search queries by allowing the database engine to locate data more efficiently without scanning entire tables.</p><blockquote>A user search feature in an e-commerce platform requires quick lookups by email. Indexing the email column can enhance performance.</blockquote><p>Example SQL index creation:</p><pre>CREATE INDEX idx_email ON users(email);</pre><h3>2. Using Materialized Views</h3><p>Materialized views store precomputed query results, reducing the need to repeatedly execute complex joins and aggregations.</p><blockquote><strong>A reporting dashboard that frequently retrieves sales summaries can use materialized views instead of recalculating data on each request.</strong></blockquote><p>Example materialized view:</p><pre>CREATE MATERIALIZED VIEW sales_summary AS<br>SELECT product_id, SUM(amount) AS total_sales<br>FROM orders<br>GROUP BY product_id;</pre><h3>3. Scaling Vertically for Increased Capacity</h3><p>Increasing CPU, RAM, or storage can help a database handle more queries and process large datasets faster.</p><p><strong><em>A social media platform experiencing slow query execution times due to high traffic may benefit from upgrading its server hardware.</em></strong></p><h3>4. Denormalization to Reduce Complex Joins</h3><p>Denormalization restructures data to minimize costly joins, improving query speed at the expense of increased storage.</p><blockquote><strong>Instead of joining user and order tables frequently, store user information directly in the orders table to reduce query complexity.</strong></blockquote><p>Example denormalized structure:</p><pre>ALTER TABLE orders ADD COLUMN user_name VARCHAR(255);</pre><h3>5. Implementing Database Caching</h3><p>Caching frequently accessed data reduces the need to fetch it from the primary database, improving response times.</p><blockquote><strong>A news website can cache popular articles to avoid querying the database for every request.</strong></blockquote><p>Example Redis caching:</p><pre>const redis = require(&#39;redis&#39;);<br>const client = redis.createClient();</pre><pre>async function getArticle(id) {<br>  const cachedArticle = await client.get(`article:${id}`);<br>  if (cachedArticle) return JSON.parse(cachedArticle);<br>  const article = await fetchFromDatabase(id);<br>  client.set(`article:${id}`, JSON.stringify(article));<br>  return article;<br>}</pre><h3>6. Using Replication for Load Distribution</h3><p>Database replication creates copies of data across multiple servers, allowing read queries to be distributed efficiently.</p><blockquote><strong>An analytics system that processes extensive read queries can use read replicas to balance the load.</strong></blockquote><p>Example MySQL replication setup:</p><pre>CHANGE MASTER TO MASTER_HOST=&#39;master_db&#39;, MASTER_USER=&#39;replica_user&#39;, MASTER_PASSWORD=&#39;password&#39;;<br>START SLAVE;</pre><h3>7. Sharding to Distribute Load</h3><p>Sharding splits large databases into smaller partitions (shards) to distribute the load across multiple servers.</p><blockquote><strong>A global user database can shard users by region to improve performance and reduce server load.</strong></blockquote><h3>8. Table Partitioning for Better Query Performance</h3><p>Partitioning divides large tables into smaller, more manageable pieces, improving query efficiency.</p><blockquote><strong>A logging system can partition logs by date for faster searches.</strong></blockquote><p>Example PostgreSQL partitioning:</p><pre>CREATE TABLE logs (<br>  id SERIAL PRIMARY KEY,<br>  log_date DATE NOT NULL,<br>  message TEXT<br>) PARTITION BY RANGE (log_date);</pre><h3>9. Query Optimization for Efficiency</h3><p>Rewriting inefficient queries ensures they execute faster and consume fewer resources.</p><blockquote><strong>A slow-running query using </strong><strong><em>SELECT *</em> can be optimized by selecting only the required columns.</strong></blockquote><p>Example query optimization:</p><pre>SELECT name, email FROM users WHERE status = &#39;active&#39;;</pre><h3>10. Choosing Appropriate Data Types</h3><p>Using the most efficient data types saves storage and speeds up processing.</p><blockquote><strong>Replacing a </strong><strong><em>VARCHAR(255)</em> with an </strong><strong><em>INTEGER</em> for storing user IDs improves performance.</strong></blockquote><p>Example data type optimization:</p><pre>ALTER TABLE users MODIFY COLUMN user_id INT;</pre><h3>11. Limiting Indexes to Avoid Overhead</h3><p>While indexes improve read performance, excessive indexing can slow down write operations.</p><blockquote><strong>A heavily written transactions table should only index frequently searched fields to balance read and write performance.</strong></blockquote><h3>12. Archiving Old Data</h3><p>Moving old, infrequently accessed data to an archive keeps the active database lightweight and efficient.</p><blockquote><strong>A banking system can archive transactions older than five years to a separate table.</strong></blockquote><p>Example data archiving:</p><pre>INSERT INTO archive_transactions SELECT * FROM transactions WHERE created_at &lt; &#39;2019-01-01&#39;;<br>DELETE FROM transactions WHERE created_at &lt; &#39;2019-01-01&#39;;</pre><p>By implementing these optimization strategies, developers and database administrators can significantly enhance database performance, ensuring faster response times, improved scalability, and efficient resource utilization. Regular monitoring and tuning are crucial to maintaining optimal database health.</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=00567d43ce07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essentials Part 3 - How to Improve Database Security?]]></title>
            <link>https://medium.com/@Jayana_Gunaweera/how-to-improve-database-security-e9079b2b1867?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9079b2b1867</guid>
            <category><![CDATA[database-security]]></category>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Wed, 29 Jan 2025 12:02:26 GMT</pubDate>
            <atom:updated>2025-02-08T04:00:50.241Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Improve Database Security👮</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FlmESJbLZJO1Rewy4z68oA.gif" /></figure><p>Database security is crucial in protecting sensitive data from unauthorized access, breaches, and cyber threats. Implementing strong security measures helps prevent data leaks, corruption, and compliance violations. Here are key strategies to enhance database security with practical examples.</p><h3>1. Use Strong Authentication &amp; Authorization</h3><p>Ensure that only authorized users can access the database by enforcing strict authentication and role-based access control (RBAC).</p><blockquote><strong>Restrict user permissions to minimize access rights.</strong></blockquote><p>Example PostgreSQL RBAC:</p><pre>CREATE ROLE read_only_user;<br>GRANT CONNECT ON DATABASE mydb TO read_only_user;<br>GRANT SELECT ON ALL TABLES IN SCHEMA public TO read_only_user;</pre><h3>2. Encrypt Data at Rest and in Transit</h3><p>Data encryption ensures that even if an attacker gains access, the data remains unreadable without decryption keys.</p><blockquote><strong>Encrypt stored data using AES-256 and enforce SSL/TLS for data transmission.</strong></blockquote><p>Example MySQL encryption:</p><pre>ALTER TABLE users MODIFY COLUMN password VARBINARY(256) ENCRYPTED WITH (ENCRYPTION_TYPE = &#39;AES256&#39;);</pre><p>Enable SSL in MySQL:</p><pre>ALTER USER &#39;db_user&#39;@&#39;%&#39; REQUIRE SSL;</pre><h3>3. Implement Database Firewalls</h3><p>Firewalls prevent unauthorized access by filtering traffic based on predefined rules.</p><blockquote><strong>Restrict database access to specific IP addresses.</strong></blockquote><p>Example PostgreSQL pg_hba.conf:</p><pre>host all all 192.168.1.0/24 md5</pre><h3>4. Regularly Update and Patch Databases</h3><p>Keeping database software up to date prevents exploitation of known vulnerabilities.</p><blockquote><strong>Automate database updates and security patches.</strong></blockquote><p>Example updating MySQL:</p><pre>sudo apt update &amp;&amp; sudo apt upgrade mysql-server</pre><h3>5. Backup Data Securely</h3><p>Regular backups help recover data in case of breaches or corruption. Encrypt and store backups in a separate, secure location.</p><blockquote><strong>Schedule encrypted backups using PostgreSQL.</strong></blockquote><p>Example backup command:</p><pre>pg_dump -U db_user -F c -Z 9 -f backup.pgsql mydb</pre><h3>6. Implement Least Privilege Principle</h3><p>Limit user access to only necessary privileges to reduce risk.</p><blockquote><strong>Grant limited permissions to a database service user.</strong></blockquote><p>Example MySQL privilege management:</p><pre>GRANT SELECT, INSERT, UPDATE ON mydb.* TO &#39;app_user&#39;@&#39;localhost&#39;;</pre><h3>7. Secure Database Configuration</h3><p>Disable unused services and tighten configuration settings to minimize attack surfaces.</p><blockquote><strong>Disable remote root access in MySQL.</strong></blockquote><p>Example MySQL secure settings:</p><pre>sudo mysql_secure_installation</pre><h3>8. Monitor and Log Database Activities</h3><p>Enable logging and real-time monitoring to detect suspicious activities.</p><blockquote><strong>Enable PostgreSQL logging for tracking queries.</strong></blockquote><p>Example PostgreSQL logging settings:</p><pre>ALTER SYSTEM SET log_statement = &#39;all&#39;;</pre><h3>9. Prevent SQL Injection Attacks</h3><p>Use prepared statements and parameterized queries to prevent malicious SQL execution.</p><blockquote><strong>Secure query execution using Python and MySQL.</strong></blockquote><p>Example Python with MySQL:</p><pre>cursor.execute(&quot;SELECT * FROM users WHERE username = %s&quot;, (user_input,))</pre><h3>10. Regularly Audit Security Policies</h3><p>Conduct periodic security audits to identify and fix vulnerabilities.</p><blockquote><strong>Run automated security scans on database permissions and configurations.</strong></blockquote><p>Example PostgreSQL audit:</p><pre>SELECT * FROM information_schema.role_table_grants WHERE grantee = &#39;public&#39;;</pre><p>By implementing these database security best practices, you can safeguard sensitive information, prevent data breaches, and ensure compliance with security standards. Regular monitoring, encryption, and access control are essential components of a robust security strategy. 🔐</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9079b2b1867" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essentials Part 2 -How to Improve Web Application Security?]]></title>
            <link>https://medium.com/@Jayana_Gunaweera/how-to-improve-web-application-security-7efd256e21f6?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7efd256e21f6</guid>
            <category><![CDATA[web-application-security]]></category>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Wed, 29 Jan 2025 11:54:31 GMT</pubDate>
            <atom:updated>2025-02-08T03:58:34.789Z</atom:updated>
            <content:encoded><![CDATA[<h3>🔐 How to Improve Web Application Security</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NWNQgkFr7y9zoNsOuTK1hg.gif" /></figure><p>Ensuring the security of web applications is essential to protect sensitive data, maintain user trust, and prevent cyberattacks. Here are some key techniques developers can implement to strengthen web application security.</p><h3>1. Implement Strong Authentication and Authorization</h3><p>Using robust authentication mechanisms ensures that only authorized users can access sensitive parts of the application.</p><blockquote><strong>A banking application requires multi-factor authentication (MFA) to secure user accounts.</strong></blockquote><p><strong>Example Implementation of MFA in Node.js:</strong></p><pre>const speakeasy = require(&#39;speakeasy&#39;);<br>const qrcode = require(&#39;qrcode&#39;);</pre><pre>// Generate a secret key for a user<br>const secret = speakeasy.generateSecret({ length: 20 });<br>console.log(secret.base32); // Store this securely</pre><pre>// Generate a QR Code for Google Authenticator<br>qrcode.toDataURL(secret.otpauth_url, (err, imageUrl) =&gt; {<br>  console.log(imageUrl); // Show this to the user for scanning<br>});</pre><h3>2. Use HTTPS and Secure Cookies</h3><p>Ensuring encrypted communication prevents man-in-the-middle attacks and data interception.</p><blockquote><strong>A social media platform secures user sessions with HTTP Strict Transport Security (HSTS) and secure cookies.</strong></blockquote><p><strong>Example Nginx Configuration for HTTPS:</strong></p><pre>server {<br>    listen 443 ssl;<br>    server_name example.com;<br>    ssl_certificate /etc/ssl/cert.pem;<br>    ssl_certificate_key /etc/ssl/key.pem;<br>    add_header Strict-Transport-Security &quot;max-age=31536000; includeSubDomains&quot; always;<br>}</pre><h3>3. Prevent SQL Injection</h3><p>Sanitizing inputs and using prepared statements helps prevent SQL injection attacks.</p><blockquote><strong>A login form vulnerable to SQL injection allows attackers to gain unauthorized access.</strong></blockquote><p><strong>Example Secure SQL Query in Python (Flask + MySQL):</strong></p><pre>import mysql.connector</pre><pre>conn = mysql.connector.connect(user=&#39;user&#39;, password=&#39;password&#39;, database=&#39;testdb&#39;)<br>cursor = conn.cursor()</pre><pre>username = input(&quot;Enter username: &quot;)<br>password = input(&quot;Enter password: &quot;)</pre><pre>query = &quot;SELECT * FROM users WHERE username = %s AND password = %s&quot;<br>cursor.execute(query, (username, password))<br>result = cursor.fetchall()</pre><h3>4. Secure API Endpoints</h3><p>APIs should use authentication, rate limiting, and validation to prevent abuse.</p><blockquote><strong>A public API is being abused by bots, leading to performance degradation.</strong></blockquote><p><strong>Example Express.js Middleware for Rate Limiting:</strong></p><pre>const rateLimit = require(&#39;express-rate-limit&#39;);<br>const limiter = rateLimit({<br>  windowMs: 15 * 60 * 1000, // 15 minutes<br>  max: 100 // Limit each IP to 100 requests per window<br>});</pre><pre>app.use(&#39;/api/&#39;, limiter);</pre><h3>5. Protect Against Cross-Site Scripting (XSS)</h3><p>Escaping and validating user input prevents the execution of malicious scripts.</p><blockquote><strong>A comment section allows users to inject harmful scripts that execute in other users’ browsers.</strong></blockquote><p><strong>Example HTML Encoding in React:</strong></p><pre>function SafeComponent({ userInput }) {<br>  return &lt;div dangerouslySetInnerHTML={{ __html: userInput }} /&gt;; // Avoid this!<br>}</pre><p>Better approach:</p><pre>import DOMPurify from &#39;dompurify&#39;;<br>function SafeComponent({ userInput }) {<br>  return &lt;div dangerouslySetInnerHTML={{ __html: DOMPurify.sanitize(userInput) }} /&gt;;<br>}</pre><h3>6. Enable Content Security Policy (CSP)</h3><p>CSP restricts the sources from which content can be loaded, reducing the risk of XSS attacks.</p><blockquote><strong>An attacker tries to load a malicious script from an external domain.</strong></blockquote><p><strong>Example CSP Implementation in Express.js:</strong></p><pre>app.use((req, res, next) =&gt; {<br>  res.setHeader(&quot;Content-Security-Policy&quot;, &quot;default-src &#39;self&#39;&quot;);<br>  next();<br>});</pre><h3>7. Validate and Sanitize User Input</h3><p>Rejecting unexpected input prevents various security vulnerabilities.</p><blockquote><strong>A user enters unexpected input that causes errors or exploits vulnerabilities.</strong></blockquote><p><strong>Example Input Validation with Express Validator:</strong></p><pre>const { body, validationResult } = require(&#39;express-validator&#39;);</pre><pre>app.post(&#39;/register&#39;, [<br>  body(&#39;email&#39;).isEmail(),<br>  body(&#39;password&#39;).isLength({ min: 6 })<br>], (req, res) =&gt; {<br>  const errors = validationResult(req);<br>  if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() });<br>  // Continue with user registration<br>});</pre><h3>8. Implement Secure Session Management</h3><p>Proper session management prevents session hijacking and fixation attacks.</p><blockquote><strong>An e-commerce application uses secure session handling to prevent unauthorized access.</strong></blockquote><p><strong>Example Secure Session Handling in Express.js:</strong></p><pre>const session = require(&#39;express-session&#39;);<br>app.use(session({<br>  secret: &#39;secureSecretKey&#39;,<br>  resave: false,<br>  saveUninitialized: true,<br>  cookie: { secure: true, httpOnly: true }<br>}));</pre><p>Securing web applications requires continuous effort and the implementation of best practices. By using strong authentication, preventing common vulnerabilities, enforcing security headers, and properly managing sessions, developers can create robust and resilient applications that protect user data and prevent attacks.</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7efd256e21f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essentials Part 1 -How to Optimize Web Application Performance?]]></title>
            <link>https://medium.com/@Jayana_Gunaweera/how-to-optimize-web-application-performance-7856d67e0e0c?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7856d67e0e0c</guid>
            <category><![CDATA[performance-optimization]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Mon, 27 Jan 2025 05:42:47 GMT</pubDate>
            <atom:updated>2025-02-08T03:59:16.290Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HeR1uN_r1wtakw4s2w8bBg.gif" /></figure><p>Optimizing web applications is crucial to ensure a fast, responsive and efficient user experience. Poor performance can lead to high bounce rates and decreased user satisfaction. This article explores essential techniques that every developer can implement to improve web application performance.</p><h3>1.🚀 Implementing Server-Side Rendering (SSR)</h3><p>Server-Side Rendering (SSR) can significantly improve the initial load time of web pages by generating HTML on the server and delivering fully rendered pages to clients. This approach enhances SEO and provides better performance, especially for content-heavy applications.</p><blockquote><strong>Consider an e-commerce application with thousands of products. Instead of making the user wait for the client-side JavaScript to fetch data, SSR can provide preloaded content, improving page load speed.</strong></blockquote><p>Example SSR implementation with Next.js:</p><pre>import { useEffect, useState } from &#39;react&#39;;</pre><pre>export async function getServerSideProps() {<br>  const res = await fetch(&#39;https://api.example.com/products&#39;);<br>  const data = await res.json();<br>  return { props: { products: data } };<br>}</pre><pre>export default function ProductPage({ products }) {<br>  return (<br>    &lt;div&gt;<br>      {products.map((product) =&gt; (<br>        &lt;h2 key={product.id}&gt;{product.name}&lt;/h2&gt;<br>      ))}<br>    &lt;/div&gt;<br>  );<br>}</pre><h3>2.🚀 Utilizing Lazy Loading</h3><p>Lazy loading defers the loading of non-essential resources until they are needed, reducing initial page load time and enhancing user experience.</p><blockquote><strong>On a blog website, images and videos can be lazy-loaded, ensuring that only the visible content loads first.</strong></blockquote><p>Example lazy loading with React:</p><pre>const LazyImage = React.lazy(() =&gt; import(&#39;./ImageComponent&#39;));</pre><pre>function App() {<br>  return (<br>    &lt;React.Suspense fallback={&lt;div&gt;Loading...&lt;/div&gt;}&gt;<br>      &lt;LazyImage /&gt;<br>    &lt;/React.Suspense&gt;<br>  );<br>}</pre><h3>3.🚀 Leveraging a Content Delivery Network (CDN)</h3><p>Using a CDN to distribute static content across multiple geographically distributed servers can reduce latency and improve load times for users worldwide.</p><blockquote><strong>Serving large CSS, JS and image files from a CDN instead of the origin server can greatly reduce loading times for users in different regions.</strong></blockquote><p>Example CDN configuration with Nginx:</p><pre>location /static/ {<br>  proxy_pass http://cdn.example.com;<br>  expires 1y;<br>  access_log off;<br>  add_header Cache-Control &quot;public, max-age=31536000&quot;;<br>}</pre><h3>4.🚀 Compressing and Optimizing Images</h3><p>Reducing image file sizes without compromising quality is essential to minimize bandwidth usage and enhance loading speed.</p><blockquote><strong>An online portfolio website with high-resolution images can use WebP format to significantly reduce file size.</strong></blockquote><p>Example image optimization:</p><pre>&lt;picture&gt;<br>  &lt;source srcset=&quot;image.webp&quot; type=&quot;image/webp&quot;&gt;<br>  &lt;img src=&quot;image.jpg&quot; alt=&quot;Optimized Image&quot;&gt;<br>&lt;/picture&gt;</pre><h3>5.🚀 Minimizing HTTP Requests</h3><p>Reducing the number of HTTP requests through bundling and minification of CSS and JavaScript files can drastically improve loading performance.</p><blockquote><strong>A complex dashboard with multiple scripts can be bundled into a single minimized file using Webpack.</strong></blockquote><p>Example with Webpack:</p><pre>module.exports = {<br>  optimization: {<br>    minimize: true,<br>  },<br>};</pre><h3>6.🚀 Optimizing Database Queries</h3><p>Efficient database queries with proper indexing and structuring can significantly improve data retrieval performance.</p><blockquote><strong>A social media platform that frequently searches users by email can create an index to speed up query performance.</strong></blockquote><p>Example SQL query optimization:</p><pre>CREATE INDEX idx_user_email ON users(email);<br>SELECT * FROM users WHERE email = &#39;user@example.com&#39;;</pre><h3>7.🚀 Implementing Caching Techniques</h3><p>Using both browser and server-side caching can reduce server load and speed up response times.</p><blockquote><strong>A weather app can cache API responses for an hour to avoid repeated calls to the backend.</strong></blockquote><p>Example server-side caching with Redis:</p><pre>const redis = require(&#39;redis&#39;);<br>const client = redis.createClient();</pre><pre>app.get(&#39;/data&#39;, async (req, res) =&gt; {<br>  const cachedData = await client.get(&#39;data&#39;);<br>  if (cachedData) return res.json(JSON.parse(cachedData));<br>  const freshData = await fetchFromDatabase();<br>  client.set(&#39;data&#39;, JSON.stringify(freshData));<br>  res.json(freshData);<br>});</pre><h3>8.🚀 Using Pagination and Infinite Scrolling</h3><p>Fetching large datasets in smaller chunks using pagination or infinite scrolling improves performance and usability.</p><blockquote><strong>An e-commerce website listing thousands of products can fetch 10 items per page to enhance loading speed.</strong></blockquote><p>Example pagination with React:</p><pre>function fetchPageData(page) {<br>  fetch(`/api/data?page=${page}`)<br>    .then(response =&gt; response.json())<br>    .then(data =&gt; setData(data));<br>}</pre><h3>9.🚀 Prioritizing Asynchronous Operations</h3><p>Minimizing blocking JavaScript operations by using Promises and async/await makes applications more responsive.</p><blockquote><strong>Fetching user profile details asynchronously allows the UI to load other components without delay.</strong></blockquote><p>Example async operation:</p><pre>async function fetchData() {<br>  try {<br>    let response = await fetch(&#39;https://api.example.com/data&#39;);<br>    let data = await response.json();<br>    console.log(data);<br>  } catch (error) {<br>    console.error(&#39;Error fetching data:&#39;, error);<br>  }<br>}</pre><p>By implementing these optimization techniques, developers can ensure that web applications perform efficiently, providing users with a seamless experience. Continuous monitoring, performance testing and adopting best practices are key to maintaining high-performance web applications.</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7856d67e0e0c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Data Integration Made Easy with the Ballerina CData Connect Connector]]></title>
            <link>https://blog.stackademic.com/how-data-integration-made-easy-with-the-ballerina-cdata-connector-370085e2a88d?source=rss-2c76dcf502f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/370085e2a88d</guid>
            <dc:creator><![CDATA[Jayana_Gunaweera]]></dc:creator>
            <pubDate>Fri, 01 Nov 2024 06:17:16 GMT</pubDate>
            <atom:updated>2026-01-27T07:50:28.588Z</atom:updated>
            <content:encoded><![CDATA[<p>Data integration has become a cornerstone for modern applications, often requiring connectivity with various databases and APIs. One of the standout tools for such connectivity is the CData connect connector of Ballerina, a language designed specifically for integrating data sources with ease. In this article, we’ll dive into the Ballerina CData connector’s features, configuration and usage, showcasing how you can use it to integrate and manage diverse data sources efficiently.</p><p><strong>What is CData connect connector?</strong></p><p>CData Connect connector is a consolidated connectivity platform that lets you easily connect any application, on-premises or in the cloud, with real-time data from anywhere. Connect any application with data from 250+ enterprise data sources. On-Premise, or in the Cloud.</p><p><strong>What is the Ballerina CData Connect Connector?</strong></p><p>The CData Connect <a href="https://ballerina.io/">Ballerina</a> connector allows you to programmatically access all of the CData Connect applications, databases, APIs, services via the Java Database Connectivity (JDBC) API. It provides operations to execute a wide range of standard DDL Commands, SQL Commands and SQL Functions for querying data sources.</p><h3>Setting up the prerequisites</h3><ol><li>Download and install Java SE Development Kit (JDK) version 17. You can install either <a href="https://adoptopenjdk.net/">OpenJDK</a> or <a href="https://www.oracle.com/java/technologies/downloads/">Oracle</a>.</li></ol><ul><li>Note: Set the JAVA_HOME environment variable to the path name of the directory into which you installed JDK.</li></ul><p>2.Download and install <a href="https://ballerina.io/">Ballerina Swan Lake</a>.</p><p>3.Create CData connect account</p><p>4.Create a Salesforce account</p><h3>Step 1: Setting Up a Salesforce Account</h3><p>To begin, you’ll need a Salesforce account. If you don’t have one, you can create it on the <a href="https://www.salesforce.com/">Salesforce website</a>. Once signed up, log in to access your profile.</p><h3>Step 2: Configuring CData Connect Connector for Salesforce</h3><p>Log in to your CData Connect account and navigate to the <strong>Connections</strong> tab. Click <strong>Add Connection</strong> and select <strong>Salesforce</strong> from the list.</p><ul><li><strong>Permissions</strong>: Head to the <strong>Permissions</strong> tab to set CRUD (Create, Read, Update, Delete) permissions for Salesforce. This will allow the connection to fully access your Salesforce data.</li><li><strong>Connection Status</strong>: Initially, the connection status will display as “unauthenticated.”</li></ul><p>Provide a name for the connection, such as “Salesforce1.” Select <strong>Basic Authentication</strong> as your authentication scheme and enter your <strong>Salesforce username and password</strong>. You’ll also need a <strong>security token</strong> from your Salesforce profile settings. Once generated, enter it in the security token field. Save and test the connection; if successful, the status will change to “authenticated.”</p><h3>Step 3: Setting Up the Directory for Ballerina and Cloning the CData Connector</h3><p>Create a new directory for your project in Ballerina:</p><pre>mkdir ballerinaCData<br>cd ballerinaCData</pre><p>Next, fork and clone the Ballerina CData Connector from GitHub:</p><pre>git clone https://github.com/YOUR_GITHUB_USERNAME/module-ballerinax-cdata.connect.git<br>cd module-ballerinax-cdata.connect</pre><p>Set up these environment variables: (This may be an optional step unless you verified your github account or get a 401 code when building the library.)Run below codes in terminal.</p><pre>$env:packageUser = &quot;your_github_username&quot;<br>$env:packagePAT = &quot;your_github_personal_access_token_with_repo_scope&quot;</pre><p>Update the configuration in the code: (Update all files batchExecute.bal,query.bal,execute.bal,storedProcedureScenario1.bal and storedProcedureScenario2.bal)</p><pre>configurable string user = &quot;your_cdata_connected_email_address&quot;;<br>configurable string password = &quot;your_cdata_connect_access_token&quot;;</pre><h3>Step 4: Building the Library</h3><p>Build the Ballerina CData Connect Connector Library:</p><pre>./gradlew clean build</pre><p>While building the library batchExecute.bal,query.bal,execute.bal,storedProcedureScenario1.bal and storedProcedureScenario2.balwill automatically execute with the build.</p><h3>Scenario 1:Writing Ballerina Code to fetch latest LeadIDs from a salesforce Lead</h3><p>Create a .bal file in your examples folder. Ex: fetchLatestLeadIDs.bal.Let’s break down the main code components.</p><h4>1. Importing Required Libraries</h4><pre>import ballerina/io;<br>import ballerina/sql;<br>import ballerinax/cdata.connect as cdata;<br>import ballerinax/cdata.connect.driver as _;</pre><ul><li>ballerina/io: Manages input and output.</li><li>ballerina/sql: Used for SQL queries.</li><li>cdata.connect: Connects Ballerina with Salesforce.</li></ul><h4>2. Configuring CData connect Credentials</h4><p>Define the username and password as configurable variables:</p><pre>configurable string user = &quot;your_cdata_connected_email_address&quot;;<br>configurable string password = &quot;your_cdata_connect_access_token&quot;;</pre><h4>3. Initializing the CData Client</h4><p>Initialize the CData client using the configured credentials:</p><pre>cdata:Client cdataClient = check new (user, password);</pre><h4>4. Writing the SQL Query</h4><p>Define the SQL query to retrieve Salesforce leads:</p><pre>sql:ParameterizedQuery sqlQuery = `SELECT Id, FirstName, LastName, Company, CreatedDate<br>                                   FROM Salesforce1.Salesforce.Lead<br>                                   ORDER BY CreatedDate DESC LIMIT 10`;</pre><p>This query fetches the latest 10 leads, ordered by the created date.</p><h4>5. Executing the Query</h4><p>Run the query and capture the results in a stream:</p><pre>stream&lt;record {}, sql:Error?&gt; resultStream = check cdataClient-&gt;query(sqlQuery);</pre><h4>6. Printing Lead Details</h4><p>Loop through each record to print lead information:</p><pre>boolean hasLeads = false;</pre><pre>check from record {} lead in resultStream<br>    do {<br>        io:println(&quot;Lead Details: &quot;, lead);<br>        hasLeads = true;<br>    };</pre><h4>7. Checking for Results</h4><p>If no leads are found, print a message:</p><pre>if !hasLeads {<br>    io:println(&quot;No leads found.&quot;);<br>}</pre><h4><strong>8. Execute created .bal file</strong></h4><pre>bal run fetchLatestLeadIDs.bal</pre><h3>Created Lead in Salesforce account</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NLckKXzhBg-05_6Ka68p7w.png" /><figcaption>Lead</figcaption></figure><h3>Output displayed in the terminal after executing .bal file</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*18fnPseD4DigT51ojSHgZA.png" /><figcaption>Output after executing fetchLatestLeadIDs.bal</figcaption></figure><h3>Scenario 2: Writing Ballerina Code to insert data to a Lead</h3><pre>import ballerina/io;<br>import ballerina/sql;<br>import ballerinax/cdata.connect as cdata; // Get the CData connector<br>import ballerinax/cdata.connect.driver as _; // Get the CData driver<br><br>// Connection Configurations<br>configurable string user = &quot;your_cdata_connected_email_address&quot;;<br>configurable string password = &quot;your_cdata_connect_access_token&quot;;<br><br>// Initialize the CData client<br>cdata:Client cdataClient = check new (user, password);<br><br>public function main() returns error? {<br> // Define an SQL query to insert a new Lead record into Salesforce<br> sql:ParameterizedQuery sqlQuery = `INSERT INTO Salesforce1.Salesforce.Lead<br> (FirstName, LastName, Company) VALUES (&#39;John&#39;, &#39;Doe&#39;, &#39;WSO2&#39;)`;<br><br> // Execute the query using the CData client<br> _ = check cdataClient-&gt;execute(sqlQuery); // Assign result to `_` to ignore it<br> io:println(&quot;New lead successfully inserted into Salesforce.&quot;);<br><br>}</pre><h3>Scenario 3: Writing Ballerina Code to insert multiple records to a Lead</h3><pre>import ballerina/io;<br>import ballerina/sql;<br>import ballerinax/cdata.connect as cdata;  // Get the CData connector<br>import ballerinax/cdata.connect.driver as _;       // Get the CData driver<br><br>// Connection Configurations<br>configurable string user = &quot;your_cdata_connected_email_address&quot;;<br>configurable string password = &quot;your_cdata_connect_access_token&quot;;<br><br>// Initialize the CData client<br>cdata:Client cdataClient = check new (user, password);<br><br>public function main() returns error? {<br><br>    // Define multiple records to insert<br>    var insertRecords = [<br>        {<br>            FirstName: &quot;Gloria&quot;,<br>            LastName: &quot;Shania&quot;,<br>            Company: &quot;ABC&quot;<br>        },<br>        {<br>            FirstName: &quot;Shane&quot;,<br>            LastName: &quot;Warny&quot;,<br>            Company: &quot;BCA&quot;<br>        },<br>        {<br>            FirstName: &quot;Neo&quot;,<br>            LastName: &quot;Mark&quot;,<br>            Company: &quot;CAB&quot;<br>        }<br>    ];<br><br>    // Generate parameterized SQL insert queries for each record<br>    sql:ParameterizedQuery[] insertQueries =<br>        from var data in insertRecords<br>        select `INSERT INTO Salesforce1.Salesforce.Lead<br>                (FirstName, LastName, Company)<br>                VALUES (${data.FirstName}, ${data.LastName}, ${data.Company})`;<br><br>    // Execute the batch of insert queries<br>    _ = check cdataClient-&gt;batchExecute(insertQueries);<br><br>    io:println(&quot;Batch insert completed successfully.&quot;);<br>}</pre><h3>Scenario 4: Writing Ballerina Code to retrieve contacts from a Lead</h3><pre>import ballerina/io;<br>import ballerina/sql;<br>import ballerinax/cdata.connect as cdata;  // Get the CData connector<br>import ballerinax/cdata.connect.driver as _;       // Get the CData driver<br><br>// Connection Configurations<br>configurable string user = &quot;your_cdata_connected_email_address&quot;;<br>configurable string password = &quot;your_cdata_connect_access_token&quot;;<br><br>// Initialize the CData client<br>cdata:Client cdataClient = check new (user, password);<br><br>public function main() returns error? {<br><br>    // SQL query to fetch the latest contacts<br>    sql:ParameterizedQuery sqlQuery = `SELECT Id, FirstName, LastName, Email, CreatedDate<br>                                       FROM Salesforce1.Salesforce.Contact<br>                                       ORDER BY CreatedDate DESC LIMIT 10`;<br><br>    // Execute the query and get the result stream<br>    stream&lt;record {}, sql:Error?&gt; resultStream = cdataClient-&gt;query(sqlQuery);<br><br>    // Iterate over each result and print contact details<br>    check from record {} contact in resultStream<br>        do {<br>            io:println(&quot;Contact Details: &quot;, contact);<br>        };<br><br>    // No need to explicitly close the result stream as Ballerina handles it automatically.<br>}</pre><h3>Additional Salesforce Integration Scenarios</h3><p>With the Ballerina CData Connector, you can do a lot more than basic queries. Here are some powerful use cases:</p><ol><li><strong>Retrieve and Analyze Records</strong>:</li></ol><ul><li>Fetch data on leads, contacts, opportunities, and more.</li><li>Perform SQL-like operations, sorting and filtering data to fit your analysis needs.</li></ul><p><strong>2.CRUD Operations</strong>:</p><ul><li>Insert, update, and delete records (ex: leads, contacts and accounts) directly into Salesforce to keep CRM data fresh.</li></ul><p><strong>3.Data Synchronization</strong>:</p><ul><li>Use scheduled functions in Ballerina to sync Salesforce data with other systems like ERP platforms or external databases.</li></ul><p><strong>4.Execute Stored Procedures</strong>:</p><ul><li>Automate tasks like lead scoring or permission checks using Salesforce’s stored procedures.</li></ul><p><strong>5.Secure and Scalable Operations</strong>:</p><ul><li>Implement OAuth or custom authentication to keep your data secure and manage data loads efficiently for high-volume operations.</li></ul><h3>Wrapping Up</h3><p>In just a few steps, you’ve set up a powerful Salesforce integration using Ballerina CData connect connector. Whether you’re inserting records, running queries, or automating tasks, this setup allows you to make the most of Salesforce data within your applications.</p><p>Check out the <a href="https://github.com/ballerina-platform/module-ballerinax-cdata.connect">Ballerina CData Connect connector github repository</a> and the <a href="https://central.ballerina.io/ballerinax/cdata.connect.driver/1.1.1">Ballerina Central</a> for more information on the connector and keep exploring how you can use Ballerina for powerful data integrations.</p><p>You can also see my You tube tutorial of ballerina CData connect connector integrations via below <a href="https://youtu.be/fj7zj2xRas0">link</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ffj7zj2xRas0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dfj7zj2xRas0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ffj7zj2xRas0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5bccdc821f4b7ef70a7824d11c95f72/href">https://medium.com/media/c5bccdc821f4b7ef70a7824d11c95f72/href</a></iframe><p>Thank you for following along!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=370085e2a88d" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/how-data-integration-made-easy-with-the-ballerina-cdata-connector-370085e2a88d">How Data Integration Made Easy with the Ballerina CData Connect Connector📊🔗</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>