<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jayamal Jayamaha on Medium]]></title>
        <description><![CDATA[Stories by Jayamal Jayamaha on Medium]]></description>
        <link>https://medium.com/@jayamaljayamaha?source=rss-1ab1b353e11d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lc6jA1f4h-4tEEFsBLdRBg.jpeg</url>
            <title>Stories by Jayamal Jayamaha on Medium</title>
            <link>https://medium.com/@jayamaljayamaha?source=rss-1ab1b353e11d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:14:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jayamaljayamaha/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[2025 — The Year a Series of Moments Rewired My Nervous System with Firmware I Never Even Dreamed Of]]></title>
            <link>https://jayamaljayamaha.medium.com/2025-the-year-a-series-of-moments-rewired-my-nervous-system-with-firmware-i-never-even-dreamed-of-5fde48fbdf89?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fde48fbdf89</guid>
            <category><![CDATA[life]]></category>
            <category><![CDATA[life-experience]]></category>
            <category><![CDATA[life-lessons]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Sat, 14 Feb 2026 17:20:40 GMT</pubDate>
            <atom:updated>2026-02-14T17:20:40.175Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qF-Ay2AmRdGrgIBd_mnW4w.png" /></figure><p>Even though one month of 2026 has already passed, this is an article I must write not only as a way to rewind my own memories, but also for your benefit. I assure you, 2025 was a far more interesting year than any other year I have experienced so far in my life.</p><p>2025 was a year in which I was exposed to a wide series of events that reshaped my entire personality with next-level firmware. Even now, when I think back on those moments, it sends a strong dopamine rush through my neurons. If we place positivity and negativity on a scale, those experiences covered the full range throughout the year, and the intensity of that range was incredibly high. Because of that, the amount I learned during this year was equally immense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JFKK7kefJDpLsJhzjFmW6g.png" /></figure><p>Important: I’m not going to write about every life story that happened. Instead, I will focus on something I never had before.</p><p>It all started at IFS. I was selected for the IFS Welfare Team. I was nominated by Hansaka Weerasinghe, who was my manager, the coolest manager I’ve had in my professional life. He was a former welfare member, and when his term was ending, he proposed me for the 2025 Welfare Team. I truly think that was one of the best things that happened in my life.</p><p>If I talk about IFS and the Welfare Society, IFS was the largest software company in Sri Lanka. It wasn’t just software engineers, but there were consultants, support staff, customer success managers, and many others. The Welfare Society was the entity responsible for planning events that enhanced employee engagement and made IFS a well-bonded and emotionally safe place for employees (at least at that time).</p><p>Through that society, we organized many welfare events and also helped organize some events beyond welfare activities. If I list them…</p><ul><li>Thaipongal</li><li>MasterMind</li><li>Women’s Day</li><li>CSR donation</li><li>IFS Tech Conclave</li><li>IFS දහම් සර</li><li>Sports Day</li><li>Children’s Day</li><li>Christmas Carols</li></ul><h4>Thaipongal</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*oyVXL-sVbUtE4MVZvEcoYA.png" /></figure><p>This was the very first event we organized together after the new Welfare Team was appointed. Thai Pongal is a Sri Lankan religious festival, and the event was planned and coordinated by our team. At that time, most of the others were not very familiar to me, so I was trying to find ways to get to know everyone better to understand their personalities and their vibes.</p><p>The first impression was amazing. All of us seemed to be on the same wavelength, very playful, positive, and easy to connect with.</p><h4>MasterMind</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*k8ymyI9WG7UHqXmZEdWYLA.png" /></figure><p>This was a quiz competition, fully planned and organized by our Welfare Society from start to finish. It consisted of three rounds: the first round was held online, the second round took place on the company premises, and the grand finale was held at Cinnamon Grand in a truly grand manner.</p><p>Our tasks included:</p><ul><li>Designing the full competition flow</li><li>Coordinating between teams</li><li>Forming rules and instructions</li><li>Preparing refreshments for the second round</li><li>Designing gifts and coordinating with manufacturers</li><li>Visiting hotels and selecting the venue for the grand finale</li><li>Planning the grand finale event</li><li>Coordinating with event planners (this was my responsibility and one of the biggest tasks)</li><li>Planning the competition flow along with the dinner arrangements</li></ul><p>These are just the tasks that come to mind — the responsibilities were divided among our team members.</p><p>In the end, the outcome was fascinating. We received our first positive compliments from the senior leadership team, employees, and even former Welfare Team members. That meant our efforts had a real impact on the organization, and it was a truly rewarding feeling.</p><p>Beyond the appreciation we received, we genuinely enjoyed the entire process — the human interactions, the teamwork, and the collective effort. It left a lasting memory and created a new neural pathway in my mind: the belief that we can come together and create meaningful impact in a community.</p><h4>Women’s Day</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*j1FSFo5K3jQuf77PQuBbCQ.png" /></figure><p>This event was organized in celebration of Women’s Day. We presented gift packs to female employees as a gesture of recognition and appreciation, along with cupcakes for all employees. In addition, an inspirational talk was delivered by senior female leaders to all the women in the organization.</p><p>Our team handled the entire event. We coordinated with Spa Ceylon to purchase the gift items, carefully packed them into elegant gift packs, and organized the inspirational talk. Finally, we went floor by floor, personally distributing the gifts to the female employees.</p><h4>CSR Donation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*sLUoJ8qKWYrP7woM4qamqg.png" /></figure><p>IFS runs a CSR program with Veragahamula Central College, a school with great potential but limited opportunities. Representing IFS, our Welfare Society visited the school and distributed educational materials, hygiene items, and other essential supplies. We also conducted English learning sessions for the students.</p><p>Our team coordinated the entire initiative from start to finish by purchasing the materials, arranging transport, traveling to the location, and distributing the items. The whole process went smoothly, and we truly enjoyed being part of it, especially knowing we were making a meaningful impact on the community.</p><h4>IFS Tech Conclave</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*-GpIpH_ESRBaBR_Lzk1isw.png" /></figure><p>This was a tech conference and meetup focused on IFS technology. It was primarily organized by the R&amp;D division at IFS, not entirely by the Welfare Team. However, they requested support from us, and several of our team members contributed to organizing the event.</p><p>My responsibility was to coordinate with the lanyard printing vendors. I contacted them, negotiated the design, and ensured the lanyards were completed on time. The event was held at Cinnamon Life — City of Dreams, a newly built hotel, which made the experience even more special.</p><p>The event was fantastic. It was my first experience participating in and helping organize such a large-scale tech conference in such a grand venue. On the event day, we managed the registration desk, registered participants, and distributed lanyards along with small gifts.</p><h4>IFS දහම් සර</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*V-Zm54Zrt8mVMLN8gWQw2A.png" /></figure><p>This was a <em>Bhakthi Geetha</em> event organized along with a <em>dansal</em> for the Poson festival in Sri Lanka. As the Welfare Team, we handled everything from start to finish, giving employees the opportunity to come together, practice, and sing devotional songs.</p><p>On the event day, we decorated the entire premises and conducted the <em>dansal</em>. Our responsibilities included deciding the theme and event name, reserving the venue, and negotiating with food suppliers to prepare the food for the <em>dansal</em> — a task handled by Shanaka and me. On the event day, our Welfare Team members coordinated and managed the food distribution.</p><p>We faced many challenges during both the planning stage and the event day itself, but we successfully overcame them through teamwork. At times, we had to think quickly and find alternative solutions, but in the end, we made the event a success. We received amazing feedback from the senior leadership team as well as from employees.</p><h4>Sports Day 2025</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*o_dgefseJP0NEHcV_KTDsg.png" /></figure><p>Here came the biggest event at IFS — and the biggest experience I had in 2025.</p><p>Employees were divided into four houses and competed against each other in 18 different sports. The competition lasted for nearly two months and concluded with a grand final Sports Day held at Havelock Grounds, Sri Lanka.</p><p>Our Welfare Team first gathered to decide the theme for the year. The previous theme had been <em>Harry Potter</em>, and for 2025 we chose <em>Avatar</em>. The four houses were: Air Nomads, Water Tribe, Earth Kingdom, and Fire Nation.</p><p>For the opening ceremony, we planned a special event where house captains randomly selected their teams. Our Welfare Team worked together to design and prepare all the materials required for the ceremony.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*YRSFp8S1BTKH0MZltlIKdg.png" /></figure><p>We then planned all 18 sports and defined the procedures for each. One main coordinator and one supporting member were assigned to every sport. The assigned person was responsible for reserving venues such as grounds and swimming pools, coordinating with referees, and handling other logistical tasks. I was assigned to 8-ball pool. It was an interesting responsibility because it gave me real exposure to managing a large-scale organizational event.</p><p>In addition to that, I was responsible for managing food for the events. I collected food counts from each house, coordinated with vendors, ensured timely delivery to the venues, and handled distribution.</p><p>Shanaka and I also built a fantastic web application to display real-time points and rankings for each house across all sports. That added a modern and competitive edge to the entire experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*DWbyO-nd41ztflzHZ6RPsw.png" /></figure><p>As Welfare Team members, we attended every sporting event. We enjoyed the competitions, spent time with people, and got to know many new individuals and their talents. Some days we stayed until midnight; other days we arrived early in the morning. It was a truly intense but rewarding experience. I also had the opportunity to visit many sports grounds around Colombo. That was something I had never experienced before.</p><p>Finally came the grand finale. We planned it separately because it was a massive event. We had to coordinate with event planners and food vendors for a grand dinner since it was more luxurious than a five-star hotel, along with fun activities, house tent arrangements, and all the necessary logistics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*7rYpxxWg9dEbPwpw0u7_EA.png" /></figure><h4>Children’s Day 2025</h4><p>This event was organized for the children of IFS employees, giving them a joyful time to gather and participate in fun activities in celebration of Children’s Day. As the Welfare Team, we planned the entire event by reserving the second floor of Orion City, coordinating with food vendors, and working with a professional children’s event organizing team. They were fantastic and knew exactly how to engage young children in a fun and safe way. We also arranged a bouncy castle area and a sweet shop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*khsWLbFgsQxhMVm-Rulv_A.png" /></figure><p>One of the most enjoyable parts was selecting and buying toys for children across different age categories. Our entire team went to Pettah (Kotuawa), Colombo, walked through the streets, and explored various shops. We compared options, debated with each other, and made trade-offs to find the best toys at the best prices. Walking around Colombo searching for toys was a fun experience, and I discovered that many cross-street shops offer unique toys at affordable prices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*EVSQDd4l7WCJ2ES8ojKGyg.png" /></figure><p>On another day, we packed the toys into elegant gift bags, which was also a fun team activity. We even created a mini “production pipeline” in which each person had a specific task: one opened the bag, another placed an item inside, then passed it along the line until it reached the final station for organized storage and quick distribution.</p><h4>Christmas Carols</h4><p>This was a difficult chapter.</p><p>In the middle of preparing for the event, I was laid off from IFS. The only reason given was organizational restructuring.</p><p>At that point, I had two options:</p><ol><li>Resist the decision and fight back legally, which could take a long time.</li><li>Accept the situation, take the offer, focus on upgrading my skills, and move forward to find my next corporate chapter as soon as possible.</li></ol><p>I chose the second option.</p><p>Yes, I could have fought back, but it would have consumed a significant amount of my time and energy. Realistically, the outcome would have been uncertain. Instead of investing my focus in resistance, I chose to invest it in growth.</p><p>I have not come this far in life by raising my hand and waiting. I came this far by facing extremely difficult challenges and overcoming them with focus, determination, and my strongest skill, which is critical observation and analysis. I have trained my mind to believe that no matter what life throws at me, I can do that anyone can’t do, and I will find a way forward.</p><p>So I redirected my energy toward finding my next corporate opportunity. By the time I am writing this, I have already secured a better opportunity within just one month. In addition to that, I received the VRS package from IFS, and honestly, it was a decent offer. In many ways, it felt like a double win for me.</p><p>But honestly, the only thing I regret is that I had to leave behind some of the best people I found in my life, found a lot of great souls when I was at IFS, and the time I spent with them is invaluable, loving, and memorable.</p><h4>What I gained in my life</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WTGsn6QkmzfcQC4S_xMy4A.png" /></figure><p>If I say that 2025 was one of the best years I’ve ever had, then there must be something lasting that I gained from it. Otherwise, it would have been good only for the moment — not for the future.</p><p>Behind the loving memories, the incredible people, and the unforgettable events, there were powerful lessons. And those lessons are what truly made 2025 valuable.</p><p><strong>1. Alone We Are Limited. Together We Are Powerful.</strong></p><blockquote>What we can achieve alone is limited. But as a team, those limits expand into something extraordinary.</blockquote><blockquote>Great things are built through the small contributions of individuals step by step with all moving toward a shared goal. Applying this to my personal life, I realized that big dreams become reality through consistent small actions taken with direction.</blockquote><blockquote>For large systems — whether a country or a corporation — it’s the aligned effort of individuals working toward a clear goal that creates long-term greatness.</blockquote><p><strong>2. The Power of Shared Energy</strong></p><blockquote>Playful, positive, like-minded people are rare. If you find them, keep them and build strong, lasting connections.</blockquote><blockquote>But success requires more than just a team. A team must share a clear goal. Ego exists in every human and that’s natural. But a truly great team consists of individuals who can balance personal ego with collective purpose. That balance creates extraordinary outcomes.</blockquote><p><strong>3. Divide and Conquer Works — In Life Too</strong></p><blockquote>We accomplished massive events by breaking them down into small, actionable tasks and assigning them strategically. At first, many of those events felt impossible. But once divided into smaller parts, they became manageable — and eventually successful.</blockquote><blockquote>This principle applies directly to life. Huge dreams, ambitious goals, large businesses, corporate organizations, everything can be conquered by breaking it down into small, executable steps and completing them one by one. Divide and conquer is not just a strategy but it’s a life philosophy.</blockquote><p><strong>4. Human Connection Is a Life Asset</strong></p><blockquote>Human connections are one of the greatest assets you can add to your life portfolio. During my university years and even in my early corporate journey, I wasn’t particularly strong at building meaningful connections. But 2025 gave me the opportunity to learn and refine that skill.</blockquote><blockquote>The ability to communicate in a way that makes others feel accepted, valued, and truly heard without judgment builds lasting relationships. I consciously practiced this, and I believe I’ve added it as a permanent skill in my system.</blockquote><p><strong>5. Everyone Has a Unique Superpower</strong></p><blockquote>Every person has a unique skill set, which is something they can do better than others. Identifying those strengths and aligning them with a larger goal creates a competitive advantage.</blockquote><blockquote>In workplaces, organizations, or even countries, placing the right people with the right skills in the right roles enhances the quality of the final outcome. I believe the ability to observe and recognize these strengths in individuals has now become part of my default thinking.</blockquote><p><strong>6. Growth Lives Outside the Comfort Zone</strong></p><blockquote>Putting yourself out there, meeting new people, taking on responsibilities, and exposing yourself to unfamiliar situations open doors in unexpected ways.</blockquote><blockquote>Staying home feels safe. Comfort feels relaxing. But there is no real growth inside a comfort zone. I can confidently say that. When you place yourself in environments with a high probability of opportunity, your chances of growth multiply over time.</blockquote><p><strong>7. People Are More Interesting Than We Assume</strong></p><blockquote>People are not dangerous, boring, or unapproachable as they may initially seem. Don’t judge from a distance. its all depends on how you approach them.</blockquote><blockquote>Talk to them. Listen to their stories. You’ll realize that everyone carries an interesting life story with experiences you’ve never lived and perspectives you’ve never considered. Listening to others expands your mind in ways isolation never can, and you will realize they are normal people who live normal life as you, but with different tastes.</blockquote><p><strong>8. The Skill of Managing Complexity</strong></p><blockquote>Balancing corporate responsibilities with additional commitments is a critical life skill.</blockquote><blockquote>Yes, you can simplify life by limiting responsibilities, but you may also limit your growth by missing great chances that could be yours. Taking on diverse responsibilities, managing complex tasks, and handling pressure builds mental capacity.</blockquote><blockquote>It strengthens your “mind muscle.” And that expanded capacity opens the path toward the kind of greatness you once dreamed of as a child.</blockquote><p><strong>9. Enjoy the Process</strong></p><blockquote>If you focus only on the outcome, you will miss the real thrill of the journey.</blockquote><blockquote>The moment you achieve something, your mind immediately asks, <em>“What’s next?”</em> That’s why true happiness doesn’t live in the destination, but it lives in the process. In the effort, the learning, the little progress, and the growth along the way. The game becomes playful and meaningful only when you enjoy playing it.</blockquote><p><strong>10. Control Your Response, Not the Events</strong></p><blockquote>You can set a clear ultimate goal, but you cannot control everything that happens in life. Instead of labeling events as good or bad, see them as experiences. Every experience gives you something valuable if you frame it correctly.</blockquote><blockquote>When I got laid off, I chose not to stay frustrated. I focused on my next move. That mindset turned a setback into a double profit. Life may be unpredictable, but your response defines your direction.</blockquote><h4>Conclusion</h4><p>I don’t think a long conclusion is needed. Anyone who reads this will understand why I say 2025 was the year that <strong>rewired my nervous system with firmware I never even dreamed of. </strong>This wasn’t just a year of events. It was a year of upgrades — in mindset, resilience, connection, execution, and awareness.</p><p><strong><em>If you take something from my 2025 story, let it be this:</em></strong></p><blockquote>Yes, luck exists. But luck does not come looking for you while you stay at home inside your comfort zone. You have to place yourself where luck is present.</blockquote><blockquote>Put yourself in rooms where things are happening. Expose yourself to challenges. Meet people. Take responsibility. Move.</blockquote><blockquote>Luck visits those who show up. And when preparation meets exposure — what people call “luck” becomes inevitable.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fde48fbdf89" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retrieving Wisdom from Coursera Courses: How I Built an AI Bot with RAG and LangChain]]></title>
            <link>https://jayamaljayamaha.medium.com/retrieving-wisdom-from-coursera-courses-how-i-built-an-ai-bot-with-rag-and-langchain-882650cc4b05?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/882650cc4b05</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Wed, 18 Jun 2025 15:43:46 GMT</pubDate>
            <atom:updated>2025-06-18T15:43:46.720Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uNg-uCmAnfa6sZtE" /></figure><p>Recently, I completed the IBM AI Engineering course on the Coursera platform, and I can confidently say it was a rich and rewarding experience. The course provided a strong balance between theory and hands-on practice, offering deep insights into the field of AI.</p><p>But knowledge, no matter how powerful, is meaningless if it’s not applied. It’s like a sharp sword never taken to battle. It has full potential, yet is gathering dust. Leaving knowledge unused is not just a missed opportunity; it’s almost a disservice to ourselves, to humanity, and to the universe.</p><p>As we say in Sinhala: <strong>“යුද්දෙට නැති කඩුව කොස් කොටන්නද?”</strong></p><p>So, I decided to put what I had learned into action. While exploring how to begin, I had a moment of insight — why not use this knowledge to improve my own learning process? If I could build a bot capable of retrieving information through concepts taught in the course, just like a question-answering system. It would allow me to quickly recall the parts I’d forgotten. This way, I could reinforce and apply the concepts more effectively, without the need to rewatch course videos repeatedly.</p><p>Before diving into the implementation, it’s important to understand what RAG (Retrieval-Augmented Generation) and LangChain are, and how they can be used to build customized Large Language Model (LLM) applications tailored to specific needs.</p><h4>What is RAG</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OzLdmzW_NPcBcRa4" /></figure><p><strong>RAG (Retrieval-Augmented Generation)</strong> is a technique that enhances the performance of Large Language Models (LLMs) by integrating external knowledge sources into the generation process. Instead of relying solely on the LLM’s pre-trained internal knowledge, RAG dynamically retrieves relevant information from a custom or external knowledge base</p><p>Large Language Models (LLMs) are trained on vast amounts of publicly available and commonly used text data. While this makes them highly effective for general-purpose tasks, their knowledge is inherently limited to the data they were trained on.</p><p>To truly leverage the power of LLMs for specific domains or personalized use cases, we need to provide them with relevant contextual knowledge, drawn from our own custom knowledge bases. This is where a technique called <strong>Retrieval-Augmented Generation (RAG)</strong> comes into play. RAG enables LLMs to access external data sources at runtime, enhancing their responses with up-to-date or domain-specific information.</p><p>The typical workflow of RAG looks like this:</p><ul><li>Indexing phase<br>This phase starts with collecting custom knowledge from text files, PDFs, databases, or online sources. The data is then split into smaller, manageable chunks to ensure efficient retrieval without losing context.</li><li>Embedding and Storing phase<br>In this phase, the text chunks are converted into numerical vector representations using an embedding model. These vectors, along with the original text chunks, are then stored in a vector database for efficient retrieval later.</li><li>Retrieval phase<br>In this phase, the user’s query is also converted into a vector embedding using the same embedding model, and the RAG system performs a similarity search in the vector database to retrieve the most relevant chunks.</li><li>Augmentation phase<br>In this phase, the retrieved relevant chunks are combined with the user’s original query to form a prompt, which is then passed to the underlying LLM to generate the final output.</li><li>Generation Phase<br>In this phase, the LLM receives the augmented prompt and generates a human-readable, contextually relevant response by leveraging its language understanding capabilities.</li></ul><h4>What is LangChain</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tVbNLlkUrXtz3jp1" /></figure><p>LangChain is an orchestration framework that brings all components together, managing and coordinating the different phases of advanced agentic AI systems. In our specific use case, LangChain is responsible for…</p><ul><li><strong>Get user input and pass it for indexing.</strong><br>LangChain provides dedicated loaders for various file types, which handle the processing and chunking of data automatically. This allows LangChain to seamlessly manage the data ingestion phase in our pipeline.</li><li><strong>Knowledge base Management.</strong><br>LangChain offers tools for generating embeddings and integrating with vector databases. After chunking, it passes the segments to an embedding model and stores the resulting vectors efficiently in a connected vector database.</li><li><strong>Contextual Retrieval.</strong><br>LangChain enables the embedding of user queries and, retrieval of relevant context from a vector database.</li><li><strong>Context Augmentation.</strong><br>LangChain enables dynamic prompt construction using templates and augments this capability by incorporating retrieved relevant documents. This process enhances the accuracy and relevance of responses generated by the underlying language model.</li><li><strong>Final Result Generation.</strong><br>LangChain ties all the components together, sequencing them into a cohesive pipeline that orchestrates the flow from user input to the final response. It routes the constructed prompt through the LLM, and once the output is generated, LangChain can process and format the response using its built-in or custom-defined parsers.</li></ul><h4>How the AI Bot was Built</h4><p>Now that we have a solid understanding of RAG and LangChain, it’s time to internalize these concepts through hands-on practice. To truly embed this knowledge into our nerves, we need to apply it step by step. Let’s dive into how I implemented this bot, demonstrating the practical application of these powerful technologies.</p><p><strong>Step 1: Prepare the knowledge base</strong></p><p>The first step is to prepare our knowledge base. We want our bot to respond to user queries using a specific, curated knowledge set rather than relying solely on the general knowledge the LLM was trained on. This knowledge can come from various sources such as text files, PDF documents, online resources, or external databases. For this project, I downloaded the transcripts of Coursera course videos as text files, which were then categorized and organized into a dedicated folder.</p><p><strong>Step 2: Download required libraries</strong></p><pre># Install Libraries<br>!pip install torch<br>!pip install transformers langchain_community langchain_text_splitters langchain_core<br>!pip install sentence-transformers<br>!pip install chromadb<br>!pip install huggingface_hub<br>!pip install accelerate<br>!pip install gradio</pre><p>For this task, we require the following libraries. Here’s a brief overview of their roles:</p><blockquote><strong>transformers</strong> – A high-level API for working with large language models (LLMs), which we’ll use for model integration.</blockquote><blockquote><strong>langchain_community</strong> – Provides various document loaders (e.g., for text and PDF files), supports the LangChain pipeline, embedding models, and vector databases.</blockquote><blockquote><strong>langchain_text_splitters</strong> – Helps break down long documents into manageable chunks for better processing and embedding.</blockquote><blockquote><strong>langchain_core</strong> – Offers essential components like output parsers and prompt templates using LangChain Expression Language (LCEL).</blockquote><blockquote><strong>chromadb</strong> – A vector database used to store and retrieve document embeddings efficiently.</blockquote><blockquote><strong>accelerate</strong> – Optimizes model loading and execution by intelligently leveraging available hardware, such as CPUs and GPUs.</blockquote><blockquote><strong>gradio</strong> – Enables us to build interactive user interfaces, especially suited for showcasing machine learning models.</blockquote><p><strong>Step 3: Import the required packages from the downloaded libraries</strong></p><pre># Module Imports<br>import os<br>import torch<br>from langchain_community.llms import HuggingFacePipeline<br>from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline<br>from langchain_community.document_loaders import TextLoader<br>from langchain_community.document_loaders import PyPDFLoader<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br>from langchain_community.embeddings import HuggingFaceEmbeddings<br>from langchain_community.vectorstores import Chroma<br>from langchain_core.runnables import RunnablePassthrough<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.prompts import ChatPromptTemplate<br>import gradio as gr</pre><p><strong>Step 4: Document preprocess stage</strong></p><p>In this stage, we write functions to prepare files for the chunking process, define file loaders based on file types, and then read and split large text data into manageable chunks.</p><blockquote>This function handles loading files from a specified location by type and renames them to a standardized format for consistent processing.</blockquote><pre>def preprocess_data_files():<br>    for filename in os.listdir(folder_path):<br>        full_old_path = os.path.join(folder_path, filename)<br>        new_filename = filename.replace(&quot;,&quot;,&quot;&quot;).replace(&quot; &quot;, &quot;-&quot;)<br>        full_new_path = os.path.join(folder_path, new_filename)<br>        os.rename(full_old_path, full_new_path)<br><br>    txt_list: list[str] = [filename for filename in os.listdir(folder_path) if os.path.splitext(filename)[1] ==&#39;.txt&#39;]<br>    pdf_list: list[str] = [filename for filename in os.listdir(folder_path) if os.path.splitext(filename)[1] ==&#39;.pdf&#39;]<br>    return txt_list, pdf_list</pre><blockquote>Next, we define functions to load files based on their type. One function is designed specifically for loading plain text files, while another handles PDF files</blockquote><pre># TXT loader<br>def text_loader(txt_filepath: str):<br>    txt_loader = TextLoader(txt_filepath)<br>    return txt_loader.load()<br><br># PDF loader<br>def pdf_file_loader(pdf_filepath: str):<br>    pdf_loader = PyPDFLoader(pdf_filepath, extract_images=False,)<br>    return pdf_loader.load()</pre><p>Next, we define a function to split the file content into smaller, manageable chunks. We use theRecursiveCharacterTextSplitter which iteratively breaks down large text blocks based on a specified maximum character length while maintaining a fixed overlap with the previous chunk. This overlapping strategy helps preserve the continuity of context across chunks, allowing the embedding model to generate more coherent and meaningful vector representations.</p><pre>def split_text_for_chunks(document):<br>    text_splitter= RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)<br>    return text_splitter.split_documents(document)</pre><p><strong>Step 5: Model Creating Stage.</strong></p><p>In this step, we define the LLM using Hugging Face’s transformers pipeline, which provides a high-level, unified interface for working with language models. The pipeline abstracts away the complexity of tasks such as tokenization, text generation, and contextual processing, allowing us to configure parameters like the model, tokenizer, maximum tokens, and temperature with ease. LangChain supports seamless integration with Hugging Face pipelines through its wrappers, making it straightforward to incorporate these models into LCEL (LangChain Expression Language) chains.</p><pre>def define_LLM():<br>    model_name: str = &#39;mistralai/Mistral-7B-Instruct-v0.2&#39;<br>    tokenizer = AutoTokenizer.from_pretrained(model_name)<br>    model = AutoModelForCausalLM.from_pretrained(<br>        model_name,<br>        torch_dtype=torch.bfloat16,<br>        device_map=&#39;auto&#39;<br>    )<br>    pipe = pipeline(<br>        &#39;text-generation&#39;,<br>        model=model,<br>        tokenizer=tokenizer,<br>        max_new_tokens=1024,<br>        temperature=0.7,<br>        do_sample=True,<br>        top_p=0.95,<br>        repetition_penalty=1.1<br>    )<br><br>    return HuggingFacePipeline(pipeline=pipe)</pre><p><strong>Step 6: VectorDB Creating Stage.</strong></p><p>In this stage, we initialize both the embedding model and the vector database. The embedding model is responsible for converting each text chunk into a high-dimensional vector representation. These vectors, along with their associated text or metadata, are then stored in the vector database. This setup enables efficient semantic search and retrieval based on the meaning of the user’s query.</p><blockquote>This function initializes the embedding model using the BAAI/bge-small-en-v1.5 model, with customized arguments tailored for our use case.</blockquote><pre>def initialize_embedding_model():<br>    embedding_model_name = &quot;BAAI/bge-small-en-v1.5&quot;<br>    embeddings_model = HuggingFaceEmbeddings(<br>        model_name=embedding_model_name,<br>        model_kwargs={&#39;device&#39;: &#39;cpu&#39;},<br>        encode_kwargs={&#39;normalize_embeddings&#39;: True}<br>    )<br>    return embeddings_model</pre><blockquote>This function initializes the vector database. If a database file already exists, it loads the existing data; otherwise, it creates a new database instance. At this stage, we don’t index any text chunks yet, as we handle indexing later in bulk due to the large volume of data to be stored.</blockquote><pre>def intialize_vector_DB(embeddings_model):<br>    vector_db_path = &quot;./chroma_db_hf_embeddings&quot;<br>    if os.path.exists(vector_db_path) and os.listdir(vector_db_path):<br>        vector_db = Chroma(<br>            persist_directory=vector_db_path,<br>            embedding_function=embeddings_model<br>        )<br>    else:<br>        vector_db=Chroma.from_documents(<br>            documents=[],<br>            embedding=embeddings_model,<br>            persist_directory=vector_db_path<br>        )<br>        vector_db.persist()<br>    return vector_db</pre><blockquote>Next, we define a function to add individual chunks of a document to the vector database and persist them. This function is necessary because, as mentioned earlier, we delayed adding chunks until after the vector database has been initialized, rather than during its initialization.</blockquote><pre>def load_chunks_to_vector_db(doc_chunks, vector_db):<br>    vector_db.add_documents(doc_chunks)<br>    vector_db.persist()</pre><blockquote>Next, we write a function that reads formatted file paths from arrays of text and PDF files, loads each file using the appropriate loader, splits the content into manageable chunks, and finally adds these chunks to the vector database.</blockquote><pre>def read_files_and_load(txt_list: list[str], pdf_list: list[str], vector_db):<br>    for txt_filename in txt_list:<br>        full_path = os.path.join(folder_path, txt_filename)<br>        loaded_file = text_loader(full_path)<br>        chunks = split_text_for_chunks(loaded_file)<br>        load_chunks_to_vector_db(chunks, vector_db)<br><br>    for pdf_filename in pdf_list:<br>        try:<br>            full_path = os.path.join(folder_path, pdf_filename)<br>            loaded_file = text_loader(full_path)<br>            chunks = split_text_for_chunks(loaded_file)<br>            load_chunks_to_vector_db(chunks, vector_db)<br>        except Exception as e:<br>            print(f&#39;Exception occured when reading {pdf_filename}&#39;)</pre><p><strong>Step 7: LangChain Creating Stage.</strong></p><p>Here, we assemble all the functions we created earlier to build the final LangChain workflow by invoking them sequentially in the correct order. After initializing LangChain, we take the user’s query, process it through the chain, and generate an answer by retrieving relevant knowledge from our knowledge base via the RAG process.</p><blockquote>The following function initializes LangChain by assembling the previously created functions. Here, we also craft a well-engineered prompt template designed to elicit accurate and efficient responses from the LLM.</blockquote><pre>def create_langchain():<br>    txt_list, pdf_list = preprocess_data_files()<br>    embedding_model = initialize_embedding_model()<br>    vector_db = intialize_vector_DB(embedding_model)<br>    read_files_and_load(txt_list, pdf_list, vector_db)<br>    retriever = vector_db.as_retriever(search_kwargs={&quot;k&quot;: 3})<br>    llm = define_LLM()<br>    prompt = ChatPromptTemplate.from_template(template)<br>    rag_chain = (<br>        {&quot;context&quot;: retriever, &quot;question&quot;: RunnablePassthrough()} | prompt | llm | StrOutputParser()<br>    )<br>    return rag_chain<br><br>qa_rag_chain = create_langchain()</pre><blockquote>The prompt template we used is</blockquote><pre>template = &quot;&quot;&quot;You are an AI assistant for question-answering tasks.<br>            Use the following pieces of retrieved context to answer the question concisely.<br>            If you don&#39;t know the answer, just say that you don&#39;t know.<br>            Keep the answer to three sentences maximum.<br>            <br>            Question: {question}<br>            Context: {context}<br>            Answer:&quot;&quot;&quot;</pre><blockquote>The following function invokes LangChain by receiving the user’s question, which will be collected through a user interface created in the next step. It then processes LangChain’s response to extract the final answer and returns it to the user.</blockquote><pre>def process_query(user_question):<br>    print(user_question)<br>    if not user_question:<br>        return &quot;Please enter a query to get a response.&quot;<br>    try:<br>        response = qa_rag_chain.invoke(user_question)<br>        answer_prefix = &quot;Answer:&quot;<br>        answer_start_index = response.find(answer_prefix)<br>        if answer_start_index != -1:<br>            extracted_answer = response[answer_start_index + len(answer_prefix):].strip()<br>        else:<br>            print(&quot;Warning: &#39;Answer:&#39; prefix not found in response. Printing full output.&quot;)<br>            extracted_answer = &quot;&quot;<br>        return extracted_answer<br>    except Exception as e:<br>        print(f&quot;An error occurred: {e}&quot;)</pre><p><strong>Step 8: Create A User Interface.</strong></p><blockquote>In this function, we create an interactive user interface using <strong>Gradio</strong>, a Python library specifically designed for building UIs for AI applications. We implement a simple interface consisting of a single text box for user input and an output box to display the model’s response.</blockquote><pre>demo = gr.Interface(<br>    fn=process_query, <br>    inputs=gr.Textbox(<br>        lines=5,<br>        label=&quot;Your Input Query:&quot;,<br>        placeholder=&quot;Type your question here...&quot;<br>    ),<br>    outputs=gr.Textbox(<br>        lines=10,<br>        label=&quot;Output Response:&quot;,<br>        interactive=False<br>    ),<br>    title=&quot;Simple RAG Query Interface&quot;,<br>    description=&quot;Enter a query and get a response from the (simulated) RAG system.&quot;<br>)<br><br>demo.launch()</pre><h4>Final Product Demo</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QVXSc-Q5x_nifSlK" /></figure><p>This is a demo of the final product we built using the concepts and code discussed above. You may notice that response generation takes some time. This is due to the size of our knowledge base and the limitations of our computing resources. In the next article, we’ll explore the root causes behind this latency and discuss strategies to improve performance significantly.</p><p>Instead of just reading this article, I encourage you to apply what you’ve learned by building your own question-answering bot using a custom knowledge base. It’s the best way to reinforce the concepts in a practical, hands-on way.</p><p>You can find the GitHub repository for this project below — feel free to fork it, modify it, and make it your own!</p><ul><li><a href="https://github.com/DisorderedJJ/AI-Arsenal/tree/main/coursera-course-bot">AI-Arsenal/coursera-course-bot at main · DisorderedJJ/AI-Arsenal</a></li><li><a href="https://drive.google.com/file/d/1Mta6UcHRffjn6nhCTWa6Kam9N_6MBvNY/view?usp=sharing">QA Bot Demo.mp4</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=882650cc4b05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mathematics for AI ??]]></title>
            <link>https://jayamaljayamaha.medium.com/mathematics-for-ai-c956f6205e1d?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c956f6205e1d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[mathematics-for-ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Wed, 04 Jun 2025 16:29:36 GMT</pubDate>
            <atom:updated>2025-06-04T16:29:36.230Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mathematics for AI ??</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TenOLwbEOKeYRLPq" /></figure><p>This is the next continuation of our AI, ML, and DL article series. As we discussed in previous articles, now we know the core reactor of AI, ML and DL technologies are mathematical concepts. They predict future indicdents based on patterns previously happend by leveraging mathematics. So in this article we are going to explore how mathematics will give the power to these technologies to become a divine eye (divine eye means the ability to see future using 6th sense)</p><h4>What is the role of mathematics?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4IS_V1nhld70Z5hL" /></figure><p>If we examine the majority of groundbreaking inventions, theories, and discoveries throughout history, we find that their foundation is mathematics. However, mathematics itself is not just a static tool which existed since the beginning of the universe — it is a human discovery, shaped and refined through continuous contributions over centuries.</p><p>Humans are highly conscious beings, unlike other living creatures. The human mind doesn’t easily accept phenomena as they are. We want to feel them in our soul — we need to know why and how. We call this desire (the very thing that keeps us attached to the cycle of life, as Lord Buddha described — and that’s perfectly okay; after all, we’re all human) the curiosity of the human mind. We sense that something is hidden within these phenomena, but we can’t fully feel it, and more importantly, we can’t control it. This lack of control creates ambiguity, and some minds are driven to resolve that ambiguity with clarity and reason.</p><p>Humans gradually uncovered a framework called mathematics as a solution to that ambiguity, and it perfectly resonated, filling the gap between universal phenomena and how we experience them. This framework evolved step by step, each layer building on the previous one, enabling us to describe increasingly complex phenomena with precision and logic.</p><p>Take an apple falling from a tree — we see it, we know something is happening, but it doesn’t quite reach our soul. This ambiguity doesn’t let rational minds stay calm. Some humans, particularly those driven by a need for clarity and understanding, couldn’t settle for vague interpretations. Their minds craved something more precise. So they began developing a framework, layer by layer — one that could convert these universal phenomena into structured understanding, and then translate that understanding back into action to influence and control those phenomena once again, in an attempt to feel and grasp the truth more tangibly.</p><p>To truly understand and eventually manipulate these phenomena, they had to be made <em>measurable</em> and <em>quantifiable</em>. However, our thoughts, emotions, and even many of our perceptions are not inherently measurable. In order to bridge this gap between human experience and external reality, we needed a language. A language that allows us to translate phenomena into the measurable and quantifiable domain, and then back into phenomena we can understand and manipulate. That’s where mathematics comes in. It was invented as a universal, quantifiable language, one that could act as a broker between the intangible desires of the human mind and the physical laws of the universe.</p><h4>Intuition in mathematics for AI.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o03iReYRvlwZyIC4" /></figure><p>If you observe intentionally with a rational mind, there is no such thing as black and white in the universe. All events are happening within a range. From the energy spectrum in this universe to human behavior, everything unfolds as a spectrum of possibilities. In that sense, everything is possible in this universe; it’s just a matter of probability (the likelihood of an event happening, in simple terms).</p><p>If we take an example for simplicity, consider the act of driving a vehicle. From successfully reaching your destination to getting into an accident and total destruction, every outcome is possible, just with different probabilities. If you get drunk and drive, the probability of an accident is much higher, but that doesn’t mean you’re definitely going to crash. And if you drive while obeying all the traffic laws, the probability of an accident is lower, but that doesn’t guarantee you’ll reach your destination safely either.</p><p>Since many possibilities exist, each of those possibilities leads to many effects. And every one of those effects, in turn, creates more possibilities. From this, a giant web of cause and effect emerges in the universe, where one cause leads to an event, and that event becomes the cause for another. This vast network of cause and effect, filled with infinite probabilities, forms a deeply complex and interconnected system. We call this the <strong>law of causality</strong> — the natural order of the universe. As souls traveling through this universe, we are also bound to and intertwined with this intricate web.</p><p>So, as things become more complex and interconnected, a kind of event mechanism emerges — one that operates according to patterns. Through mathematics, we can begin to grasp this pattern-like behavior. Mathematics, developed through human perception, allows us not only to understand these patterns but also to influence them by identifying and controlling the key influencing factors.</p><h4>So maths for AI??</h4><p>As we discussed in the previous article, what AI basically does is capture the patterns of an incident and predict an outcome based on those patterns. Our human brain cannot consciously catch and process the patterns that are part of the complex web of causality. (Though they can be sensed spiritually — but that’s a topic for another time.) And that’s where mathematics comes in to save us in this logical realm.</p><p>Using mathematics, we can implement mechanisms to detect hidden patterns and identify the different components that influence those patterns within the web of causality. Then, we can build a model that runs on computational hardware, capable of predicting how the causality web will behave when we influence different parameters in different ways. This is what we call <strong>AI (Artificial Intelligence).</strong></p><h4>Conclusion</h4><p>So that is why mathematics will be the driving force behind AI, enabling it to absorb universal phenomena, make predictions, and eventually influence them. Hope you got a good understanding about why we needed maths and why we use that in AI. Maybe you have a different perception about that, so let me know your thoughts as well in the comment section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c956f6205e1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Actually Happening inside AI, DL, or ML]]></title>
            <link>https://jayamaljayamaha.medium.com/what-is-actually-happening-inside-ai-dl-or-ml-83333368eb27?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/83333368eb27</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[mathematics-for-ml]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Tue, 15 Apr 2025 15:11:54 GMT</pubDate>
            <atom:updated>2025-04-15T15:11:54.561Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6GteFgEaaJYrzSNP" /></figure><p>In today’s era of AI hype, many people perceive Artificial Intelligence, Machine Learning, and Deep Learning systems as some kind of alien technology, mysterious, magical, and beyond human understanding. There’s a common misconception that these systems deliver perfectly intelligent and accurate results without any flaws. As a result, people have some fear towards it and often assume there’s no need to understand how they work, just use them and trust the outcome. However, this belief is misleading.</p><p>If I told you that AI, Deep Learning (DL), and Machine Learning (ML) systems are fundamentally built on mathematical operations used to recognize, learn, and model patterns from real-world phenomena, and use that mathematical model to predict future outcomes, would you believe it? It’s true. At their core, these systems rely on complex mathematical calculations to generate new content, predict outcomes, and analyze data probabilistically based on previously learned patterns. While the processes may seem mysterious, they are grounded in mathematics, just at a level of complexity that can be challenging for humans to follow without the aid of computational tools.</p><p>Major inventions and revolutions do not emerge in the world spontaneously — they manifest for a reason, aligned with the needs and conditions of their time. When we seek to deeply understand such transformative technologies, especially those that resonate with us on an intrinsic level.</p><h4>Then why do we need computers?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4-wCLfS0Cs2AyKlZ" /></figure><p>This is a billion-dollar question. If artificial intelligence (AI), machine learning (ML), and deep learning (DL) are fundamentally based on mathematical concepts, then why are they categorized as part of computer science?</p><p>The answer lies in the very reason computers were invented. Humans created computers to perform calculations faster and more accurately than the human brain. While the human brain is capable of a wide range of functions — such as reasoning, emotions, creativity, and memory — it is not optimized for performing large-scale mathematical operations quickly and consistently. To handle the complexity and scale of mathematical computations required in AI and related fields, we needed a specialized tool. That tool is the modern computer.</p><p>As time progressed, computer technology matured significantly. Humans developed more efficient mechanisms for performing specific tasks by enhancing computer architecture by adding new hardware components like memory, CPU with multiple cores, parallel processing units like GPUs, introducing abstraction layers through software, enabling parallel processing, and improving context switching speed. Unlike the human brain, which is not optimized for rapid task switching or repetitive computations, computers can perform such operations with remarkable efficiency and consistency. This technological evolution opened up a new dimension of possibilities, allowing humans to greatly enhance their productivity by leveraging the strengths of machines.</p><h4>Evolution of computers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yRKIgpOaLfm-4lOX" /></figure><p>In the early days, humans performed all the mathematical computations related to machine learning manually. Even the foundational concepts behind <strong>Deep Learning</strong> and <strong>Artificial Intelligence</strong> were initially built upon <strong>machine learning algorithms</strong>. However, due to the growing complexity of these calculations, it eventually became impractical for humans to handle them without assistance.</p><p>As time progressed, organizations and the open-source community began introducing <strong>abstraction layers</strong> to manage hardware more efficiently, leading to the development of <strong>operating systems</strong>. These systems made it easier to interact with the underlying hardware without needing to write machine-level code.</p><p>Following this, individuals and researchers introduced even higher layers of abstraction in the form of <strong>programming languages</strong>. Over time, robust ecosystems formed around these languages with the creation of <strong>supporting libraries and modules</strong>. These included:</p><ul><li><strong>Data structures</strong> to efficiently manage and organize collections of data</li><li><strong>Algorithms</strong> for operations like sorting, searching, and insertion</li><li><strong>Mathematical modules</strong> for performing complex computations such as mean, max, min, square root, etc.</li><li><strong>Concurrency and parallel processing modules</strong> to execute multiple tasks simultaneously</li><li><strong>GPU-based processing</strong> for accelerating deep learning and heavy computational workloads</li></ul><p>Today, the computer has become a well-equipped powerhouse capable of solving the <strong>complex and computationally intensive demands</strong> of machine learning and deep learning algorithms. Tasks such as <strong>regression analysis</strong>, <strong>convolutions</strong>, and <strong>matrix operations</strong>, which were once tedious or impossible to perform manually, can now be executed with incredible speed and precision.</p><p>Recognizing this computational power, <strong>AI researchers and scientists</strong> began to ask: <em>Why not fully leverage this to solve AI, ML, and DL challenges?</em></p><p>These algorithms require the ability to:</p><ul><li><strong>Compute large volumes of operations at high speed</strong></li><li><strong>Store and manipulate vast numbers of variables in parallel</strong></li><li><strong>Efficiently structure, cache, and retrieve intermediate results</strong></li><li><strong>Permanently save final outputs in memory without relying on physical storage like paper</strong></li></ul><p>All of these capabilities are inherent in modern computer systems, delivered in a way that surpasses the limits of the human brain in both speed and accuracy.</p><p>Starting with simple machine learning algorithms, researchers gradually moved toward <strong>more complex models</strong>, such as <strong>deep learning architectures</strong>, <strong>neural networks</strong>, and <strong>multi-component AI systems</strong>. And this evolution is far from over — the field continues to grow, pushing boundaries further with each advancement.</p><h3>What is next?</h3><p>To truly understand what’s happening beneath the surface of <strong>Artificial Intelligence (AI)</strong>, <strong>Machine Learning (ML)</strong>, and <strong>Deep Learning (DL)</strong>, it’s essential to explore their <strong>origins and foundational structures</strong>. Gaining this understanding requires us to <strong>deconstruct the complexity step by step</strong>, examining each conceptual and technical layer with care and clarity.</p><p>In this upcoming article series, I will guide you through that journey, breaking down these intricate topics into digestible parts by addressing some of the most fundamental and thought-provoking questions as follows.</p><ul><li>What is the role of mathematics and concepts used in AI, ML and DL?</li><li>Why are patterns important and their usage in AI, ML and DL?</li><li>How do statistics and probability relate to this field?</li><li>How have the mathematical concepts of learning evolved?</li><li>What role does this technology play in today’s society?</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83333368eb27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Artificial Intelligence, Deep Learning, and Machine Learning Series]]></title>
            <link>https://jayamaljayamaha.medium.com/artificial-intelligence-deep-learning-and-machine-learning-series-d86c6755adf5?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d86c6755adf5</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Sat, 12 Apr 2025 06:22:58 GMT</pubDate>
            <atom:updated>2025-04-12T06:22:58.071Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wVvW4Kzw098QEa78.jpg" /></figure><p>The entropy of the world is increasing at an unprecedented rate. If I repeat that in an alternative way, the world is advancing at an accelerating pace like never before. This rapid progression is largely driven by technological advancements. Throughout history, several critical events have significantly contributed to the rise of technology. One of the most groundbreaking milestones was the discovery of nuclear energy, which revolutionized the world and set humanity on a new trajectory. From my perspective, the next technological breakthrough with an equally transformative impact is the rise of Artificial Intelligence (AI), following in the footsteps of nuclear energy as a game-changing force in human history.</p><p>As humans, we are naturally resistant to change and often fear it. Historically, we tend to embrace change and adapt only when faced with existential threats. For example, during the time of nuclear energy discovery, the world was engaged in World War. Humanity had no choice but to adapt to nuclear advancements. Believe it or not, this rapid technological evolution may soon become our next existential threat. Technology is deeply interconnected with the economy, which, in turn, influences politics. Politics shapes society, beliefs, and even our very existence. This creates a continuous cycle where each factor impacts the next, driving an ever-evolving and interdependent world.</p><p>So why do we wait until the last moment? As wise individuals, we should have the ability to anticipate these shifts and prepare in advance. Instead of being late resistors to the change, we should become early adopters. This is precisely why I began learning about Artificial Intelligence (AI). Through this series, I aim to document my personal journey — sharing my understanding of AI, Deep Learning (DL), and Machine Learning (ML), how I perceive their mechanisms, and how they can be applied to solve real-world problems.</p><h3>AI, DL, ML? Why three terms?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BVGxvKJe2egYSMND" /></figure><p>You may be wondering what the meanings and true differences of Artificial Intelligence (AI), Deep Learning (DL), and Machine Learning (ML)</p><p>The events in this world often follow recognizable patterns. Algorithms aim to predict outcomes by analyzing these patterns in events or objects. To do this effectively, we must first train the algorithms on relevant patterns so they can learn and make accurate predictions.</p><p>In that sense</p><blockquote>Machine Learning is a technique used to train mathematical algorithms on data that represents patterns in objects or past events. Once trained, these algorithms can predict the probability of future outcomes. ML is often used to identify patterns with linear relationships between input features and outputs. This basic form of pattern recognition is applied in tasks like spam detection (learning from past spam), housing price prediction (based on historical data), and object classification (by analyzing similarities).</blockquote><blockquote>Deep Learning is a technique used to recognize more advanced patterns that traditional Machine Learning often cannot capture. It uses a layered approach, allowing the model to identify complex, non-linear relationships between input features and outputs. To achieve this, deep learning models incorporate activation functions and are structured as neural networks, designed to mimic the way neurons work in the human brain. Instead of relying on a single algorithm, deep learning builds a network of interconnected layers and optimizes it by applying learning algorithms across multiple pathways.</blockquote><blockquote>In AI, we build neural networks along with other essential components to mimic human learning from past mistakes and emulate human intelligence. Instead of relying solely on neural networks, AI systems incorporate rule-based logic, reasoning, search algorithms, and more to make intelligent decisions. A good example is self-driving cars. Beyond recognizing patterns in video feeds, they must make real-time, rational decisions — like reacting to sudden obstacles or unexpected behavior on the road — which goes beyond just learning from past data.</blockquote><h4>Conclusion</h4><p>As we’ve discussed above, we now have a foundational understanding of the importance of staying aware of the rapid advancements in AI, Machine Learning (ML), and Deep Learning (DL). This is just the beginning of a series where we will dive deeper into the fascinating world of emerging technologies.</p><p>This article wraps up our introductory exploration, but in the upcoming posts, I’ll be sharing more advanced insights and practical knowledge from the field. So stay tuned as we continue to navigate and swim through this exciting tech paradigm.</p><p>Remember, understanding the need for AI, its core principles, and the mechanisms behind it is far more valuable than simply applying frameworks to build models. By learning the basics, you can uncover hidden patterns and make groundbreaking discoveries.</p><p>Stay curious, stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d86c6755adf5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Incentives vs. Emotional Drive: What Truly Motivates People to Perform Better?]]></title>
            <link>https://jayamaljayamaha.medium.com/incentives-vs-emotional-drive-what-truly-motivates-people-to-perform-better-1a2f25de3205?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a2f25de3205</guid>
            <category><![CDATA[human-psychology]]></category>
            <category><![CDATA[social-science]]></category>
            <category><![CDATA[emotional-triggers]]></category>
            <category><![CDATA[thinking-pattern]]></category>
            <category><![CDATA[incentives]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Sun, 02 Mar 2025 06:06:26 GMT</pubDate>
            <atom:updated>2025-03-02T06:06:26.533Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*usGDdPRHODtIlOMDdbsICw.png" /></figure><p>Throughout life, we engage in various tasks and activities — whether it’s our job, personal goals, or social work. But have you ever paused to reflect on what truly motivates you to do them? Is it the benefits and rewards you gain? Or is it the inner sense of fulfillment and emotional satisfaction? Perhaps it’s something else entirely. Moreover, have you ever compared your performance when working for incentives or personal gain versus when driven by emotional passion or patriotism? This article delves into these questions to explore what truly drives human motivation.</p><h4>Human Psychological Behaviour</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SD6-oKPb37C1nJaPjnTQWw.jpeg" /></figure><p>To understand this better, we need to know how our brain works to drive us toward different activities. Our brain has a reward system controlled by a chemical called dopamine. This chemical is responsible for motivation, pleasure, and goal-setting. Based on our past experiences, our brain learns which activities bring rewards and makes us want to do them again. Because of this, we are naturally drawn to activities that give us the highest dopamine release, making us feel satisfied and motivated.</p><p>Based on this mechanism, humans often engage in activities they do not enjoy because of the future benefits they anticipate. Similarly, we may abandon activities that bring immediate pleasure if they do not provide enough long-term reward. This shows that people can temporarily compromise their dopamine-driven motivation depending on individual brain capacity and specific circumstances. However, such compromise cannot be sustained indefinitely. Humans will not tolerate this compromise in the long term. This is how the human brain has evolved primarily for survival.</p><h4>How does this relate to human activities?</h4><p>As you can understand, the human brain operates on incentives. The primary incentive we receive is the release of dopamine (though the brain’s functions are much more complex, for the sake of this article, I have focused on this concept). All the activities we engage in within this materialistic world ultimately affect our brain, and the motivation and consistency behind these activities are determined by the reward signals our brain receives. For example…</p><p><strong>The current materialistic world operates on a system that…</strong></p><blockquote>Humans work in jobs because they receive financial incentives. With the money they earn, they can buy food, and consuming delicious food triggers a high level of reward in the brain.</blockquote><blockquote>People invest money with the expectation of high returns, which can be used to buy properties such as luxury homes and comfortable vehicles. These possessions provide a sense of achievement, which is ultimately converted into a reward in the brain.</blockquote><blockquote>Humans study and work hard to climb the corporate career ladder or gain political power systematically. This brings a sense of VIP status, respect, and influence, all of which are transformed into rewards in the brain.</blockquote><blockquote>Volunteering provides a sense of self-fulfillment and pride, which also gets converted into a reward in the brain.</blockquote><blockquote>Working due to patriotism gives a feeling of contributing to the homeland and gaining social acceptance, leading to a sense of importance and pride, which ultimately becomes a reward in the brain.</blockquote><blockquote>Living a lifestyle that differs from the average person, such as being wealthy, owning expensive material possessions, or holding high status and privilege, gives a sense of being special and important. Volunteering for social and global causes further enhances this feeling, all of which are transformed into rewards in the brain.</blockquote><h4>So what is the catch….</h4><p>Based on all the points discussed above, you might now wonder — what is the real significance of this? Both incentives and emotional drive contribute to the brain’s reward system, ultimately influencing human actions and performance. However, the real challenge lies in the collective consciousness and the energy vibe that shapes the reality we live in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wCocQ6-Vl79MHf4E" /></figure><p>Collective consciousness is the shared set of concepts, values, ideas, and moral attitudes accepted by the vast majority of a society or multiple societies. This creates an energetic influence that shapes how people perceive and interpret reality. Ultimately, everything that manifests in this world is influenced by collective consciousness and the energy vibe it generates. No matter how hard we try to think rationally and see beyond illusions, we cannot completely escape its influence as long as we exist within the very environment shaped by it.</p><p>so as examples</p><blockquote>Society created money as a necessity for acquiring basic needs like food, shelter, and clothing. To earn it, people engage in economic activities which is a complex system that has been established by society, and the more they earn, the better their access to these essentials.</blockquote><blockquote>Society has established the norm that owning expensive items attracts more attention, which serves as a brain reward. To acquire luxury possessions, one must earn more money and gain respect, making money a key motivator to push beyond personal limits.</blockquote><blockquote>Society has created a system where climbing the career or political ladder earns more respect, granting access to rare opportunities and a clearer path to success — serving as a brain reward. To reach or maintain higher positions, one must work hard.</blockquote><p>But the catch is</p><p>The system shaped by collective consciousness and energy dynamics does not provide a consistent or heightened sense of pleasure, respect, or importance for actions driven by emotional fulfillment, patriotism, or self-purpose. While incentives and emotional drive ultimately translate into brain rewards, the impact differs significantly. Brain rewards we get through emotionally driven activities are fleeting — once experienced, they remain constant.</p><p>In contrast, material possessions, social status, and attention offer accumulating brain rewards, as they can be continuously increased over time, leading to a growing sense of gratification. The downside of material possessions is that they are temporary and degrade over time unless we constantly maintain them. When they start to decline due to our mistakes, the loss of brain reward is difficult to absorb. The greater the pleasure they provide, the greater the pain we feel when they are lost.</p><h4>But be mindful</h4><p>The biggest downside of incentives versus emotional triggers is that certain individuals or groups can manipulate your emotions for their own selfish gain. I will present a few examples, specifically related to the Sri Lankan context and the current timeline, as it will be easier to find relevant facts for me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*AfzwlPRTn43_hTSaCjuV1Q.png" /></figure><blockquote>Mahinda Rajapaksa became an iconic figure in Sri Lankan politics by promoting the ideology of a Sinhala Buddhist nation. His words and charismatic personality strongly resonated with the majority of Sri Lankans, as he addressed their powerful emotional trigger point. During his rise, very few Sri Lankans questioned his policies, economic reforms, infrastructure development plans, technological advancements, or diplomatic strategies that could have provided long-term benefits to the citizens. Instead, people voted purely based on emotional triggers. Ultimately, the only ones who gained significant incentives were Mahinda Rajapaksa and his family, while the country and its citizens were led into bankruptcy — using the very votes of those emotionally driven citizens.</blockquote><blockquote>Hans Wijayasuriya, former CEO of Dialog Axiata, was appointed as Chief Advisor on Digital Economy to President AKD. That role, he accepted without any salary or benefits. Many Sri Lankans saw this as a patriotic act, believing he left his high-paying CEO position purely out of love for the country. However, as a seasoned CEO, his mindset is likely driven by strategy and future incentives rather than emotions. It’s possible he envisions shaping Sri Lanka’s digital economy to align with future opportunities — whether securing a role in a major tech company entering Sri Lanka, launching his own ventures, or even pursuing a political agenda. However, his aim of shaping Sri Lanka’s digital economy is promising and could ultimately benefit the people of Sri Lanka. It is important to understand, though, that his motivation does not stem from a deep love for the country, but rather from his personal agenda for greater incentives, higher status, and societal respect.</blockquote><blockquote>Anura Kumara Dissanayake, also known as AKD, is the president of Sri Lanka. One of his main promises was to remove incentives from Parliment Ministers and give them a lower salary, showing that the NPP wants to serve the country, not benefit from power. He has started doing this, with the idea that leaders should not live a more luxurious lifestyle than the people they serve. But do you really think high performers will work for less money? If you’ve worked hard to get where you are — by learning new skills, managing a company, or climbing the career ladder — would you give up the status and rewards you’ve built? It’s easy to say you would, but when you’re in a competitive environment, your mindset changes. You’re always looking to grow, never willing to settle. So, if someone is willing to work for less salary or fewer benefits, it could suggest they lack confidence in their abilities, aren’t competitive, or might have a hidden agenda. This could lead to poor performance or even dishonesty. A better approach would be to offer higher incentives based on performance, create fair competition, and strengthen the legal system.</blockquote><blockquote>Dhammika Perera, Sri Lanka’s richest businessman, started DP Education, one of the largest online platforms for teaching programming to low-income students. At first, it seems like a great initiative, offering free programming lessons to help students become software engineers and earn good salaries. This idea sounds good and emotionally appealing, but the reality is more complicated. To understand why, think about basic economics: when there is an oversupply of something, its value drops. DP Education may help students learn programming, but the market is already full of people with these skills, which means their salaries will likely stay low. Dhammika Perera, being a businessman, makes decisions based on data, not emotions. He sees there is high demand for software engineers in Sri Lanka, and the price for such workers is high. To lower the future cost of hiring software engineers, he’s investing in increasing the number of available workers. However, the problem is that software engineering isn’t like other jobs. It requires rare cognitive skills that can’t be easily established only by just teaching.</blockquote><p>As you can see, they’ve played on your emotions to cloud your judgment. This is one of the greatest negative impacts that can happen to you. Someone else may benefit greatly by using emotional triggers to manipulate you, but if you approach things with a rational mindset, you will see things differently and make better decisions.</p><h4>So what are the solutions?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CD4dS-eiF0Gl0iYVx-Va4A.jpeg" /></figure><p>So here comes the billion-dollar question — what is the most consistent solution to this? Should we align our actions based on incentives, or should we allow emotional triggers to guide us?</p><p>As we discussed earlier, all our actions are ultimately controlled by our brain. Throughout our lives, our brains have been trained and shaped by the reality we experience and the energy around us. The decisions we make are influenced by the energy and environment we live in, shaping how we respond to situations and opportunities.</p><p>The key factor is <strong>consistency</strong> if you want to improve your life step by step. As I explained earlier, <strong>incentives</strong> are the most important driver for the brain’s reward system. People work mainly for incentives, not for emotional triggers. The more frequently you receive meaningful incentives, the more motivated your brain becomes to create a better life.</p><p><strong>as examples</strong></p><blockquote>When you receive your monthly salary, you work consistently to earn that incentive. Over time, you naturally start working to increase that salary, and as a result, your skills and brain capacity also grow steadily.</blockquote><blockquote>If you are the owner or director, higher company profits mean higher incentives for you. This motivates your brain to work harder, find new ways to grow, and consistently improve your performance — leading to your personal and professional development.</blockquote><p>Now we know that incentives are the main motivator in life. However, you might still feel like something is missing — maybe a sense of selfishness for focusing only on your own incentives. This happens because humans are social creatures, and our brains can’t function well without social connections. We all want to feel accepted, and valued, and leave a legacy — and these are all emotional needs. To fill that emotional gap, you can use emotional triggers as a secondary source of motivation.</p><p><strong>as example</strong></p><blockquote>You can set aside some of your time for social activities, like CSR projects or community development work.</blockquote><blockquote>When voting, first analyze the candidate’s economic policies, infrastructure plans, and technology proposals. If these align with your interests, then you can also consider their patriotic promises. However, always compare and balance both sides — sometimes the value you lose from compromising your patriotic ideals is greater than the benefits you gain from policies alone.</blockquote><blockquote>You can work for higher incentives, but also be ready to go the extra mile when your country or organization needs you. For example, if you are a digital economy advisor, you deserve a high salary for your expertise — but you can also work extra hours, sacrifice personal time, and aim to build the best digital infrastructure in Asia, or even the world, for the good of your country.</blockquote><blockquote>As a business owner, you can aim for high profits each year, but you can also allocate part of those profits to employee well-being, create new jobs, or support charity work — not just for tax benefits, but because you genuinely care about your country and want to improve the quality of life in your society.</blockquote><p>Next time you’re about to make a decision or take action, prioritize the incentives you’ll receive. But don’t forget to consider emotional triggers as well — it’s like adding spice to your main dish, enhancing the experience without overpowering the core.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a2f25de3205" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Story of the Flax Bundle]]></title>
            <link>https://jayamaljayamaha.medium.com/the-story-of-the-flax-bundle-b8420979046d?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8420979046d</guid>
            <category><![CDATA[life-lessons]]></category>
            <category><![CDATA[personality-development]]></category>
            <category><![CDATA[manifestation]]></category>
            <category><![CDATA[belief-systems]]></category>
            <category><![CDATA[mindset]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Sun, 22 Dec 2024 06:46:11 GMT</pubDate>
            <atom:updated>2024-12-22T06:46:11.319Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IDvFJASdzSSCs03Y" /></figure><h4>Introduction</h4><p>The story of the flax bundle is a cherished part of Sri Lankan culture, passed down to us by our grandparents through generations. While I won’t resort to elaborate storytelling, this simple tale holds a profound motivational and educational message. By uncovering the hidden wisdom within this story, we can find valuable lessons that inspire us to grow and move our lives one step forward. Let’s explore its timeless guidance and discover how it can help us shape a better tomorrow.</p><h4>Story</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mgCRU8HAqsqDqI1emmhMqg.png" /></figure><p>In a small village, two brothers, both hardworking merchants, lived and worked with a shared vision. They purchased goods from local farmers and traveled from village to village, bringing valuable items to new communities. One day, they came across two bundles of flax, carefully harvested by the farmers.</p><p>Before they set out to sell the flax bundles, one brother decided to take an extra step. He carefully seasoned the flax, enhancing its strength and durability, knowing that by improving its quality, it would bring even greater value in the long run. The other brother, however, questioned the need for this extra effort. “Why bother with modifications?” he said. “We can just sell them as they are and still make a profit.</p><p>The thoughtful brother saw more potential in the seasoned flax bundles and decided, “Instead of selling them raw, I’ll make fabric.” He believed this extra effort would bring greater rewards. But his lazy brother laughed at him. “Why bother? We can just sell them as they are and still make a profit.”</p><p>The thoughtful brother thought, “Instead of just selling fabric, I can make different types of cloth in various colors.” Meanwhile, the other brother relaxed, dreaming about the profit he’d make and what he would do with it, without considering the extra effort his sibling was putting in.</p><p>Days passed, and the long-awaited market day arrived. Both brothers went to the market, eager to sell their goods. By the end of the day, the thoughtful brother had sold all his colorful, well-crafted clothes, earning a profit ten times greater than his initial investment in the flax bundle</p><p>Meanwhile, the lazy brother struggled to sell his flax bundles. Most of the flax had passed its recommended date, making it undesirable to buyers.</p><h4>What can we learn</h4><p>This tale imparts two valuable lessons — one about personal growth and the other about economic strategy.</p><p><strong>Personality lesson</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B0zZ7Cd5Mmiz_z0a" /></figure><blockquote>This tale highlights the danger of clinging to old, traditional ideas without evolving your knowledge or thinking patterns. The lazy brother stuck to the outdated practice of simply buying from farmers and selling to customers, adding no value to the flax bundles. He relied solely on tradition, which left him unprepared for change. In contrast, the creative brother embraced new ideas and applied his knowledge, transforming the flax into a more valuable product. His effort paid off with a significantly higher return. However, this success didn’t come easily — it required vision, hard work, and dedication.</blockquote><blockquote>To thrive in life, one must cultivate the courage to step out of their comfort zone and break free from the traditional ideologies that society has ingrained for years. Embracing new, progressive ideas and putting in the effort to learn and apply them to daily life is the key to growth and success. This journey demands resilience, as resistance from peers and society is inevitable. However, those who persevere and remain steadfast in their vision will reap the highest rewards in the long run, proving that effort and a forward-thinking mindset are the ultimate investments in a thriving future.</blockquote><p><strong>Economic lesson</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*je8hS-XZs9FZUlfO" /></figure><blockquote>This tale emphasizes the importance of adding value to raw materials through creativity and innovation. The lazy brother stuck in his traditional approach, tried to sell raw flax bundles, and failed even to recover his investment. Meanwhile, the creative brother transformed the flax into high-quality, colorful clothes, earning ten times the profit he initially invested. It’s a powerful lesson that value addition and forward thinking always yield the highest returns.</blockquote><blockquote>In today’s complex economic world, it is crucial to add unique value to raw products or services before presenting them to customers. The more innovative and valuable your contribution, the greater the return you can expect. However, it’s equally important to ensure that the value you add aligns with current market trends, meets customer demands, and leverages the latest technology to enhance efficiency. This approach not only helps you stand out in a competitive market but also enables you to achieve higher profits more smartly and sustainably.</blockquote><h4>Recap</h4><p>In summary, this tale teaches both a personality and an economic lesson. Unlearning traditional beliefs and embracing progressive ones is challenging but essential for success. In today’s fast-paced world, adapting to new mindsets benefits not just your personal life but every aspect of it.</p><p>Life reflects our thoughts and beliefs — stronger, progressive ideas lead to greater heights, while outdated ones risk failure. Success requires effort, but it’s a price worth paying for growth and fulfillment.</p><p>The best way to embrace new beliefs is by listening to others’ perspectives without immediate judgment. Engage with different cultures, religions, and communities. Watch personality, technology, and spiritual podcasts. Read books from diverse countries, explore human psychology, global politics, and the evolution of civilizations. These practices will broaden your understanding and help upgrade your traditional belief systems effectively.</p><p>Whether you’re in a profession or running a business, always be creative and add value to your service or product. Pursue further education, like a new degree, master’s, or PhD, to gain fresh knowledge and enrich your professional journey. Don’t get stuck in one career path — challenge yourself, learn new skills, and step into more advanced roles. Mastering social skills will also enhance your career and business. Embrace competition, and instead of staying in your comfort zone, outshine others by offering more value and continuously improving what you provide.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8420979046d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Five Scalable and Highly Available Deployment Architectures with AWS]]></title>
            <link>https://jayamaljayamaha.medium.com/five-scalable-and-highly-available-deployment-architectures-with-aws-dd821753c460?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd821753c460</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Wed, 07 Feb 2024 05:41:51 GMT</pubDate>
            <atom:updated>2024-02-07T05:41:51.103Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_9N_A-kLlQL3ihEV.png" /></figure><p>In the enterprise software domain, a majority of organizations leverage cloud service providers for deploying their software products or solutions. While different types of architectures cater to various business needs, the effectiveness of a software solution extends beyond its initial design. Without adhering to an efficient, scalable, and highly available deployment strategy, the value of a well-architected software solution diminishes. Such oversight not only constitutes a financial loss for the organization but also impacts revenue adversely.</p><p>In this article, I will expound upon five deployment strategies for your software application, emphasizing three foundational pillars that undeniably contribute significant business value to an organization. Those three factors are,</p><blockquote>Scalability<br>Ability to scale in or scale out the individual instances based on load hitting the application</blockquote><blockquote>High Availability<br>Ensuring the capability to remain resilient in the face of a catastrophe or failure is imperative. The objective is to sustain the functionality of the application, even in scenarios where instances may be inadvertently shut down.</blockquote><blockquote>Cost-effectiveness<br>Maximizing value return to the organization through the minimization of both infrastructure and deployment costs</blockquote><h4>Deployment 1 — Multi AZ EC2 instances combining with ASG and ALB as the load balancer.</h4><p>In this approach, we structure our resource stack as follows</p><ul><li>Create a target group with multiple EC2 instances spanning across more than one availability zone. It is worth noting that more than one target group can be created based on the specific needs of the application.</li><li>Develop an Auto Scaling Group within the target group, specifying the launch template and scaling metrics to ensure dynamic scaling based on demand.</li><li>Generate an Application Load Balancer (ALB) to act as a load balancer. The ALB distributes traffic to the EC2 instances in target groups according to defined rules. This ensures effective load distribution and enhances the overall reliability and performance of the system.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*lsQS2F4y-WEQT9u17Ixwjw.png" /></figure><h4>pros</h4><blockquote>This architecture ensures high availability as the EC2 instances are deployed across two availability zones. Thus, if one AZ becomes non-functional, the application can still maintain its functionality.</blockquote><blockquote>This architecture ensures scalability by deploying EC2 instances within an auto-scaling group. The launch configuration template and scaling policy should be set up to enable scaling based on the incoming load. In case of high demand, additional EC2 instances are automatically spun up to manage the load, ensuring uninterrupted application functionality.</blockquote><blockquote>ALB routes requests to EC2 instances based on the routing policy, effectively balancing requests across each EC2 instance in the target group.</blockquote><h4>cons</h4><blockquote>Manage your infrastructure with either on-demand or savings plans.</blockquote><blockquote>Write raw launch commands, such as installing relevant packages and libraries, starting the server if applicable, installing Docker if in use, pulling the image, and running the containers.</blockquote><blockquote>During the scaling process, it will spin up an entire new instance of the selected EC2 family. Unlike AWS ECS, individual tasks cannot be deployed.</blockquote><h4>Deployment 2—Tasks deployment on EC2 with docker and ECS combining ALB as the load balancer.</h4><p>In this approach, we structure our resource stack as follows</p><ul><li>Create an ECS cluster by specifying the EC2 launch type and allocating the EC2 instances for that specific cluster.</li><li>Establish a service for each microservice component within this cluster and specify the target group to be utilized.</li><li>Subsequently, formulate a task definition by delineating the memory and CPU units for the specific task. Include the Docker image path, environment variables, and credentials for Docker Hub, if chosen over ECR. Alternatively, ECR can be utilized.</li><li>Conclude the process by generating an Application Load Balancer with the necessary rules. Proceed to configure the ECS service and target groups for integration with the created load balancer.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*8geCJqKlTjnOoswg2annXw.png" /></figure><h4>pros</h4><blockquote>This architecture ensures high availability by deploying ECS tasks across two availability zones. If one zone becomes non-functional, our application can continue functioning seamlessly.</blockquote><blockquote>With ECS, we eliminate the need to worry about application deployment and configuring launch commands. We only need to specify the Docker image, memory, and CPU in the task definition, and ECS will handle the deployment on the appropriate EC2 instances. It’s important to note that in ECS, the smallest and functional unit we consider is a task, not an EC2 instance.</blockquote><blockquote>Auto-scaling is seamlessly managed by ECS. In the event of high load, ECS will scale by spawning new tasks proportionally to the load.</blockquote><blockquote>The load balancer efficiently handles and routes the load to the appropriate target groups, consisting of ECS tasks.</blockquote><blockquote>During the scaling process, instead of adding an entire new EC2 instance, ECS adds a new task to the existing EC2 instances based on resource availability using Docker.</blockquote><blockquote>This architecture is particularly well-suited for containerized applications. Moreover, you pay for the underlying resources you use without any additional cost for the ECS service.</blockquote><h4>cons</h4><blockquote>Given that this strategy utilizes EC2 launch configuration, the responsibility falls on us to manage our own EC2 instances, including tasks such as installing patches and updates.</blockquote><blockquote>It’s important to note that this approach is most suitable for containerized applications.</blockquote><blockquote>However, it may be susceptible to challenges such as underutilization or overprovisioning of EC2 instances for unpredictable workloads. These factors can have financial implications for the organization.</blockquote><h4>Deployment 3 — Tasks deployment with ECS fargate combining ALB as the load balancer.</h4><p>This strategy mirrors the previous one, with the key distinction being the adoption of a fully serverless approach. Here, we abstain from managing any resources independently, moving away from the use of self-managed EC2 instances.</p><ul><li>Create an ECS cluster by specifying the Fargate type.</li><li>Establish a service for each microservice component within this cluster and specify the target group to be utilized.</li><li>Subsequently, formulate a task definition by delineating the memory and CPU units for the specific task. Include the Docker image path, environment variables, and credentials for Docker Hub if chosen over ECR. Alternatively, ECR can be utilized.</li><li>Conclude the process by generating an Application Load Balancer with the necessary rules. Proceed to configure the ECS service and target groups for integration with the created load balancer.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*t-O32krFDz7hzSjKne-awA.png" /></figure><h4>pros</h4><blockquote>Similar to the previous architecture, this design ensures high availability by deploying ECS tasks across two availability zones, maintaining functionality in case of zone failure.</blockquote><blockquote>Leveraging ECS eliminates concerns about application deployment and launch command configuration. Specify Docker image, memory, and CPU in the task definition, allowing ECS to deploy tasks on Fargate instances, with the task as the smallest functional unit.</blockquote><blockquote>Auto-scaling is managed by ECS, dynamically adding tasks in response to high loads.</blockquote><blockquote>The load balancer efficiently directs traffic to target groups of ECS tasks.</blockquote><blockquote>During scaling, a new task is added to the target group, avoiding the need for an entire new EC2 instance.</blockquote><blockquote>This architecture is well-suited for containerized applications, with payment based on underlying resources, excluding extra ECS service costs.</blockquote><blockquote>Unlike the previous setup, AWS manages EC2 instances in Fargate. Our primary responsible element is a task, making it effective for applications with unpredictable loads and mitigating concerns about underutilization or overprovisioning.</blockquote><h4>cons</h4><blockquote>Not a good option to use if your application has a predictable load.</blockquote><blockquote>best suited for containerized applications.</blockquote><h4>Deployment 4 — AWS lambda functions integration with API gateway.</h4><p>For a fully serverless approach with minimal cloud operations, ideal for microservices with execution times under 15 minutes, use this strategy.</p><p>No need to build a REST API from scratch, simply implement the required functionality in concise code. The API gateway manages all endpoint routing.</p><ul><li>Create an API gateway resource.</li><li>Establish lambda resources for each intended endpoint and deploy the code.</li><li>Configure the API gateway endpoint by linking each lambda function to its corresponding endpoint.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*QEBE5b1V_QA9NFNo0ZAVBQ.png" /></figure><h4>Pros</h4><blockquote>Create only the necessary functionality. For Node.js, it involves a function without a server; for Java, a function without a main class or method, and similarly for other languages.</blockquote><blockquote>This results in a lightweight and small codebase, leading to faster development and easy management.</blockquote><blockquote>Adopt a fully serverless approach, eliminating the need to configure servers, load balancers, target groups, or auto-scaling. AWS handles these aspects; you deploy, and AWS manages the rest.</blockquote><blockquote>Utilizing small executable functions reduces code complexity and development effort, resulting in cost savings.</blockquote><blockquote>Due to scalability based on load, this strategy is well-suited for applications with unpredictable loads.</blockquote><h4>Cons</h4><blockquote>Ideal for small to moderately complex projects, but not suitable for REST APIs with over 20 endpoints.</blockquote><blockquote>Each task must be completed within 15 minutes, otherwise the lambda function will be terminated automatically.</blockquote><h4>Deployment 5— AWS EKS integration combined with ALB.</h4><p>This Kubernetes deployment on AWS caters to organizations that prioritize Kubernetes over other container-managed services. Some entities adhere to internal policies, while for others, Kubernetes is deeply ingrained in their technical architecture. In such contexts, this strategy proves highly suitable.</p><ul><li>To initiate the process, a Kubernetes cluster needs to be established and deployed within the AWS environment.</li><li>Worker nodes can be deployed using three strategies — <br> self-managed nodes, managed node groups, and Fargate.</li><li>Worker node target groups should be configured to span across multiple Availability Zones (AZs).</li><li>An Application Load Balancer (ALB) can be configured by integrating an ingress controller with ALB.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*Zm7x5t-4yeIsE4X8wE1-QQ.png" /></figure><h4>Pros</h4><blockquote>You can leverage built-in Kubernetes functionalities if you’re already familiar with the Kubernetes ecosystem.</blockquote><blockquote>Additionally, you can integrate other AWS services with Kubernetes to enhance business value, such as CloudWatch, IAM, KMS, and S3.</blockquote><blockquote>The control plane of the Kubernetes cluster will be managed by AWS, leaving you responsible only for node and node group management.</blockquote><blockquote>AWS will deploy the most efficient EC2 instances as pods, aligning with your memory and CPU configurations. This eliminates concerns about overprovisioning or underutilization.</blockquote><blockquote>In managed node groups, auto-scaling will be handled by AWS EKS; we simply need to configure the maximum and minimum instances.</blockquote><blockquote>Due to scalability based on load, this strategy is well-suited for applications with unpredictable loads.</blockquote><h4>Cons</h4><blockquote>The learning curve is somewhat steep as it requires knowledge of both Kubernetes and AWS infrastructure.</blockquote><blockquote>Costs tend to be higher compared to ECS. Since the control plane is managed by AWS, there are expenses associated with that in addition to paying for the underlying resources.</blockquote><h3>Ending</h3><p>In this article, I have presented five well-recognized and organized deployment architectures that can significantly enhance the business needs of organizations. However, before implementing any of the above deployments in the real world, it’s crucial to consider the following trade-offs. Neglecting these trade-offs could result in the implemented architecture being detrimental to the organization.</p><ul><li>Always be mindful of your organization’s needs and implement the architecture in a way that minimizes costs while achieving maximum business requirements.</li><li>Consider scenarios that may lead to overprovisioning and underutilization, and mitigate them as much as possible.</li><li>Review your architecture with peers, engineering leads, and architects to gain valuable feedback.</li><li>Identify and address bottlenecks that could impact resource allocation and optimize accordingly.</li><li>Utilize monitoring tools such as Datadog to track logs, CPU, memory metrics, and costs.</li><li>Plan for future requirements rather than focusing solely on short-term goals, and configure the architecture accordingly.</li><li>Document the architecture, any modifications made over time, and critical points to ensure clarity and facilitate future maintenance.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd821753c460" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quarkus - A Supersonic, Subatomic Java]]></title>
            <link>https://medium.com/sysco-labs/quarkus-a-supersonic-subatomic-java-e7c3ba510d79?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7c3ba510d79</guid>
            <category><![CDATA[graalvm]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[quarkus]]></category>
            <category><![CDATA[quarkus-framework]]></category>
            <category><![CDATA[jvm]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Thu, 19 Oct 2023 11:09:39 GMT</pubDate>
            <atom:updated>2023-10-19T11:09:39.001Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P95zlWYuEM9SYp3L.png" /></figure><p>I am an avid Java developer and have been working with Springboot for over three years. It has been my go-to framework for most of my projects.</p><p>However, recently a colleague introduced me to the Quarkus framework.</p><p>Initially, I was skeptical and asked why I should switch to Quarkus when I already have a great framework like Springboot. But, since Quarkus is relatively new, I decided to learn it more thoroughly.</p><p>Before we dive into the Quarkus, we need to learn some important things to understand the ground level that holds the base for Quarkus.</p><h4>What is <strong>GraalVM</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/0*7lcBQ3h8NuJmm4mK" /></figure><p>GraalVM is a Polyglot runtime environment that can be used with multiple programming languages. GraalVM uses high performance Just in Time(JIT) and Ahead of Time(AOT) compilation techniques to build the applications as high-performance and low startup time native executables.</p><p>Before going further let’s clarify some important concepts.</p><blockquote><strong>Polyglot runtime environment<br></strong>We can use GraalVM to compile and build not only for Java but for other programming languages as well. GraalVM contains language specific runtime components and libraries so we can use this same run time environment to run the application which were written in different languages like C,C++,Python,Ruby, JavaScript etc.</blockquote><blockquote><strong>Just in Time compilation (JIT)<br></strong>This is a compiler that will compile byte code to machine code and this will happen during application runtime. This compiler will read the byte code and run it through a interpreter and also will optimize the code with some advanced optimization concepts. Also this JIT compilation will be used by the languages with dynamic class loading and languages which uses reflection like Java.</blockquote><blockquote><strong>Ahead of Time compilation (AOT)</strong><br>As the name suggest this compiler will compile the code to machine code without compiling it to byte code and it will create a executable out of the code. After that this executable can be run on a specific system without a need of any kind of virtual machine like JVM. The compiled executable will be more efficient, less memory usage, has faster startup time and use as a container friendly native executable than the application that uses JIT compilation. But in this method we need to do some additional configurations if our code need to use dynamic class loading and reflection features.</blockquote><p>GraalVM uses both techniques when compiling a project. As we discuss Java mostly in this article, we can create a jar by compiling Java code with JIT compiler and can be run on the GraalVM, or we can create a native executable by compiling the code with AOT compiler and can be used in containerized environments. However, the JIT compiler of this GraalVM will compile the Java code to byte code in a more efficient way than the JIT compiler in JVM.</p><h4>What is Quarkus</h4><p>Quarkus is a Kubernetes-Native Java framework, which uses GraalVM and HotSpot techniques to develop. Since this is using GraalVM, all the Optimization features and advanced features of GraalVM are available in Quarkus as well.</p><h4>How has Quarkus become so remarkable?</h4><blockquote><strong>Low startup time and low memory footprint</strong><br>Since Quarkus uses GraalVM by default instead of JVM, the optimized compilation capabilities that GraalVM has will be applicable to this Quarkus as well. Quarkus will implement this low startup time and low memory footprint by leveraging AOT compilation of GraalVM. Also when we run the Quarkus application as jar file, it will have a low startup time and low memory foot print as well. This will happen because GraalVM do some advance optimization techniques when compiling the Java code to bytecode rather than JVM.</blockquote><blockquote><strong>Low package size</strong><br>Since Quarkus can leverage the native executable generation capability of GraalVM, We can use AOT compilation and build a native executable, which is more lightweight by size than a jar file in Springboot.</blockquote><blockquote><strong>Container and orchestration friendly<br></strong>Quarkus is Kubernetes-Native framework, so at the inside of the framework, it has a great support to Kubernetes and orchestration. Secondly, since GraalVM can create native executables, we can easily containerize it and run the application as containers. Also since the native executable is small in size, we can easily create small sized images and store it in a Docker registry.</blockquote><blockquote><strong>Low memory and CPU usage<br></strong>Since we can leverage AOT compilation of GraalVM, we can create native executables. Afterwards our application can run on a system by itself without GraalVM. So The total memory and CPU that will be used for the application will be very less since there are no JIT compilation processes need be executed when we use native executables.</blockquote><p>While we may delve into the advantages of AOT compilation, it’s essential to recognize the existence of drawbacks. Just like with anything else, there are both pros and cons to consider, and striking a balance is crucial.</p><h4>Drawbacks of AOT compilations</h4><p>Before discussing the drawbacks of AOT compilations, first, we need to look at why we need a concept called JIT compilation for Java applications. The main reason is the dynamic behavior of Java and JVM</p><p><strong>Pholimopshism and Late Binding</strong><br>To understand this concept let&#39;s see some frequently used concepts of Java below;</p><pre>interface Shape {<br>    void draw();<br>}<br><br>class Circle implements Shape {<br>    public void draw() {<br>        // Circle drawing logic<br>    }<br>}<br><br>class Rectangle implements Shape {<br>    public void draw() {<br>        // Rectangle drawing logic<br>    }<br>}<br><br>public class Main {<br>    public static void main(String[] args) {<br>        Shape shape = new Circle();<br>        shape.draw();<br>    }<br>}</pre><p>Using interfaces and implementing them by concrete classes is a fundamental concept of Pholimopshism in Java, but here, the assigning of Circle object to Shape type will happen in runtime, so our JVM doesn&#39;t know what “draw()” method needs to be run in advance until it reaches runtime.</p><p>At the runtime, the JIT compiler will decide which “draw()” method (either draw method of interface or draw method of circle class) to be executed, it will then generate the machine code to that specific “draw” method and then run it through the interpreter. Therefore, the JIT compiler will decide the most optimized and specific way to dynamically execute the above code block at runtime. This process is called Late Binding</p><p>Achieving this can be somewhat challenging in AOT compilation since it’s difficult to pre-determine and bind the exact drawing method to be executed at runtime in this application.</p><p><strong>Use of Reflection and Dynamic Class Loading<br></strong>JIT compiler frequently uses Reflection and Dynamic Class loading when it compiles byte code to machine code. First, the JIT compiler needs to decide what kind of classes needs to be loaded at a specific point of execution, in order to execute a particular piece of code. Then the JIT compiler needs to decide which methods should be executed at that particular point of execution. So, to obtain those details about methods and variables, the JIT compiler uses Reflection.</p><p>Loading only necessary classes without generating machine code for all the classes in advance, will mean more efficient and optimized execution of the program. This is because we load only the required machine code of classes to memory at a specific point of execution.</p><p>On the other hand, JVM needs to decide which classes and their methods need to be loaded, based on dynamic configurations and use inputs at runtime. To do that JIT compiler need a concept called Reflection. JIT uses Reflection to load and read metadata of classes, load methods and execute them in runtime, and extract and modify variables based on dynamic configurations at runtime.</p><p>Sometimes above mechanisms won&#39;t work as efficiently as in the JIT compiler when we use the AOT compiler. JIT compiler will do advanced optimization steps depending upon the situation of execution, but in AOT compilation, the compiled code may not be optimized very well in the above kind of scenarios. Once more, there are methods to configure this Reflection aspect in AOT. In Quarkus, we can employ specific annotations for this purpose. Through this configuration approach, we can also alleviate the drawbacks above.</p><h4>Limitations of Quarkus</h4><p>While Quarkus does offer noteworthy solutions, it’s important to note that it comes with certain limitations. Being a relatively new framework introduced around 2019, and as of the time of writing this article, it’s prudent to be aware of these limitations when making technology stack decisions for a project in order to maximize the benefits for an organization.</p><blockquote><strong>Learning curve and limited eco system</strong><br>There is a considerable learning curve for the Quarkus, because this is a new framework and it has a limited community when compared to Springboot. You can find tons of reference materials and videos for Springboot but for Quarkus it is quite low. You may need to take time to resolve if you come across issues or misunderstandings.</blockquote><blockquote><strong>Build time</strong><br>Since Quarkus uses some advanced optimization techniques in build time, and uses AOT compilation to generate the native image, the build time of this Quarkus application will be high. Also building processes may require some high resource utilization as well (RAM and CPU). If it can’t allocate the required resources, the build process may fail. The build process is a kind of resource and time intensive process in Quarkus.</blockquote><blockquote><strong>May contain bugs</strong><br>Since this is relatively a new framework, it has not covered all the edge cases when we use this in production. So this may contain some bugs. Also this is a open source project, the community will take some time to report and fix those bugs. The lack of production experience in any kind of framework should be highly considered when you consider to use this framework to build enterprise grade software in your organization.</blockquote><h4>Conclusion</h4><p>As we can now learn, Quarkus is a very good framework for building enterprise-grade applications with great performance by using Reactive and NonReactive styles simultaneously. But, since I am a Springboot lover, we have the option to use GraalVM with AOT compilation and achieve the same level of performance and startup time benchmarks as Quarkus in the Springboot framework as well. The motivation for this article is to introduce and draw your attention to this new framework and to its very important base concepts. I will write about the new GraalVM and AOT features in Springboot 3 in another article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7c3ba510d79" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sysco-labs/quarkus-a-supersonic-subatomic-java-e7c3ba510d79">Quarkus - A Supersonic, Subatomic Java</a> was originally published in <a href="https://medium.com/sysco-labs">Sysco LABS Sri Lanka</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparison of Efficiency and Resilience to High Load of Java and NodeJs REST API Architectures.]]></title>
            <link>https://medium.com/sysco-labs/comparison-of-efficiency-and-resilience-to-high-load-of-java-and-nodejs-rest-api-architectures-f63acd0d3295?source=rss-1ab1b353e11d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f63acd0d3295</guid>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[reactive-programming]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Jayamal Jayamaha]]></dc:creator>
            <pubDate>Mon, 24 Apr 2023 06:08:20 GMT</pubDate>
            <atom:updated>2023-04-24T06:08:20.964Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*juxSTN3YNY1mNTIdymYIWA.png" /></figure><p>In this article, I am going to do an experiment. In this experiment, I am going to test and compare four REST API architectural styles and give insight into how the efficiency and resilience to the high load of each architecture. I am going to do this experiment using the scientific method.</p><h4>The Observation and the Problem that we discovered</h4><p>Backend REST APIs are some main software that will be used by many business organizations to achieve their business goals. At the end of the day, almost every business organization&#39;s goal is to increase their revenue by giving a well-optimized value to their customers with high customer satisfaction which leads to retaining their customer base for a long period of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*5UI5BlW7aEG5EWCca3z4_w.png" /></figure><p>In the above scenario, technology plays a major role in order to achieve those business goals. So a REST API is a good technological tool to reach to customer base easily and manage other business activities such as employee details, revenue and growth statistics, inventory management tasks and etc.</p><p>But in order to use the technology in the best and most profitable way, we need to build software in a way that a very efficient, scalable, and resilient. Otherwise, the ROI of that software for a business organization will become a waste and sometime it will harm revenue as well.</p><p>So in the context of REST APIs, there are many technologies and many architectural styles. As an architect or developers, we need to choose the right technology with the right architectural pattern in order to give maximum value to a business organization. So in this experiment, I am going to test efficiency and resilience to a high load of four architectural patterns for REST APIs.</p><h4>Background Research</h4><p><strong>Architecture patterns</strong></p><p>There are many architectural patterns that we use to develop REST APIs, but in this article, I will test only four patterns</p><p>Three from Java</p><ul><li><strong>Standard Sprinboot REST API application</strong><br>The standard architecture that normally a REST API will be going to build using Springboot</li><li><strong>Sprinboot REST API application using CompletableFutures </strong><br>CompletableFuture in java is a way to achieve concurrency. In this architecture, I will test a Sprinboot REST API that will execute more than one data object in a concurrent and async way using CompletableFutures</li><li><strong>Reactive Spring WebFlux application.</strong><br>In this architecture, I will test a REST API that was built in a reactive way using Spring WebFlux. This architecture will work as an event-driven model using Flux and Mono.</li></ul><p>One from NodeJs</p><ul><li><strong>Standard NodeJs REST API application using Express<br></strong>Standard NodeJs REST API using Express framework Which will work as a Non-Blocking and Async way (Pretty much the same as Reactive java)</li></ul><p><strong>Load Testing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*oGO3UdTGojmkjpj8.png" /></figure><p>In order to run a load test, JMeter will be used in this experiment. JMeter is an open-source tool to do a load test for any API. In this experiment, I will use JMeter DSL which is a version of JMeter that we can use as a coding approach rather than run JMeter as a software application. The reason to use JMeter DSL rather than JMeter itself is that we can create the HTTP request POST body programmatically rather than define a concrete body. So we can create a request body in the way we need it programmatically.</p><p>This is how I create this request body</p><pre>{<br>  &quot;images&quot;: [<br>    {<br>      &quot;name&quot;: &quot;cute dog&quot;,<br>      &quot;width&quot;: 120,<br>      &quot;height&quot;: 120,<br>      &quot;format&quot;: &quot;jpg&quot;,<br>      &quot;url&quot;: &quot;https://images.examples.com/cute_dog_123&quot;,<br>      &quot;size&quot;: 1320000,<br>      &quot;device&quot;: &quot;DSLR&quot;,<br>      &quot;number_of_pixels&quot;: 14400,<br>      &quot;created_date&quot;: &quot;2017-07-21T17:32:28Z&quot;,<br>      &quot;last_modified_date&quot;: &quot;2018-07-21T17:32:28Z&quot;,<br>      &quot;captured_by&quot;: &quot;sandun&quot;<br>    }<br>  ]<br>}</pre><p>With the following java code</p><pre>public String getRequestBody(PreProcessorVars preProcessorVars) {<br>     JSONArray imagesArray = new JSONArray(IntStream.range(0, 10).mapToObj(value -&gt; new JSONObject(getImageDetailsAsMap(value))).toArray());<br>     Map&lt;String, JSONArray&gt; imageDetailsMap = new HashMap&lt;&gt;();<br>     imageDetailsMap.put(&quot;images&quot;, imagesArray);<br>     return new JSONObject(imageDetailsMap).toString();<br> }<br><br> public Map&lt;String, String&gt; getImageDetailsAsMap(int index) {<br>     long next = random.nextInt();<br>     Map&lt;String, String&gt; imageDetailsMap = new HashMap&lt;&gt;();<br>     imageDetailsMap.put(&quot;name&quot;, &quot;cute dog - &quot; + Math.abs(next));<br>     imageDetailsMap.put(&quot;width&quot;, &quot;120&quot;);<br>     imageDetailsMap.put(&quot;height&quot;, &quot;120&quot;);<br>     imageDetailsMap.put(&quot;format&quot;, index % 3 == 0 ? &quot;jpg&quot; : index % 3 == 1 ? &quot;png&quot; : &quot;jpeg&quot;);<br>     imageDetailsMap.put(&quot;url&quot;, &quot;https://images.examples.com/cute_dog_&quot; + Math.abs(next));<br>     imageDetailsMap.put(&quot;size&quot;, &quot;1320000&quot;);<br>     imageDetailsMap.put(&quot;device&quot;, index % 5 == 0 ? &quot;DSLR&quot; : index % 5 == 1 ? &quot;PHONE&quot; : index % 5 == 2 ? &quot;DIGITAL_CAMERA&quot; : index % 5 == 3 ? &quot;GO_PRO&quot; : &quot;PIXEL&quot;);<br>     imageDetailsMap.put(&quot;number_of_pixels&quot;, &quot;14400&quot;);<br>     imageDetailsMap.put(&quot;created_date&quot;, &quot;2017-07-21T17:32:28Z&quot;);<br>     imageDetailsMap.put(&quot;last_modified_date&quot;, &quot;2018-07-21T17:32:28Z&quot;);<br>     imageDetailsMap.put(&quot;captured_by&quot;, &quot;sandun - &quot; + Math.abs(next));<br>     return imageDetailsMap;<br> }</pre><h4>Defining Boundaries</h4><p>We cannot do an experiment by covering all the logical scenarios at the same time practically, so we define the boundaries for an experiment. The same thing applies to this experiment as well. Rather than cover the entire REST API paradigm I will test this based on the below boundaries.</p><ul><li>I will consider a POST endpoint only because it will write data to the database. in order to write data, some validations and exception handling have to be done. So in the POST endpoint, there is some work to process in addition to IO operations.</li><li>I will run this on my machine and I will not use any cloud service. So the latency of an HTTP request will be less than some production deployment on a cloud service like AWS. But this experiment is about comparison, so you can get an idea about efficiency by comparing rather than focusing on the absolute value.</li><li>The application will be containerized and will be run using docker. I will use the Linux environment as the host operating system.</li><li>I will run this program on one instance and will not use any load balancing. That means you can gain a high load efficiency than this in a prod environment using many instances with load balancing. But again this is a comparison, so we compare all this using one instance. So you can gain an idea rather than focusing on absolute values.</li><li>I won&#39;t use any Authorization and Authentication to the endpoint. In real-world applications, this concept is a must and it will affect the performance of overall HTTP request latency. again don’t focus on absolute values because I will be doing a comparison here.</li><li>I will only use database operation as an IO operation. but this experiment can be done with any IO operation and any amount of them. For example, 3rd party API calls. Kafka or any streaming operation, OAuth authentication with social media, etc.</li><li>There are no high computation-heavy processes in those four APIs. These APIs are typical business application APIs like <br><strong>getting data -&gt; validate -&gt; save in DB<br>getting data -&gt; validate -&gt; send to Kafka stream<br>getting data -&gt; validate -&gt; call 3rd party API -&gt; save in DB</strong></li></ul><h4>The Hypothesis</h4><p>While normal java applications will work in a blocking way, NodeJs will work in a nonblocking way to IO operations and also in an async way</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*CzWmOElO2kcYlGZw0crCxw.png" /><figcaption>Typical Springboot Application</figcaption></figure><p>But Java async approach will use many threads to achieve concurrent behavior in addition to tomcat container threads than the normal approach. So the async approach should work more efficiently than a normal springboot application</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*_ZuDwYROEYrBGciB0zU5Cg.png" /><figcaption>Java CompletableFuture Async Application</figcaption></figure><p>But java is still blocking IO operations, while NodeJs is not. So NodeJs should work in an efficient way with a high load</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*H7Fik9mnHZQCp7f4iojJ9Q.png" /><figcaption>NodeJs Application</figcaption></figure><p>But when we come to the reactive approach in java, we can achieve a non-blocking way with a reactive approach. In NodeJs it will use an event loop with one main thread to treat all HTTP requests. Even NodeJs work in a non-blocking way, it will work in a single main thread with a single event loop for that thread. But in Spring WebFlux, it uses a reactor-netty server. In reactor netty, there is an event loop mechanism to process events plus we can have multiple event loops (Not more than a number of cors in CPU). So each event loop can accept requests and treat them in a parallel way. So the Java Reactive approach should work in an efficient way than NodeJs with high loads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*DWSfyksJL5VCtVmSAxSxuw.png" /><figcaption>Spring WebFlux application</figcaption></figure><p>Remember in java, physical threads are limited and thread creation is expensive. So if all the threads are allocated, then new HTTP requests have to wait until one thread will become available. On the other hand, we use the same physical thread pool for both tomcat threads and the threads that we create to process one data item from the payload in the async approach.</p><p>Remember in NodeJs, before going to call DB, the main thread will be busy validating and processing the payload in POST request so that new HTTP requests have to wait until the main thread will become available. The same happens after the IO operation.</p><p>Finally, the efficiency and resilience to high load should vary like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*Giq6PZQe8Ej_oNiyX-rwAA.png" /></figure><p>But in order to be sure we need to do the experiment.</p><h4>Testing</h4><p>Tech stack used in the experiment</p><ul><li>Springboot</li><li>Spring WebFlux</li><li>Gradle</li><li>ExpressJs</li><li>PostgreSQL</li><li>JMeter DSL</li></ul><p><strong>I will test the APIs in the following order with 6 cores CPU and 6GB available memory</strong></p><ul><li>Test each API with 10 concurrent users</li><li>Test each API with 100 concurrent users</li><li>Test each API with 1000 concurrent users</li><li>Test each API with 10000 concurrent users</li></ul><h4>Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*KFCOZTTRJmYqLc3jIvA-7g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*SmEkZ9QlVgsEs4F5AL0xsw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*VdiGnRqnSkl5kmWU63YTKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*YSirtdSk2BixRWb1NDwnVw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*oVeU4EYQV0Do8N4c4skPTA.png" /></figure><p>As you can observe</p><ul><li>Java Reactive approach will be the winner with the capability of handling 10000 concurrent users without any failure and with less average load time.</li><li>For lower concurrent users java async approach will be the most efficient and the java regular approach will be the next. The reactive approach will perform its best with more concurrent users than with fewer concurrent users.</li><li>But NodeJs is the less efficient approach among the four approaches. the average load time is less for all concurrent user counts than other approaches but it can handle all 10000 concurrent users without errors</li><li>So as these results, java reactive is the most prominent approach and we can say Nodejs is the next reliable approach because hence nodeJs get more time to process requests, and it can handle all the concurrent users without any errors.</li></ul><p>As we can see, the above data is insufficient to come to a final and more accurate conclusion. To be a more fair experiment we need to consider more different situations. Also In a business organization, the aim is not only to achieve high availability and speed but to achieve those aims with less cost. So let’s do this experiment with a different set of resource utilization.</p><p>We can allocate a number of CPU cores that will be allocated to a container to use with the docker run command. So the container will use the defined number of cors when we use the API.</p><p><strong>I will test the APIs again in the following order with 1 core CPU and 4 cores CPU using the docker containers.</strong></p><ul><li>Test each API with 10 concurrent users</li><li>Test each API with 100 concurrent users</li><li>Test each API with 1000 concurrent users</li><li>Test each API with 10000 concurrent users</li></ul><h4>Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tf5MF8i7J0o5A1TERS3M0A.png" /></figure><ul><li>As you can see in this diagram we cannot notice much difference between the values of six, four, and one cores.</li><li>This will happen because NodeJs will use only one event loop, so only one core is sufficient to run the event loop. but to run other worker threads full NodeJs app may need additional cores, which is why we notice some error percentage for one core.</li><li>However, it is a waste of resources if we use six cores because we won’t achieve any performance gain after 4 cores.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g5YdrW0S2pRcUc9cj4rD_Q.png" /></figure><ul><li>We can see an interesting observation here. The reactive application can handle 10000 concurrent even one core without any failures and also with the highest efficiency</li><li>This reactive app can utilize the resources in the best efficient way. So this will lead to achieving high availability and speed at a low cost.</li><li>Increasing cores can achieve better performance as this reactive app can create more than one event loops based on available cores. So resources can be utilized as the business needs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2v-Y1FS2BoMvTFeFyfEIWw.png" /></figure><ul><li>In this regular app, as you can see error percentage will increase when we decrease the resources. even this cannot handle 1000 concurrent users without failures with less than four cores</li><li>Also, the average load time also will increase when we decrease the resources</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L6fv7WXoRyzWRK-hlDDReg.png" /></figure><ul><li>In this aync app, as you can see error percentage will increase when we decrease the resources. but better than the regular app</li><li>Also, the average load time also will increase when we decrease the resources</li></ul><h4>Final Conclusion</h4><p>After observing all the above results and thinking logically, we can come up with the below conclusions.</p><ul><li>Java reactive is a very prominent architecture to use because it can handle more concurrent users with fewer resources. So it will not only work with high load and high availability, but it will reduce the cost of the business as well.</li><li>If an organization has a team with java developers and wants to serve their customers with a high availability system, java reactive architecture is a good solution.</li><li>If an organization has fewer customers to serve and has a team with java developers, then the java async and regular approach is the most suitable approach. because the cost will be high to hire a development team that has experience in a different technology like NodeJs.</li><li>Also to use the java async approach with completable futures, require some experienced developers in java. So in some situations, java regular way is the most cost-effective approach. That decision should be taken by considering lots of factors, like the experience of the development team, the number of customers that the business has, cost constraints that have to handle the software system and cloud and etc.</li><li>NodeJs is the next very prominent architecture after the java reactive approach because it can handle more concurrent users with fewer resources than java regular and async way.</li><li>If an organization has a development team that has developers with experience in NodeJs and has a large customer base, then this NodeJs is the suitable architecture to use.</li><li>If an organization has a moderate size customer base and developers with javascript backgrounds with some cost constraints, then this NodeJs is the best to use.</li></ul><p>So in this experiment according to our observations and conclusions, our hypothesis will become true as like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*Giq6PZQe8Ej_oNiyX-rwAA.png" /></figure><p>But you should keep in mind any architecture among the above four architectures is a usable and most suitable architecture depending on the business requirements and technology infrastructure of the organization. So the most suitable one should be decided by considering various factors.</p><h4>Code Repositories</h4><p>Here I am attaching all the code repositories that I used for this experiment for your reference.</p><p><strong>JMeter App I used for testing</strong><br><a href="https://github.com/jayamaljayamaha/experiment-jmeter-dsl-app">https://github.com/jayamaljayamaha/experiment-jmeter-dsl-app</a></p><p><strong>Java Reactive REST API I used</strong><br><a href="https://github.com/jayamaljayamaha/experiment-reactive-app">https://github.com/jayamaljayamaha/experiment-reactive-app</a></p><p><strong>NodeJS REST API I used</strong><br><a href="https://github.com/jayamaljayamaha/experiment-express-app">https://github.com/jayamaljayamaha/experiment-express-app</a></p><p><strong>Java Async and Regular REST APIs I used</strong><br><a href="https://github.com/jayamaljayamaha/experiment-regular-app">https://github.com/jayamaljayamaha/experiment-regular-app</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f63acd0d3295" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sysco-labs/comparison-of-efficiency-and-resilience-to-high-load-of-java-and-nodejs-rest-api-architectures-f63acd0d3295">Comparison of Efficiency and Resilience to High Load of Java and NodeJs REST API Architectures.</a> was originally published in <a href="https://medium.com/sysco-labs">Sysco LABS Sri Lanka</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>