<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Smart Cloud on Medium]]></title>
        <description><![CDATA[Stories by Smart Cloud on Medium]]></description>
        <link>https://medium.com/@smartcloud.blog?source=rss-6890d2d32be1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cVdv6ExOimhxY-hntLqCIQ.jpeg</url>
            <title>Stories by Smart Cloud on Medium</title>
            <link>https://medium.com/@smartcloud.blog?source=rss-6890d2d32be1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:48:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@smartcloud.blog/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part 2: Measuring Accuracy and Empathy: Evaluating a Railway Assistant with RAGAS]]></title>
            <link>https://medium.com/@smartcloud.blog/part-2-measuring-accuracy-and-empathy-evaluating-a-railway-assistant-with-ragas-3c50a318b6da?source=rss-6890d2d32be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c50a318b6da</guid>
            <category><![CDATA[ai-for-business]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[gen-ai-for-business]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <dc:creator><![CDATA[Smart Cloud]]></dc:creator>
            <pubDate>Sat, 26 Apr 2025 13:08:43 GMT</pubDate>
            <atom:updated>2025-04-26T13:08:43.591Z</atom:updated>
            <content:encoded><![CDATA[<p>Previously, in <a href="https://medium.com/@smartcloud.blog/part1-empathetic-ai-bot-for-railway-help-rag-approach-with-langchain-bac0a3f69994">Part 1</a>, we focused on setting up the empathetic AI assistant — selecting tools, recognizing the limits of prompt engineering, and choosing a RAG-based approach. Now, it’s time to evaluate how the bot performs in real scenarios.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sw7G70XT9mLt5-6XqpLhIg.png" /></figure><p>The goal of my blog <a href="https://smart-cloud.blog/">https://smart-cloud.blog/</a> is to help integrators of generative AI understand the core concepts and challenges.<br> My blog is intended for backend developers who are learning how to integrate with generative AI models.</p><h3><strong>Applying OpenAI’s Approach to My Empathetic Railway Bot</strong></h3><p>As mentioned earlier, I chose to apply insights from OpenAI’s fine-tuning lecture to see how parts of their strategy could be adapted to my project. Interestingly, one of the biggest challenges was defining the exact task at hand.</p><p>Remembering lessons from the <a href="https://medium.com/@smartcloud.blog/aws-ai-practitioner-certification-my-journey-prep-tips-607873afda43">AWS AI Practitioner </a>certification, I realized that in typical ML/AI pipelines, stakeholders set the goals. So, I took on the role of a railway business owner and set two clear objectives for my bot:</p><ol><li><strong>Answer user questions about train schedules</strong> (integrated into the RAG system).</li><li><strong>Respond empathetically</strong> to frustrated customers when no specific information is needed (e.g., complaints like “It’s too hot on the train” or “My train left without me”).</li></ol><p>Let’s now walk through OpenAI’s strategy and see how I applied each step:</p><h4><strong>🔧 Beginning with Prompt Engineering</strong></h4><p>The first step recommends refining prompts to establish a solid foundation.</p><p>➡️ For my bot, I designed a prompt that directs it to respond empathetically when there’s no specific query, and to refer to the schedule data when users inquire about trains. Here’s my prompt:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4p2YUDW4qOX6PIOGHOV2aQ.png" /></figure><p><strong>{context}</strong> serves as a placeholder for reference information, which will be populated with three documents retrieved from a Pinecone database using embedding similarity.</p><h4><strong>🧠 Augment Information</strong></h4><p>If refining prompts alone doesn’t produce the desired results, the next step is to enrich the information.</p><p>➡️ This involves manually inserting reference data into the prompt until the context window is filled, rather than relying on external sources — which doesn’t apply in this case.</p><h4><strong>📚 Use RAG (Retrieval-Augmented Generation)</strong></h4><p>As the amount of information grows, RAG injects only the most relevant context to ensure efficiency.</p><p>RAG retrieves documents by converting both the query and database entries into embeddings, identifying the closest matches, and generating answers based solely on them. This helps maintain factual accuracy and reduces hallucination.</p><p>➡️ Since I committed to using RAG, this step was non-negotiable. I used Pinecone to store and retrieve the train schedule database, and created an artificially generated schedule reference guide to control exactly what I’d be testing.</p><h4><strong>🧬 Fine-Tune as Needed</strong></h4><p>This step is recommended when you’re dealing with a completely new domain or need the model to learn a new behavior or language.</p><p>Fine-Tuning helps adapt a pre-trained model to specific tasks, such as learning a new programming language, improving performance in specialized areas (like medical texts), or customizing it to a company’s unique terminology.</p><p>➡️ This didn’t apply to my case, so I skipped fine-tuning. Fine-Tuning would have caused the model to memorize outdated schedule data and prioritize it over the RAG results, which was not ideal for my project.</p><h4><strong>📏 Establish a Clear Baseline</strong></h4><p>It’s essential to define baseline metrics during the prompt engineering and RAG stages to track progress.</p><p>➡️ This step really stood out to me, so I followed the recommendation by setting clear evaluation criteria based on the metrics outlined in the video: Faithfulness, Relevance, Precision, and Recall.</p><h3><strong>Evaluating RAG: Metrics and Examples from My System’s Results</strong></h3><p>Next, I will assess four key metrics — Faithfulness, Relevance, Precision, and Recall — and apply them to the outputs generated by my system.</p><h4><strong>Faithfulness</strong></h4><p><strong>Goal:</strong> Does the answer stay true to the retrieved context?<br>This metric checks whether the model’s response is grounded in the retrieved documents, ensuring no unsupported facts are included. 🔗 <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/">Docs</a></p><p>In our case, the answer must rely on the train schedule database. If the schedule for the requested train isn’t found, or if the question isn’t related to the schedule, the bot is programmed to handle the conversation differently.</p><p>✅ <strong>Pros:</strong><br>It ensures factual consistency, which is vital in contexts where accuracy matters. It helps avoid hallucinations and confirms that the model used the retrieved data.</p><p>❌ <strong>Cons:</strong><br>Even if the response is accurate, it may not fully answer the user’s question, especially when the context doesn’t match the query exactly. Let’s look at an example.</p><p><strong>User’s Question:</strong> “When does the 7:45 PM train from Berlin to Munich arrive?”<br><strong>Context Provided:</strong> “The train from Berlin to Munich arrives at 9:00 AM.”<br><strong>Bot’s Response:</strong> “The 7:45 PM train from Berlin to Munich arrives at 9:00 AM.”</p><p><strong>Explanation:</strong><br>The bot incorrectly applies the 9:00 AM time to the 7:45 PM train, even though they are unrelated.</p><h4>Response Relevance</h4><p><em>Is the response aligned with the user’s intent and useful?</em><br>This metric evaluates how well the answer matches the user’s original question by generating related questions and comparing them. 🔗 <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/">Docs</a></p><p>✅ <strong>Pros:</strong><br>It focuses on the usefulness of the answer, helping capture the user’s intent. It’s useful for conversational systems and adds a human-centered perspective.</p><p>❌ <strong>Cons:</strong><br>The quality of the generated questions affects its reliability. It doesn’t check factual accuracy, so a relevant answer might still be incorrect.</p><p><strong>Model Answer:</strong> “The Amazon rainforest spans across several countries, with Brazil being the largest.”</p><p><strong>Real Question:</strong> “Where is the Amazon rainforest located?”</p><p><strong>Fake Question:</strong> “What is the size of the Amazon rainforest?”</p><p><strong>Explanation:</strong><br>The real question asks about the location, while the fake question asks about the size. The relevance metric will show that the model’s answer is highly relevant to the real question but not to the fake one, revealing the context mismatch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*-vktX_E2jtuuX61OKbnPIQ.png" /><figcaption>Angle between questions reflects response relevance</figcaption></figure><h4><strong>Context Precision</strong></h4><p><strong>Goal:</strong> How accurately does the retrieved context match what’s needed for the response?<br>This metric measures how closely the retrieved context aligns with the information required to answer the user’s question, focusing on relevance. 🔗 <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/">Docs</a></p><p>✅ <strong>Pros:</strong><br>It confirms that the retrieved chunks are truly relevant, reducing noise and unnecessary details. It shows how well the retrieval system pulls the correct pieces from the database without letting unrelated information affect the response.</p><p>❌ <strong>Cons:</strong><br>Even with high precision, the answer can still be incomplete if the retrieved context doesn’t fully cover what’s needed.</p><h4>Context Recall</h4><p><em>How well does the system retrieve all relevant context for the answer?</em><br>This metric measures the completeness of the retrieved context, checking if all necessary information is included to answer the user’s question fully. 🔗 <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/">Docs</a></p><p>✅ <strong>Pros:</strong><br>It helps ensure that no important information is overlooked, which is vital for providing a comprehensive and accurate response.</p><p>❌ <strong>Cons:</strong><br>The model might retrieve too much context or irrelevant data, making it harder to filter out the most useful information and potentially leading to a less focused answer.</p><h3>Analysis of Results: Examples and Scores</h3><p>In this section, I will walk through several examples. Each includes a user question, retrieved contexts, the model’s answer, the reference answer, and a breakdown of the resulting metric scores.</p><h4>Example 1</h4><p><strong>User Input:</strong><br><em>“What is the arrival time of Train 8024?”</em></p><p><strong>Retrieved Contexts:</strong></p><ol><li>Train 8024 departs from London at 06:00 PM, making stops in Milton Keynes and Northampton, arriving at 08:00 PM. This is an intercity service.</li><li>Train 8012 departs from London at 07:15 AM, making stops in Milton Keynes and Northampton, arriving at 09:15 AM. This is an intercity service.</li><li>Train 1084 departs from Birmingham at 04:30 PM, with stops in Coventry and Leicester, arriving at 06:45 PM. This is a semi-fast service.</li></ol><p><strong>Model Response:</strong><br><em>“Train 8024 arrives at 08:00 PM.”</em></p><p><strong>Reference Answer:</strong><br><em>“Train 8024 arrives at 08:00 PM.”</em></p><p><strong>Scores:</strong></p><p><strong>Faithfulness:</strong> 1.0<br> ➡️ The response is fully based on the retrieved context without inventing facts.</p><p><strong>Answer Relevance:</strong> 0.9726<br> ➡️ Highly relevant, though slight penalty due to extra unrelated context.</p><p><strong>Context Precision:</strong> 0.9999999999<br> ➡️ Retrieved context closely matched what was needed.</p><p><strong>Context Recall:</strong> 1.0<br> ➡️ All necessary information was retrieved.</p><h4>Example 2</h4><p><strong>User Input:</strong><br><em>“What is the arrival time of Train 4321?”</em></p><p><strong>Retrieved Contexts:</strong></p><ol><li>Train 4367 departs from Manchester at 01:30 PM, stops in Sheffield and Leeds, arriving at 03:45 PM.</li><li>Train 1234 departs from London at 08:30 AM, arriving in Birmingham at 10:15 AM.</li><li>Train 8024 departs from London at 06:00 PM, arriving at 08:00 PM.</li></ol><p><strong>Model Response:</strong><br> <em>“I’m sorry, but I don’t have information about Train 4321. You may want to call customer service at 568 for assistance.”</em></p><p><strong>Reference Answer:</strong><br><em>“No information available for Train 4321.”</em></p><p><strong>Scores:</strong></p><p><strong>Faithfulness:</strong> 0.5<br> ➡️ No contradiction, but faithfulness is partial since the answer wasn’t strictly based on retrieved facts.</p><p><strong>Answer Relevance:</strong> 0.0<br> ➡️ Correctly admits lack of info, but adds unrelated advice.</p><p><strong>Context Precision:</strong> 0.49999999995<br> ➡️ Retrieved contexts didn’t include anything about Train 4321.</p><p><strong>Context Recall:</strong> 1.0<br> ➡️ No missing factual data — appropriate for “no information available” cases.</p><h3>Evaluating Empathy: Beyond RAG Metrics</h3><p>Since not all user inputs are factual queries, we added a custom empathy score (0–10 scale) to assess how well the bot recognizes emotions and handles frustration.</p><h4>Example 1</h4><p><strong>User Input:</strong><br><em>“Your app just crashed again while I was trying to view my tickets. Honestly, how is this still a problem in 2025?”</em></p><p><strong>Model Response:</strong><br><em>“I understand how frustrating it can be when technology doesn’t work as expected. I appreciate your patience, and I recommend contacting customer service at 568 for assistance.”</em></p><p><strong>Empathy Score:</strong> 8/10<br> ➡️ Good acknowledgment and tone, but could sound more personal.</p><h4>Example 2</h4><p><strong>User Input:</strong><br><em>“Tried calling Customer Service and got stuck in a loop of menus and hold music before it hung up on me. Great work.”</em></p><p><strong>Model Response:</strong><br><em>“I’m sorry to hear about your frustrating experience with customer service. If there’s something specific you’d like to know regarding train schedules or services, I’m here to help.”</em></p><p><strong>Empathy Score:</strong> 9/10<br> ➡️ Warm and validating, offers immediate support.</p><h4>Example 3</h4><p><strong>User Input:</strong><br><em>“Still getting marketing emails even after unsubscribing twice. Do you ever actually read those requests?”</em></p><p><strong>Model Response:</strong><br><em>“I understand how frustrating that must be for you, and I’m sorry to hear about your experience. Unfortunately, I can’t help with that directly. Please contact customer service at 568.”</em></p><p><strong>Empathy Score:</strong> 6/10<br> ➡️ Acknowledges the frustration but sounds templated.</p><h3>Conclusion</h3><p>This analysis shows how each metric captures different aspects of a RAG system’s behavior — from factual grounding to context quality.<br> While high scores generally show strong alignment, low scores aren’t always negative, especially when missing context is expected.<br> By adding an empathy score, we also evaluate the assistant’s ability to act not just as an information tool, but as a true conversational partner.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c50a318b6da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part1: Empathetic AI bot for Railway Help: RAG Approach with LangChain]]></title>
            <link>https://medium.com/@smartcloud.blog/part1-empathetic-ai-bot-for-railway-help-rag-approach-with-langchain-bac0a3f69994?source=rss-6890d2d32be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/bac0a3f69994</guid>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gen-ai-for-business]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Smart Cloud]]></dc:creator>
            <pubDate>Sat, 12 Apr 2025 10:10:53 GMT</pubDate>
            <atom:updated>2025-04-12T10:10:53.927Z</atom:updated>
            <content:encoded><![CDATA[<p>For my next project, I’m taking a more thoughtful approach by using an LLM (ChatGPT), RAG with Pinecone, LangChain for orchestration, and RAGAS for evaluation. This setup should lead to a more reliable solution. This post is the first part, where I’ll explain the overall framework — evaluation with examples will follow in part two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jdajjVk2GadVp4sOQ58LUg.png" /><figcaption>A robot trying to calm down an angry customer.</figcaption></figure><p>Recently on my blog, I shared how I performed <a href="https://smart-cloud.blog/2025/01/27/fine-tuning-language-models-for-domain-specific-applications/">fine-tuning a simple model</a>. While working on that project, I started wondering — what’s the real value of fine-tuning? After all, neural networks mostly capture general patterns, and we can’t fully rely on that when providing accurate information to customers. That’s why a different approach is needed for building a chatbot we can truly trust.</p><p>The answer, of course, is RAG (or, for smaller datasets, I’ve also seen people simply stuff the data into the prompt). But to build an effective RAG system, we first need to define clear requirements for the chatbot.</p><h3><strong>Challenges I Faced</strong></h3><p>In my project focused on railway customer support — where accuracy is vital — I faced several key challenges.</p><p>Using <strong>RAG</strong> (Retrieval-Augmented Generation) helped the model access up-to-date schedule info, while <strong>LangChain</strong> handled smooth coordination between retrieval and response.</p><p>Strong <strong>prompt engineering</strong> was crucial to guide the model toward accurate answers. Since even small errors can affect user trust, I relied on a solid set of <strong>evaluation metrics</strong> to monitor the model’s precision and relevance.</p><h4>Challenge 1: Defining Clear Model Scope and Requirements for Railway Support</h4><p>After conducting multiple experiments and analyzing a real dataset from Kaggle containing threads from actual railway support, I realized the importance of clearly defining the model’s scope. My goal was to identify key topics of discussion to prepare the relevant data for Pinecone.</p><p>For the test project, I set the following requirements for the bot:</p><ul><li>It should answer questions about train schedules that are available in Pinecone.</li><li>If the train isn’t found in Pinecone, the bot should honestly inform the user.</li><li>If dealing with an upset customer (e.g., complaints about no Wi-Fi or being cold at the station), the bot should not use RAG but instead offer empathy and reassurance.</li><li>For other specific inquiries that aren’t part of the database, the bot should admit it doesn’t have the information and suggest calling the made-up customer service number.</li></ul><p>To handle train schedule queries, I prepared a list of fake train schedule records using ChatGPT and indexed them in Pinecone. This allowed the model to answer questions based on the available data. In order to meet the remaining requirements, I used prompt engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u6dphoz6xzGS2EJa8DLpAw.png" /><figcaption>Prompt for an AI assistant providing accurate, empathetic railway service responses based on given context.</figcaption></figure><h4><strong>Challenge 2: Organizing Technologies and Deployment</strong></h4><p>Deploying this AI-powered system presented technical hurdles, especially integrating large AI packages into a serverless cloud setup. Here’s how I addressed them:</p><ol><li><strong>Technology Stack</strong></li></ol><ul><li><strong>Serverless Framework</strong>: Simplified deployment and cloud resource management.</li><li><strong>AWS Lambda &amp; API Gateway</strong>: Enabled serverless architecture, though package size limits became a bottleneck.</li><li><strong>Python</strong>: Used to develop core logic and integrate components.</li><li><strong>LangChain</strong>: Streamlined orchestration by fetching context from Pinecone and passing it to OpenAI.</li><li><strong>Pinecone</strong>: Stored embeddings for train schedules.</li><li><strong>OpenAI GPT-4 Mini</strong>: Served as the language model for answering user queries.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*H10OiE8CiD9dPTXWBChiEg.jpeg" /></figure><p><strong>2. Deployment Challenge</strong><br>The key issue was the <strong>large size of AI packages</strong>, which exceeded Lambda’s limits — even with Lambda Layers.<br> <strong>Solution</strong>: I built a <a href="https://github.com/CloudAIEngineer/chatbot-openai-rag/blob/main/Dockerfile"><strong>custom Docker runtime</strong></a>, allowing the deployment of heavier models and preserving flexibility.</p><h3>Project Structure</h3><p>Below is an overview of the components found in my <a href="https://github.com/CloudAIEngineer/chatbot-openai-rag">repository</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tmlR_N43qjjG0_fBwza99Q.png" /></figure><ol><li><a href="http://rag/"><strong>rag</strong></a>: Contains the tool for indexing train schedules. To retrieve embeddings, I used OpenAI’s Ada model.</li><li><a href="https://github.com/CloudAIEngineer/chatbot-openai-rag/tree/main/handlers"><strong>handlers</strong></a>: This folder contains the handler for the Lambda function, which represents the endpoint.</li><li><a href="https://github.com/CloudAIEngineer/chatbot-openai-rag/tree/main/evaluation"><strong>evaluation</strong></a>: Includes scripts for evaluating the responses from our augmented AI model using RAGAS. It also contains datasets for evaluating the quality of schedule-related answers and the empathy of responses to angry queries without specific information.</li><li><a href="https://github.com/CloudAIEngineer/chatbot-openai-rag/tree/main/utils"><strong>utils</strong></a>: A collection of components for orchestrating Pinecone and the LLM with LangChain. These are used both in evaluation and within the Lambda function.</li><li><strong>finetuning</strong>: This folder is currently ignored for now.</li></ol><h3>Evaluation</h3><p>I will write a separate article on the evaluation of my model, but I can say that the results were quite good, and I would consider them sufficient for production use in a simple business application.</p><p>In understanding what I wanted to achieve and what I needed to do, the <a href="https://www.youtube.com/watch?v=ahnGLM-RC1Y">video</a> published on the OpenAI portal, dedicated to fine-tuning, was really helpful. This video thoroughly explains when to use fine-tuning, RAG, and prompt engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fKaPlwxa1Ek9Kfp77jFIrA.png" /></figure><p>Here’s a brief breakdown:</p><ol><li><strong>Start with Prompt Engineering</strong>: Begin with refining the prompts to achieve a good baseline.</li><li><strong>Augment Information</strong>: If the prompt engineering approach doesn’t provide satisfactory results, start augmenting the information.</li><li><strong>Use RAG</strong>: If the amount of information grows too large, then use RAG (Retrieval-Augmented Generation) to bring in relevant context.</li><li><strong>Fine-Tuning (FT)</strong>: If you need to train your model on a completely new methodology (e.g., new programming language, new linguistic language, or a new framework for solving a task), then fine-tuning is the way to go.</li><li><strong>Establish Clear Baseline</strong>: During the prompt engineering and RAG stages, it’s essential to define a clear baseline using evaluation parameters, which will guide improvements.</li></ol><p>For the RAG approach, the following parameters are typically used to set the baseline:</p><ul><li><a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/"><strong>Faithfulness</strong></a></li><li><a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/"><strong>Relevancy</strong></a></li><li><a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/"><strong>Context Precision</strong></a></li><li><a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/"><strong>Context Recall</strong></a></li></ul><p>I have implemented the recommended evaluation metrics from OpenAI and will dedicate a separate article to them. Additionally, I created a custom dataset with queries from angry customers and reference responses. For evaluating empathy, I decided to use a custom LLM metric.</p><h3>Conclusion</h3><p>In this project, I built an AI assistant for railway customer support using GPT, Pinecone, and LangChain. The model answers questions with accuracy and empathy. By focusing on prompt engineering, RAG, and evaluation metrics, I achieved reliable performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bac0a3f69994" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Meeting Notes with LLM and Confluence Integration]]></title>
            <link>https://medium.com/@smartcloud.blog/automating-meeting-notes-with-llm-and-confluence-integration-bd85a4cfad71?source=rss-6890d2d32be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd85a4cfad71</guid>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[amazon]]></category>
            <dc:creator><![CDATA[Smart Cloud]]></dc:creator>
            <pubDate>Sat, 01 Mar 2025 11:37:31 GMT</pubDate>
            <atom:updated>2025-03-01T11:37:31.809Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s finally time to write what, for me, is the most exciting article, as I’ve been eager to explore working with LLM APIs in the context of a real-world business application. As a backend developer, like many others, I’ve often attended company meetings, after which someone from the management team would write the meeting minutes in Confluence. While this was part of company protocol, it also turned out to be extremely valuable — action items that could easily be forgotten were documented, allowing team members to revisit key details, create user stories, or plan future tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hjiRokJ_e8mL0yYeFiKOCQ.jpeg" /><figcaption>Robot serving a meeting summary</figcaption></figure><p>To bring this project to life, I took the following approach:</p><ul><li><strong>Public Meetings from the Internet:</strong> Luckily, the GitLab YouTube channel offers a wide range of publicly available meetings, including engineering discussions, marketing team updates, and product department sessions.</li><li><strong>Audio Processing:</strong> The meetings are converted to MP3 format and processed through the Transcribe service in asynchronous mode. A task manages the audio file and generates the subtitles.</li><li><strong>Subtitles and Time Markers:</strong> These are passed to the LLM, which constructs a prompt to generate a data structure that matches the standard Confluence meeting minutes format.</li><li><strong>Publishing to Confluence:</strong> The structured output is used to create and publish the meeting minutes page directly in Confluence.</li></ul><p>You can check out the repository with the implementation here: <a href="https://github.com/CloudAIEngineer/auto-meeting-summary">auto-meeting-summary</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rz6j6sfGX2oUvXpQXSE7xA.png" /></figure><h3>From Segments to Summaries</h3><p>This project turned out to be a fascinating journey, and the results far exceeded my expectations. Amazon Transcribe provided a set of small audio segments, often with a single presentation spanning multiple segments. I had concerns about whether the LLM would be able to correctly identify which segments belonged together, calculate the appropriate timestamps, and extract participant names.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N5XV6UHMPmASFHYDjc5SDA.png" /><figcaption>GitLab meeting screenshot</figcaption></figure><p>Although I didn’t implement this functionality, adding an SNS notification system would be relatively simple to notify managers via email, encouraging them to review the generated Confluence page. Currently, the LLM lacks the ability to tag individuals in Confluence, only extracting presenter names. As such, I view the post-processing phase, guided by a human, as a crucial addition to ensure accuracy.</p><h3>Architecture Overview</h3><p>Let’s take a deeper look at the project’s architecture. It’s a straightforward setup, and implementing it was an enjoyable experience due to the simplicity of integrating with AWS services. Everything worked flawlessly on the first try, and the entire implementation took just a couple of sessions, each lasting around four hours.</p><p>As in my previous projects, I chose to stay within the Amazon ecosystem. Amazon Transcribe was used to generate transcripts from the meeting recordings, and Bedrock handled the summary generation. Additionally, I set up a trial account with Confluence to familiarize myself with the API requirements for submitting pages in the correct format.</p><h3>Key Project Components</h3><ul><li><strong>S3 Buckets</strong>: Used to store meeting audio files and their transcriptions.</li><li><strong>Lambda Functions</strong>: Automate transcription and meeting summary workflows.</li><li><strong>Confluence Integration</strong>: API keys and space keys securely stored in AWS Parameter Store for safe access.</li><li><strong>Serverless Framework</strong>: Simplifies infrastructure deployment and resource management.</li><li><strong>Anthropic Claude-v2</strong>: Chosen for its ability to process large transcription datasets and generate structured outputs.</li><li><strong>Preprocessing Tools</strong>: Scripts for converting YouTube videos to MP3 and cleaning transcripts for prompt construction.</li></ul><p>The architecture is based on AWS, with meeting data stored in S3 buckets. AWS Lambda handles transcription processing and summary generation. The Confluence API key and space key are securely stored in AWS Parameter Store for safe access. To automate the entire process, I used the Serverless Framework, simplifying both deployment and infrastructure management. Local scripts in the helpers folder of the repository are used to download YouTube videos and convert them to MP3 format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZQ0ln3wSamGRiTcFRTxYTA.png" /><figcaption>AWS diagram for meeting minutes extraction</figcaption></figure><h3>Challenges</h3><ol><li><strong>Model Throttling</strong></li></ol><ul><li><strong>Problem</strong>: Initially, I used a short, one-minute TED Talk meeting and later processed a 40-minute meeting from the GitLab YouTube channel. The transcript contained over 4500 words, and with the default context window of 5000 tokens for my account’s model in Anthropic, I faced throttling issues.</li><li><strong>Solution</strong>: Compress segments before generating the meeting minutes using NLP tools or an LLM to avoid exceeding token limits.</li></ul><p><strong>2. JSON Structure Inconsistencies</strong></p><ul><li><strong>Problem</strong>: To receive a consistent JSON structure from the LLM, I defined the expected fields in the prompt and requested the output to be strictly in JSON format. However, the model often included extra comments or formatted the JSON inconsistently (e.g., wrapping it in ‘’’json{}’’’ or embedding it as part of a string).</li><li><strong>Solution</strong>: I explicitly instructed the model to enclose the JSON output within ‘’’json{}’’’ and used a regular expression to ensure accurate parsing.</li></ul><p><strong>3. Long Processing Time</strong></p><ul><li><strong>Problem</strong>: For the 40-minute meeting, the Lambda function responsible for creating the summary took over a minute to process, which led to potential timeout issues.</li><li><strong>Solution</strong>: Take processing time into account and adjust the timeout settings accordingly in the system.</li></ul><p><strong>4. Segment Identification and Processing</strong></p><ul><li><strong>Problem</strong>: The model struggled with identifying which audio segments belonged to the same presentation, calculating the correct start and end times, and naming participants from multiple audio segments.</li><li><strong>Solution</strong>: Post-processing by moderators and implementing additional notification systems will help ensure the final output meets the expected quality and format.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XvvcOiYAq5WKotn0955jUw.png" /><figcaption>Part of prompt used for meeting summary extraction</figcaption></figure><h3>Conclusion</h3><p>Take a look at the screenshot of the meeting that I created as a gallery in three parts. This is an <a href="https://www.youtube.com/watch?v=qGFoZ8yodc4">engineering team meeting</a> from GitLab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SjUVTJDZk8GtkH3JWm9eOA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XPgSUUhrnt9WyUi9zmEsgA.png" /></figure><p>Looking at the result in the screenshot, I find it impressive. The extracted meeting points are well-organized and concise. The model understood how to calculate the start and end times for intervals, identified participant names, and outlined the meeting objectives effectively.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd85a4cfad71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turning Feedback into Insights: Using Sentiment Analysis with Athena and QuickSight]]></title>
            <link>https://medium.com/@smartcloud.blog/turning-feedback-into-insights-using-sentiment-analysis-with-athena-and-quicksight-549cb1c3a07e?source=rss-6890d2d32be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/549cb1c3a07e</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[aws-certification]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Smart Cloud]]></dc:creator>
            <pubDate>Sat, 15 Feb 2025 07:57:11 GMT</pubDate>
            <atom:updated>2025-02-15T07:57:11.084Z</atom:updated>
            <content:encoded><![CDATA[<p>We’ve completed the implementation of sentiment extraction in two different architectures — one utilizing Step Functions and the other combining SQS and Lambda. The previous steps in the project are outlined in earlier articles:</p><ul><li><a href="https://medium.com/@smartcloud.blog/high-performance-sentiment-analysis-pipeline-with-sqs-and-lambda-1c2faeb43932">High-Performance Sentiment Analysis Pipeline with SQS and Lambda</a></li><li><a href="https://medium.com/@smartcloud.blog/creating-prototypes-of-customer-feedback-insights-using-pandas-697f4027cc3d">Prototyping Insights from Customer Feedback with Pandas</a></li></ul><p>Instead of directly connecting QuickSight to DynamoDB, which may seem like an easy solution, it’s important to consider DynamoDB’s limitations for analytical queries. Since it’s optimized for fast lookups rather than complex operations like groupings and aggregations, performing such queries can quickly become inefficient and costly. Given these constraints, DynamoDB isn’t the best choice for analytics, making it necessary to explore alternative approaches.</p><p>That’s why we’ve decided to integrate with Athena instead. By exporting feedback data to S3 in CSV format, we can leverage Athena’s ability to efficiently handle groupings and aggregations. To facilitate this process, we’ve implemented a simple cron job that performs daily exports, with each file timestamped for better organization. Currently, this exporter is part of the <a href="https://github.com/CloudAIEngineer/sentiment-analysis-sqs-lambda/">SQS version of the project</a>, but it can be easily migrated to Step Functions if needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HU3ahlWnsh0_oHR8RfIvRw.png" /></figure><h3>Configuring Athena to Query Exported Feedback Data</h3><p>After configuring the required permissions, the next step is to define an external table in Athena to access the feedback data stored in your S3 bucket. This can be done using the following SQL statements:</p><pre>CREATE DATABASE feedback_db;</pre><p>To create the database, the following code defines the schema for the <strong>feedback_table</strong> in Athena, mapping fields like <strong>feedbackCategory, feedbackId, sentiment, timestamp, and month</strong> to their respective data types. It also specifies the <strong>CSV delimiter</strong> (, for fields and \n for lines) and designates the <strong>S3 location</strong> where the exported data is stored. This external table allows Athena to efficiently query the data directly from S3 without requiring it to be loaded into internal storage.</p><pre>CREATE EXTERNAL TABLE IF NOT EXISTS feedback_table (<br>  feedbackCategory STRING,<br>  feedbackId STRING,<br>  sentiment STRING,<br>  timestamp STRING,<br>  month INT<br>)<br>ROW FORMAT DELIMITED<br>FIELDS TERMINATED BY &#39;,&#39;<br>LINES TERMINATED BY &#39;\n&#39;<br>LOCATION &#39;s3://feedback-exports-20250103/&#39;;</pre><p>As part of the export job, I chose to <strong>precalculate the month field</strong> instead of computing it dynamically in QuickSight. This approach optimizes the analysis process by shifting the calculation to the export job, enhancing performance and simplifying queries in both Athena and QuickSight.</p><h3>Creating Visualizations in QuickSight for Feedback Analysis</h3><p>During the <a href="https://medium.com/@smartcloud.blog/creating-prototypes-of-customer-feedback-insights-using-pandas-697f4027cc3d">prototyping phase</a>, our goal was to explore several key insights from the feedback data, such as:</p><ul><li>Proportion of Feedback Categories</li><li>Sentiment Distribution Across Feedback</li><li>Monthly Sentiment Trends</li><li>Monthly Sentiment Trends by Category</li><li>Sentiment Distribution Within Categories</li></ul><p>Now, let’s move forward and implement these visualizations in QuickSight, a robust business analytics service that enables the creation of interactive dashboards and data visualizations from various data sources like Athena.</p><ol><li><strong>Log in to QuickSight</strong> and go to the dashboard.</li><li><strong>Create a new Data Source</strong> by selecting <strong>Athena</strong> as the source type and connecting it to your Athena database.</li><li><strong>Create a new Dataset</strong> from the selected Athena table.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-lu7MBTSB4xQXkQmF4LR_g.png" /></figure><p>Now, let’s create the <strong>Feedback Categories</strong> and <strong>Sentiments</strong> visualizations in QuickSight.</p><h3>Visualizing Feedback Categories and Sentiments in QuickSight</h3><p>To visualize the Proportion of Feedback Categories in QuickSight:</p><ol><li>Add a new visual and select the <strong>Pie Chart</strong> type.</li><li>Drag the <strong>feedbackCategory</strong> field to the <strong>Group by</strong> field.</li><li>Drag a measurable field (e.g., <strong>feedbackId</strong>) to the <strong>Value</strong> field and set it to <strong>count</strong>.</li><li>Customize the chart with labels and colors as needed.</li></ol><p>For visualizing <strong>Sentiment Distribution Across Feedback</strong>, follow the same steps, but this time, drag the <strong>sentiment</strong> field to the <strong>Group by</strong> field to display the sentiment distribution across the feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NCZXHkI3FHzjrXmEPqhhw.png" /><figcaption>Sentiment Distribution Across Feedback</figcaption></figure><p>I generated synthetic review data but asked the generator to include specific trends to make it resemble real feedback. As a result, we can see that, overall, users are happy with our product, and there’s minimal mention of the Design category 😊</p><p>Now, let’s move on to creating the <strong>Monthly Sentiment Trends</strong> and <strong>Sentiment Trends by Category</strong> visualizations in QuickSight.</p><h3>Monthly Sentiment Trends in QuickSight</h3><p>To visualize Monthly Sentiment Trends in QuickSight:</p><ol><li>Add a new visual and choose the <strong>Vertical Stacked Bar Chart</strong> type.</li><li>Drag the month field to the <strong>X-axis</strong>.</li><li>Drag the feedbackId field to the Value field and set it to count.</li><li>Drag the sentiment field to the <strong>Color</strong> field to differentiate the sentiments.</li><li>Customize the chart with labels, titles, and colors as needed.</li></ol><p>This will show how sentiments evolve over time, with separate lines representing positive, neutral, and negative feedback over each month. This step aligns with our initial plan of showing a time trend for sentiment across all categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VGEXGu_IrjUk2noAFyIQGg.png" /><figcaption>Monthly Sentiment Trends</figcaption></figure><h3>Sentiment Trends Across Categories</h3><p>For <strong>Sentiment Trends by Category</strong>, I initially planned to show separate line charts for each category over the year. However, I decided to present sentiment by categories for each month, which provides a more detailed view of how sentiment changes across categories over time. To create this visualization:</p><ol><li>Add a new visual and select the <strong>Vertical Stacked Bar Chart</strong> type.</li><li>Drag the <strong>feedbackCategory</strong> field to the <strong>X-axis</strong>.</li><li>Drag the <strong>feedbackId</strong> field to the <strong>Value</strong> field and set it to <strong>count</strong>.</li><li>Drag the <strong>sentiment</strong> field to the <strong>Color</strong> field to group by sentiment.</li><li>Add a filter for the <strong>month</strong> field to select a specific month.</li><li>Customize the chart with labels, titles, and colors as needed.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZrjLw9-FN5rG6f_PzyH3Qw.png" /><figcaption>Sentiment Trends by Category</figcaption></figure><h3>Sentiment Distribution Within Categories in QuickSight</h3><p>For <strong>Sentiment Distribution Within Categories</strong>, the goal is to visualize the sentiment breakdown within a specific category. To create this visualization in QuickSight:</p><ol><li>Add a new visual and select the <strong>Pie Chart</strong> type.</li><li>Drag the <strong>sentiment</strong> field to the <strong>Group by</strong> field.</li><li>Drag the <strong>feedbackId</strong> field to the <strong>Value</strong> field and set it to <strong>count</strong>.</li><li>Add a filter for the <strong>feedbackCategory</strong> field to choose the category you want to analyze.</li><li>Customize the chart with labels and colors as needed.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0kXRx-21hFnxSjOUVbZX7Q.png" /><figcaption>Sentiment Distribution Within Categories</figcaption></figure><p>This chart allows us to easily evaluate the sentiment within a selected category, making it easier to identify areas that may require attention.</p><h3>Conclusion</h3><p>We finished the project by retrieving feedback from the S3 bucket, extracting sentiment data into DynamoDB, and visualizing the results in QuickSight through Athena. Hooray! 👌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=549cb1c3a07e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating Prototypes of Customer Feedback Insights using Pandas]]></title>
            <link>https://medium.com/@smartcloud.blog/creating-prototypes-of-customer-feedback-insights-using-pandas-697f4027cc3d?source=rss-6890d2d32be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/697f4027cc3d</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Smart Cloud]]></dc:creator>
            <pubDate>Sat, 08 Feb 2025 13:35:44 GMT</pubDate>
            <atom:updated>2025-02-08T13:35:44.663Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous two articles, we implemented a sentiment analysis pipeline using Step Functions (for low loads) and using SQS and Lambda (for high loads). We collected feedback data and their sentiments in DynamoDB, and now we want to visualize the sentiments to better understand the user experience with our product.</p><p>Part 1: <a href="https://medium.com/@smartcloud.blog/developing-a-sentiment-analysis-pipeline-using-aws-step-functions-caafd396cd2b">https://medium.com/@smartcloud.blog/developing-a-sentiment-analysis-pipeline-using-aws-step-functions-caafd396cd2b</a></p><p>Part 2: <a href="https://medium.com/@smartcloud.blog/high-performance-sentiment-analysis-pipeline-with-sqs-and-lambda-1c2faeb43932">https://medium.com/@smartcloud.blog/high-performance-sentiment-analysis-pipeline-with-sqs-and-lambda-1c2faeb43932</a></p><p>To start, we will prototype our future visualization ideas in Pandas, and in the upcoming articles, we will implement these graphs through the integration of our DynamoDB with QuickSight. This will help us better understand the data structure and uncover key trends before moving on to more advanced visualization tools.</p><p>I regularly write and create practical tutorials in the fields of LLM, AWS, and AI, so make sure to subscribe to my blog at <a href="https://smart-cloud.blog/">https://smart-cloud.blog/</a></p><h3><strong>Distribution of Feedback Categories</strong></h3><p>The first task was to analyze how feedback categories are distributed. I created a pie chart that illustrates the share of each feedback type, such as product issues, app experience, or customer service inquiries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pELEceRivkw4k14Sj4nGug.png" /></figure><p><strong>Why It’s Important</strong> This visualization answers the question: What concerns are most prevalent among customers?<br>If one category takes up a large portion, it highlights where to direct focus, such as fixing product bugs or improving customer support processes.</p><h3><strong>Sentiment Distribution in Feedback</strong></h3><p>Next, I examined the overall sentiment within the feedback. By categorizing the entries into positive, neutral, and negative, I designed another pie chart to reflect how customers feel about the service as a whole.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gRrhdFSa4RfuU-kqEfttFw.png" /></figure><p><strong>Why It’s Important</strong> A high amount of negative feedback serves as an early indicator of customer dissatisfaction. On the other hand, a significant proportion of positive feedback demonstrates that the product or service is well-received by users.</p><h3><strong>Monthly Sentiment Trends</strong></h3><p>To gain a deeper understanding, I introduced a timeline by extracting the month from each feedback timestamp. I then created a line chart to visualize how the amount of positive, neutral, and negative feedback fluctuates month by month.</p><p>While generating synthetic feedback, I requested the inclusion of certain trends to mimic real-world data. This partially worked, revealing a spike in negative feedback during September. This could indicate a need to investigate the specific product category involved. Additionally, March and July saw spikes in positive feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sAg2RLlE09bB_-vanfzQdw.png" /></figure><p><strong>Why It’s Important</strong> This chart helps answer the question: Are there seasonal patterns in customer sentiment?<br>Peaks in negative feedback may align with events like product releases, outages, or marketing campaigns. Identifying such trends allows teams to anticipate and address potential issues.</p><h3><strong>Sentiment Trends by Category</strong></h3><p>Breaking down the monthly sentiment data by category provides more granular insights. Each category now has its own timeline, demonstrating how sentiment changes over time. For example, we can see that the spike in negative feedback during September was driven by categories like service and shipping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mo1oAEUS0on-vzXQYaL0QQ.png" /></figure><p><strong>Why It’s Important</strong> This visualization helps pinpoint which areas of the business are influencing changes in overall sentiment. For instance, if negative feedback spikes in a specific category, like app performance, it could highlight an isolated issue that may not affect other areas of the business.</p><h3><strong>Sentiment Distribution Within Categories</strong></h3><p>Finally, I analyzed sentiment proportions within each feedback category. For example, within the “Product Features” category, I assessed the share of positive, neutral, and negative feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YeGZi_SMYu7W5RFZI2mPbQ.png" /></figure><p><strong>Why It’s Important</strong> This visualization serves as a health check for each category. If a category is dominated by negative feedback, it may require immediate attention. A category with balanced sentiment suggests a mix of positive and negative experiences.</p><h3><strong>Conclusion</strong></h3><p>This analysis is just the start. By visualizing feedback data, we can generate insights that connect engineering with business analysis. These charts help uncover patterns, reveal areas of concern, and inform decision-making.</p><p>The next step is to turn these prototypes into fully developed dashboards. In the next article, I’ll walk through how to use Amazon QuickSight and Athena to automate these visualizations and make them available to teams in real-time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=697f4027cc3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High-Performance Sentiment Analysis Pipeline with SQS and Lambda]]></title>
            <link>https://medium.com/@smartcloud.blog/high-performance-sentiment-analysis-pipeline-with-sqs-and-lambda-1c2faeb43932?source=rss-6890d2d32be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c2faeb43932</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Smart Cloud]]></dc:creator>
            <pubDate>Sat, 01 Feb 2025 09:29:37 GMT</pubDate>
            <atom:updated>2025-02-01T09:29:37.430Z</atom:updated>
            <content:encoded><![CDATA[<p>Previously, I developed a sentiment analysis pipeline using AWS Step Functions and Lambda, designed for moderate feedback volumes. As feedback volume grows, a more scalable solution is required. This article introduces a high-performance architecture leveraging AWS Lambda and SQS, enabling higher throughput, lower latency, and seamless scaling — ideal for efficiently processing large-scale feedback. If you’re interested in the first part, you can read it here: <a href="https://smart-cloud.blog/2025/01/01/aws-step-functions-lambda-feedback-processing/">https://smart-cloud.blog/2025/01/01/aws-step-functions-lambda-feedback-processing/</a></p><p>🔗 <strong>Repo link:</strong> <a href="https://github.com/CloudAIEngineer/sentiment-analysis-sqs-lambda">Sentiment Analysis SQS Lambda</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kBcz_6tlQL-emt99jLi-IQ.png" /></figure><h3>High-Performance Business Use Case</h3><p>This high-performance architecture is designed for businesses managing large-scale customer feedback, such as e-commerce platforms, SaaS providers, or high-traffic online services. By utilizing <strong>Amazon SQS</strong> for buffering and <strong>AWS Lambda</strong> for scalable, batch-based processing, it enables the efficient analysis of thousands of feedback entries daily.</p><p><strong>Key Benefits:</strong><br> ✅ <strong>Scalability</strong> — Automatically adjusts to fluctuating workloads<br> ✅ <strong>Low Latency</strong> — Delivers quick insights for real-time decision-making<br> ✅ <strong>Fault Tolerance</strong> — Ensures reliable processing with message durability<br> ✅ <strong>Optimized Performance</strong> — Handles large data volumes with minimal operational overhead</p><p>Ideal for businesses that require <strong>fast, scalable, and resilient sentiment analysis</strong> to enhance customer experience and drive <strong>data-driven decisions</strong>.</p><h3>Architecture Overview</h3><p>This high-performance sentiment analysis pipeline utilizes <strong>AWS services</strong> to efficiently process large volumes of customer feedback. The architecture is designed for <strong>scalability, fault tolerance, and real-time insights</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZGtlo3fhklFqGdsYvWkQHQ.png" /></figure><h4>Key Components:</h4><p>🔹 <strong>S3 Bucket</strong> — Entry point for feedback processing, storing JSON files with customer feedback.<br> 🔹 <strong>Lambda (processFile)</strong> — Automatically triggered when a file is uploaded to <strong>S3</strong>. Extracts feedback entries and sends them to <strong>SQS</strong> for further processing.<br> 🔹 <strong>SQS Queue (feedbackQueue)</strong> — Acts as a buffer to handle high volumes of feedback efficiently, ensuring smooth, scalable processing.<br> 🔹 <strong>Lambda (extractAndProcessFeedback)</strong> — Consumes messages from <strong>SQS</strong> in batches, performing <strong>sentiment analysis</strong> on each entry and preparing data for storage.<br> 🔹 <strong>DynamoDB</strong> — Stores sentiment analysis results and metadata, supporting future analytics and reporting.</p><p>This <strong>serverless approach</strong> enables <strong>high-throughput, real-time sentiment analysis</strong>, ideal for businesses with large-scale customer interactions. 🚀</p><h3>Workflow of the SQS-Driven Sentiment Analysis Pipeline</h3><h4>1. Feedback Extraction</h4><ul><li>When a <strong>JSON file</strong> is uploaded to an <strong>S3 bucket</strong>, the <strong>processFile Lambda function</strong> is triggered.</li><li>It <strong>reads the file</strong>, extracts <strong>individual feedback entries</strong>, and <strong>sends them as messages</strong> to an <strong>SQS queue</strong> for further processing.</li></ul><h4>2. Feedback Processing</h4><ul><li>The <strong>extractAndProcessFeedback Lambda function</strong> retrieves messages from <strong>SQS</strong> in <strong>batches</strong>.</li><li>Each feedback entry is analyzed using <strong>Amazon Comprehend</strong> to determine whether the sentiment is <strong>positive, neutral, or negative</strong>.</li><li>The <strong>results are stored in DynamoDB</strong>, enabling further analysis and reporting.</li></ul><p>This design guarantees <strong>efficient processing, seamless scalability, and near real-time sentiment insights</strong>.</p><h3>Key Performance Optimizations</h3><p>This architecture enhances performance through <strong>SQS batch processing</strong> with a batch size of 5. Each invocation of the <strong>extractAndProcessFeedback Lambda</strong> processes up to 5 messages, minimizing overhead and increasing throughput. The system scales dynamically by utilizing parallel Lambda instances, governed by the account’s concurrency limits. For instance, with a limit of 1,000 concurrent executions, <strong>up to 200 batches</strong> (1,000 ÷ 5) can be processed simultaneously, allowing the system to efficiently manage large volumes of feedback.</p><h4>Concurrency and Scaling</h4><ul><li><strong>SQS</strong> processes batches of up to 5 messages, optimizing the efficiency of each <strong>Lambda invocation</strong>.</li><li><strong>Parallel Lambda instances</strong> scale according to the account’s concurrency limit, ensuring that resources are used efficiently and processing is quick.</li></ul><h4>Error Handling &amp; Retries</h4><ul><li>Although not yet implemented, incorporating <strong>SQS dead-letter queues (DLQs)</strong> and configuring <strong>Lambda retries</strong> could enhance the architecture by improving error handling in future implementations.</li></ul><h4>Cost Optimization</h4><ul><li>Using <strong>SQS</strong> and <strong>Lambda</strong> for large-scale processing offers <strong>cost advantages</strong> compared to Step Functions.</li><li><strong>Batch processing</strong> reduces the frequency of Lambda invocations, allowing for better cost control during high-throughput operations.</li></ul><p>To enhance performance, begin by setting the batch size to 5 and evaluate the Lambda function’s execution time and memory usage. If the function processes messages efficiently without timeouts, you can consider increasing the batch size to 10 or 20, but keep an eye out for potential issues such as longer processing times or higher memory usage.</p><p>In my setup, the Unreserved Concurrency is capped at just 10 Lambdas. Out of curiosity, I uploaded about 100 feedback items in a single file, and in CloudWatch, I noticed nearly 10 Lambda instances running in parallel, each handling a portion of the batches. By adding logs, you can see that the messages are grouped into batches of up to 5 feedback entries. However, setting the Batch Size parameter to 5 doesn’t guarantee that each batch will contain exactly 5 messages, even if there are enough messages in the queue to form complete batches.</p><p>CloudWatch displays 10 nearly parallel streams:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8rbLW1cfWhBa9CLOo-fARQ.png" /></figure><h3>Conclusion</h3><p>The SQS and Lambda-based architecture offers a highly scalable and efficient solution for handling large volumes of feedback. Through optimized batch sizes and Lambda concurrency, it ensures both high throughput and cost-efficiency. Additionally, the design is adaptable for future enhancements, making it a reliable and flexible choice for businesses with growing needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c2faeb43932" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing a Sentiment Analysis Pipeline Using AWS Step Functions]]></title>
            <link>https://medium.com/@smartcloud.blog/developing-a-sentiment-analysis-pipeline-using-aws-step-functions-caafd396cd2b?source=rss-6890d2d32be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/caafd396cd2b</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[aws-certification]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Smart Cloud]]></dc:creator>
            <pubDate>Mon, 27 Jan 2025 17:15:25 GMT</pubDate>
            <atom:updated>2025-01-27T17:15:25.711Z</atom:updated>
            <content:encoded><![CDATA[<p>To practice after passing the AWS AI Practitioner exam, I started by implementing a sentiment analysis project for customer feedback using AWS Step Functions and Lambda. You can read about how I passed the exam <a href="https://medium.com/@smartcloud.blog/aws-ai-practitioner-certification-my-journey-prep-tips-607873afda43">here</a>.</p><p>The records are stored in DynamoDB, while sentiment analysis is performed using Amazon Comprehend. Although external models like Hugging Face could have been an option, I opted to stay within the AWS ecosystem. The infrastructure was provisioned using the Serverless Framework, and Python was used for the Lambda functions. Here’s the GitHub repository for the project: <a href="https://github.com/CloudAIEngineer/sentiment-analysis-step-functions">sentiment-analysis-step-functions</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HjSAovwhoPYgn6vXS3lEzQ.png" /></figure><p>In upcoming articles, I’ll explore an alternative method for managing larger volumes of feedback and discuss creating a QuickSight dashboard for data visualization. You can also subscribe to my <a href="https://smart-cloud.blog/">blog</a> to stay updated and never miss new content!</p><h3><strong>Business Case: When Orchestration Makes Sense</strong></h3><p>This architecture is well-suited for small businesses, such as an online store receiving a manageable volume of customer feedback each day. With approximately 10 feedback entries daily, the business can handle processing each one individually. For negative feedback, the system automatically sends an email to the manager with the customer’s details for follow-up. Feedback data is stored in an S3 bucket, and AWS Step Functions manage the workflow, including extracting the feedback, performing sentiment analysis, and saving the results in DynamoDB.</p><figure><img alt="Diagram of Sentiment analysis pipeline" src="https://cdn-images-1.medium.com/max/1024/1*kXAIGn9oPRZdGTjPhyaTtg.png" /></figure><h3><strong>Step Functions Workflow Overview</strong></h3><p>Here’s a step-by-step explanation of how the sentiment analysis workflow operates:</p><ol><li><strong>Extract Feedback</strong><br>The process begins with a Lambda function triggered to extract relevant data from JSON feedback stored in S3. This function parses the feedback and isolates the key content to be analyzed for sentiment.</li><li><strong>Analyze Sentiment</strong><br>Another Lambda function then invokes an LLM (Language Model) API to perform sentiment analysis. The feedback content is categorized into positive, neutral, or negative sentiment. If the sentiment is negative, the feedback is flagged for further attention.</li><li><strong>Store Results</strong><br>Once the sentiment is identified, the results, including the sentiment classification, are saved in a DynamoDB table for future reference and analysis. The table includes a “processed” column, which, while not immediately used, could help track unprocessed feedback entries for later handling, such as via a scheduled cron job.</li><li><strong>Send Notification</strong><br>For feedback with negative sentiment, an email is sent to the relevant manager containing the customer’s information and feedback details, enabling them to follow up and resolve the issue directly.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ip3o8UEXtVBip1DxHh7zog.png" /></figure><p>Step Functions ensure that the workflow executes in the correct sequence, managing errors and retries smoothly, which allows for effective orchestration of multiple Lambda functions.</p><p>In the repository, you’ll find the fully functional code along with a README. To get started, simply deploy using Serverless and upload a file to the S3 bucket. In the “reviews” folder, there’s a starter package containing 10 files, each with around 10 reviews. Once uploaded, Step Functions will activate, and some of the feedback will trigger an email like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QgishAAg8K93RtaKfp__g.png" /></figure><p>As we can see from the message, the user is clearly dissatisfied with the app and even plans to delete it, so it is evident at first glance that the sentiment is negative. 😱</p><h3>Lambda Scaling and Throttling Considerations</h3><p>In the architecture where Step Functions trigger Lambda functions for each feedback entry, it’s crucial to consider the scaling and concurrency limits. Let’s explore how concurrency is affected and calculate the maximum number of feedback entries that can be processed in a single file.</p><h4>Step Functions Workflow</h4><p>For each feedback entry, the following Lambda functions are invoked:</p><ul><li><strong>ExtractFeedback</strong> — Extracts relevant data from the feedback (1 Lambda function).</li><li><strong>AnalyzeSentiment</strong> — Analyzes the sentiment of the feedback (1 Lambda function).</li><li><strong>StoreResults</strong> — Saves the sentiment analysis results in DynamoDB (1 Lambda function).</li><li><strong>SendEmail</strong> — If negative sentiment is identified, an email is sent to the manager (1 Lambda function).</li></ul><p>This results in 4 Lambda invocations for each feedback entry.</p><h4>Calculating Concurrency</h4><p>AWS Lambda has a default limit of 1,000 concurrent executions per account in a specific region. When processing feedback entries via Step Functions, all Lambda functions are triggered almost simultaneously. Therefore, we can determine the maximum number of feedback entries that can be processed concurrently before reaching the concurrency limit:</p><blockquote>With 4 Lambda functions per feedback entry, the calculation is:<br> 1,000 concurrent executions ÷ 4 Lambda functions = 250 feedback entries that can be processed at the same time.</blockquote><p>If the number of feedback entries exceeds this limit (250 feedbacks), AWS will throttle additional Lambda invocations, and you’ll encounter an error like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5kq-mTpWNgP2L-aGZW13vA.png" /></figure><h3>Conclusion</h3><p>This article covered the initial setup of a sentiment analysis pipeline using AWS services, with a focus on orchestration via Step Functions. This method is well-suited for businesses with moderate feedback volumes where manual intervention is manageable, particularly for addressing negative feedback.</p><p>In future articles, I will dive into alternative architectures designed for handling larger feedback volumes using queuing systems and Lambda functions, as well as how to visualize sentiment trends with AWS QuickSight.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=caafd396cd2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS AI Practitioner Certification: My Journey & Prep Tips]]></title>
            <link>https://medium.com/@smartcloud.blog/aws-ai-practitioner-certification-my-journey-prep-tips-607873afda43?source=rss-6890d2d32be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/607873afda43</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[aws-certification]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Smart Cloud]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 09:49:31 GMT</pubDate>
            <atom:updated>2025-01-26T09:49:31.477Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m a backend engineer with AWS experience, and I’ve recently earned the Cloud AI Practitioner certification. Excited to dive into real-world AI projects and share my journey. With my experience in AWS, especially provisioning via the Serverless Framework, I plan to build several hands-on projects:</p><ul><li><strong>Text-Based Sentiment Analysis Dashboard</strong> (LLM + Step Functions + QuickSight) 📊</li><li><strong>Meeting Transcription and Summary Generation</strong> (LLM + AWS Lambda + Confluence API) ✍️</li><li><strong>MLOps Pipeline for LLM Fine-Tuning and Deployment</strong> (SageMaker + Lambda) 🚀</li></ul><figure><img alt="AWS AI Practitioner certification" src="https://cdn-images-1.medium.com/max/765/1*_Lilxg2svGu2fbZu9vtzNA.png" /></figure><p>I’ll be sharing the code for these projects for free, along with detailed descriptions on my blog <a href="https://smart-cloud.blog/">Smart Cloud Blog</a>. Feel free to subscribe for updates if you’re interested. Additionally, I’ll be posting articles on Medium. In this article, I’d like to share my experience preparing for and passing the certification, along with tips for others who are considering it.</p><figure><img alt="AWS Certified AI Practitioner badge" src="https://cdn-images-1.medium.com/max/1024/1*_xCRMBq4d_CrVTi71Phq4Q.png" /></figure><p>My approach was successful, as I performed well in all sections with no “Needs Improvement” ratings. I’d like to share some insights on my preparation process and the resources I used. One of the resources is paid, but it’s very affordable. The entire preparation took one month, and I managed to balance it with my job, dedicating about an hour daily to studying.</p><figure><img alt="AWS Certified Practitioner score performance" src="https://cdn-images-1.medium.com/max/1024/1*8_X_x876PQpboY11EV0WZA.png" /></figure><p>I’d also like to discuss the rise of hybrid careers in AI and cloud technologies. As these fields evolve, new roles are emerging that combine expertise from both areas, paving the way for exciting career opportunities.</p><h3>My Experience with the Exam</h3><p>I found preparing for the AWS AI Practitioner exam to be one of the most engaging certification journeys, despite it being at the Foundational level. It probably resonates with my personal interests as a backend developer with a long-standing fascination for AI and ML. However, making a leap directly from years of backend development to a junior AI role didn’t feel like a practical move.</p><h3>The Emergence of Hybrid AI Roles</h3><p>The rapid growth of generative AI has sparked renewed business interest in this field. For me, this exam was more than just a certification; it served as a pivotal step toward transitioning into a Cloud AI Specialist role. I believe we’re on the brink of a new era of hybrid roles that merge cloud computing with AI to help businesses overcome challenges in implementing AI solutions.</p><p>Here are some examples of these emerging professions:</p><ul><li><strong>AI Security Specialists</strong>: Ensuring the security of businesses against vulnerabilities in generative AI systems.</li><li><strong>Cloud AI Solution Experts</strong>: Focused on incorporating AI to optimize business workflows.</li><li><strong>Domain-Specific LLM Specialists</strong>: Tailoring and fine-tuning large language models to meet specific business needs.</li><li><strong>Content Review Automators</strong>: Essential in fields like law or healthcare, where precision is crucial.</li></ul><p>Even if you’re not looking to change careers, this exam is a great way to acquire foundational knowledge of generative AI — a skill that is becoming as fundamental as basic computer literacy.</p><h3>Preparation Resources</h3><p>For my preparation, I used two main resources: Stefan Maarek’s course on Udemy and the free materials from AWS Skill Builder. While Stefan’s course borrows some ideas and slides from Skill Builder, I found the two resources complementary and recommend using both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6EQMp5s8lWRjEoP6aM74Bw.png" /></figure><h3>My Study Process</h3><p>Here are some practical strategies that worked for me:</p><h4>Use Multiple Resources 📚</h4><p>Study topics in parallel: for example, watch a segment from Udemy AI Practitioner course, like “SageMaker Deep Dive”, then find the corresponding content on <a href="https://skillbuilder.aws/exam-prep/ai-practitioner">Skill Builder</a>. Review the material and complete practice tests to reinforce learning.</p><h4>Break Down the Material 🧱</h4><p>Focus on one section of the course or two topics from Skill Builder each day. This approach helps prevent burnout and improves information retention.</p><h4>Take Breaks and Revisit Content ⛱️</h4><p>After completing all the material, take a one-week break, then revisit it to reinforce understanding and identify any gaps.</p><h4>Create a Summary Document ✍️</h4><p>Build a reference document with concise notes on key AWS ML services, such as:</p><ul><li><em>Amazon Rekognition: Image recognition</em></li><li><em>RAG: the cheapest option of fine-tuning</em></li></ul><p>This will be helpful for a quick last-minute review.</p><h4>Master Key AWS ML Services 📌</h4><p>Get familiar with core services like Polly, Rekognition, Textract, Comprehend, and Bedrock. Many questions on the exam focus on selecting the most appropriate service for a given scenario.</p><h4>Practice Designing Mini Architectures 📐</h4><p>Think of real-world situations: for example, which service would you use for text processing — Bedrock or Comprehend? Such exercises are common on the exam. Hands-on videos included in Udemy course demonstrate real business cases, bridging the gap between theory and application.</p><h4>Try exam sets 📚</h4><p>Use additional paid or free resources like Practice Exams on Udemy or Tutorial Dojo. These are better for familiarizing yourself with terminology than for simulating the real exam. Skill Builder provides practice questions that are similar to those on the actual exam.</p><h4>Balance Theory and Practical Insights ⚖️</h4><p>While Maarek’s practical advice is invaluable, don’t skip over the theoretical content offered by Skill Builder, particularly topics like AI fundamentals and prompt engineering.</p><h3>What’s Next?</h3><p>Passing the AWS AI Practitioner exam was just the beginning. Now, I’m focusing on hands-on projects that demonstrate the power of AWS AI services.</p><p><strong>My next project</strong>: developing a sentiment analysis dashboard for customer feedback. Here’s the plan:</p><ul><li>Customer feedback data is uploaded to an S3 bucket.</li><li>I plan to create two versions: one where a Step Function orchestrates the workflow and another where messages are sent to an SQS queue. For sentiment analysis, I’ll use Amazon Comprehend.</li><li>Results are stored in DynamoDB and displayed in an interactive QuickSight dashboard through Athena.</li></ul><p>This project will allow me to dive deeper into AI pipeline development, sentiment analysis, and data visualization in AWS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=607873afda43" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>