<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Eviden Data Science and Engineering Community - Medium]]></title>
        <description><![CDATA[Community of practice around Data Science, Machine Learning, GenAI and Data Engineering - Medium]]></description>
        <link>https://medium.com/eviden-data-science-community?source=rss----fb6ceb507922---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Eviden Data Science and Engineering Community - Medium</title>
            <link>https://medium.com/eviden-data-science-community?source=rss----fb6ceb507922---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:26:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/eviden-data-science-community" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The AI Agents are Coming]]></title>
            <link>https://medium.com/eviden-data-science-community/the-ai-agents-are-coming-a85b8fb1702d?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/a85b8fb1702d</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Daniel Dowler]]></dc:creator>
            <pubDate>Tue, 19 Dec 2023 21:16:21 GMT</pubDate>
            <atom:updated>2023-12-19T21:16:21.029Z</atom:updated>
            <content:encoded><![CDATA[<h4>12 Predictions for 2024</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ai5Zbe80tv6-NbIQmEMTsw.png" /></figure><h3>A High-Level Take from a Low-Level AI Implementer</h3><p>Change is coming with the advent of AI agents. Below, I discuss what AI agents are, and why they are coming now. I also discuss how they will be developed. Finally, I make 12 big predictions regarding AI agent impacts on businesses and consumers in 2024, with the goal of helping people get a head start on what is shaping up to be another wild ride.</p><h3>The Dawn of AI Agents</h3><p>The year 2023 has been the year of generative AI (GenAI) and large-language-models (LLMs) for generating content in ways never seen before. Indeed, there was an explosion in the marketplace of new models and new applications. However, LLMs are just the preamble to the GenAI revolution that has begun.</p><p>LLMs are impressive at generating text, images, and code–outputs that humans can use in their work (or just to have fun). But what if we could go a step further, and use LLMs not just to generate as a final step, but as the glue to accomplish more complex work. For example, what if we could have LLMs do work that requires sequential steps, specialized tools, up-to-date information, and special skills? This is where AI agents are born.</p><p>An AI agent is a program that can pair LLM models with various data and tools to complete tasks, and get work done. Agents move beyond content generation to decision making and action taking. A simple AI agent has the ability to interpret high-level instructions, make a plan, and then execute steps towards achieving a goal. With each step, an AI agent may be able to assess its progress and adjust its strategy. Crucially, these agents have access to sets of enabling tools and data that they can choose to use.</p><p>AI agents will have the means to complete actions. Envision an email agent that will be able to not only generate email text, but also to automatically converse with a recipient towards achieving a goal. Or imagine a scheduling agent with the ability to converse with multiple parties to get availability and interest information, and then schedule (or reschedule) an event based on feedback. The marrying of intelligent decision-making with real-world actions in the form of AI agents will produce powerful and valuable outcomes.</p><h3>They’re Already Here, Some Assembly Required</h3><p>In a way, the popular OpenAI chatbot, ChatGPT, is an AI agent with the goal of providing valuable conversation, and the action of responding to prompt inputs, while using recent memory as context. Recently, chat tools like ChatGPT have also been paired with tools like web search and database connections, to augment queries with up-to-date information.</p><p>The truth is, the building blocks for future AI agents are already here. First, thanks to 2023, now we have a variety of exceptional LLMs, and coding tools to work with them (langchain, Hugging Face, etc.). Second, there are already many existing software programs and functions that are connected to — and do work in — the real world. Third, there is valuable data everywhere. We just lack experience putting these pieces (LLMs, tools, and data) together, but that will develop quickly. As businesses understand the value and see the impact around them, the sense of need for implementing AI agent technology will grow fast.</p><p>While the pieces are here, there is a lot of thought that will need to go into the technical design of AI agents. Developers will need to decide what program tools an agent will have access to, and how the agent will interact with them. They will also need to decide what data is available, and how it is presented. For example, is it in JSON, SQL, or a vector database? Is there accompanying metadata? Is the data chunked intelligently? How is the data updated? And so on.</p><p>Even after figuring out tools and data, there are still many technical decisions to be made. What prompts will the AI agent use, and which LLMs will the agent be able to call? What will be done with the output? How will agent performance be evaluated so that objective improvement decisions can be made?</p><p>On the business side, assembly is also required. Product teams are new to GenAI and AI agents, just like everyone else. Managers will need to gain experience building teams with the right expertise, which will be a mix of domain understanding, product experience, software development, and AI chops. It’s likely that companies will also need to experiment with proof-of-concepts before gaining their balance in production applications.</p><p>Many companies already have the vision. Leaders are preparing their moves, and developers are gaining experience. As an AI/ML consultant, I see this in conversations. The ideas are here already. The implementations are coming soon. Over time, more specific tools will develop, along with business experience. Users will gain confidence as they interact with AI agents, and the wheels of iterative improvement will continue to turn.</p><h3>Envisioning the Future of AI Agents in 2024</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AOO2VyE3gcHizqP-xAIczQ.png" /></figure><p>When I began devising this article a few weeks back, Google hadn’t yet announced its new multi-modal model Gemini, which is able to work adeptly with various types of media (text, images, audio, etc.). The story of what Gemini can do is just at its beginning, and I feel as if the ground is shifting beneath my feet faster than I can write about it. Pairing multi-modal models like Gemini with specialized data and specialized tools, will create even bigger breakthroughs for AI agents in the coming months.</p><p>I anticipate 2024 will be another big year for AI, as innovative companies learn to implement AI agent technology. Here are a few of my AI agent predictions for 2024.</p><ol><li><strong>AI agents do simple work</strong>: AI agents will quickly graduate from novelty toy status, to doing simple, boring, routine work. Tasks like updating documents, scheduling, and auditing will all see newly capable AI agents. These are the low-hanging fruit that companies will use to dip their toes in AI agent waters. Wins here may appear trivial, but they will be a big step from hyped-up AI talk, to concrete evidence that shows businesses are in the game.</li><li><strong>AI agent specialization</strong>: the foundation generative models built and trained by large tech companies are general, and the tools they access out of the box (like web-search) are often general as well. In 2024, I expect to see more specialized models, as well as specialized tools and datasets meant for AI agent use. I expect to see some modularity emerge here as well, particularly in the case of complex tools and proprietary data. Some of these may be offered through APIs as services, or in datasets.</li><li><strong>Multi-agent development</strong>: Don’t expect all AI agents to be lone workers in silos. The natural step after specialization is to have specialized agents work together to complete more complex tasks than any one could do alone. Also, multi-agent frameworks will begin to leverage hierarchies, in which some agents focus on high level objectives, while others take care of task-specific work, and then report up.</li><li><strong>AI agents as consumers</strong>: What does it mean to get more page views from agents? Will there be websites or APIs optimized for AI agent discovery and use? Right now, the web and many other software tools are generally thought of through the lens of human consumption, but this will change as AI agents consume more content, and use more tools.</li><li><strong>More trust and power to choose and do</strong>: AI agents will progress from simple choices, to more impactful decisions. As businesses experience progress, they will learn to trust their AI Agents. These agents may be given budgets and money to spend. Perhaps they will take on tasks like purchasing stocks, or posting personalized ads with time-sensitive interactive deals. For example, imagine haggling with an AI ad agent that can make decisions.<br><em>Ad Agent</em>: “Hey Dan, enjoy 20% off on our new gadget if you purchase right now.”<br><em>Me</em>: ”I’ll do it if you can ship within 3 days, and you also send me some of your product stickers free.”<br><em>Ad Agent</em>: “Agreed! I’ll send 3 gadget stickers in addition. Shipping will cost $x, and we’ll ship to your address. Confirm to initiate payment.”</li><li><strong>AIAaaS (AI Agents as a Service)</strong>: Amazon Web Services for some years now has offered a service called Mechanical Turk, where businesses can hire armies of (human) workers to perform repetitive routine technical tasks like data labeling. Imagine being able to rent AI Agent hours to do similar things on-demand and at-scale. These agents may even become teachable, or tunable, so that they can perform better in chosen areas or tasks. Are you ready for an AI agent to review your competition submission? Or an AI tax preparer that works with your W2s and 1099s? Or how about an on-demand financial advisor that you can bounce ideas off of, and that can quickly move your funds to match your evolving goals?</li><li><strong>Some jobs will face pressure</strong>: There is no doubt, AI agents will continue the trend that GenAI has started, taking some work away from some people. Consider the degree to which,<br>1. Your work is done at a computer.<br>2. Your work is routine.<br>3. You are easily interchangeable with other people with your same skill set.<br>The higher you score on these categories, the more your job may begin to receive pressure from AI agents. Many white collar jobs are on the list. Roles like auditor, technical writer, and customer service agent can be automated to some degree. However, look to AI Agents to augment, rather than outright replace many jobs in 2024. On the other hand, consider work that requires, <br>1. Significant tasks away from a computer or digital interface<br>2. Many varied tasks<br>3. Highly specific knowledge or skills that few people have<br>In these cases, your job is relatively more insulated from AI agent automation.</li><li><strong>New job roles will emerge</strong>: As companies begin to use AI agents, they will need employees who can take care of them, and interact with them. Will the next hot job be Agent Engineer? Agent Manager? Agent Trainer? Agent Evaluator? Agent Auditor? Perhaps AgentOps? It’s hard to tell what exact titles will be, but AI agents will require human counterparts who know how to interact with them, implement them, maintain them, and improve them. AI agents will also require monitoring and oversight.</li><li><strong>Hype and doom</strong>. If there is one thing you can depend on not changing in 2024, it will be wild claims about AI. The talkers and writers in all their exuberance to gain attention (and get paid by advertisers) will continue to stoke the flames that excite us. We’ll read about how AI agents are solving all of our problems, while simultaneously also bringing about the end of the human race. If it sounds too good (or bad) to be true, it probably is. Unfortunately, this will make it hard (as it did in 2023) to tell where the real substance is.</li><li><strong>Calls for governance</strong>. Also already a feature of 2023, calls for governance will continue in 2024. The power of AI agent technology will become more apparent, and so too will the need to organize the way we work with it. However, governments will continue to struggle to guide the development of AI. They will lead somewhat blindly, and be prone to influence from lobbyist groups who are seeking to outmaneuver their own competition. Any entity that acts like they are in control of GenAI is either delusional or wishful. As I mentioned above, the building blocks of AI agents are already freely available. Many government officials don’t understand tech, and many tech leaders don’t understand AI, so on governance we may be several degrees woefully removed from sanity. Unfortunately, 2024 probably won’t be the year of effective AI governance.</li><li><strong>Lots of competition</strong>. LOTS! In 2023, major cloud providers and AI companies competed fiercely for attention in the GenAI space. Expect this to continue like a raging wildfire with more AI companies releasing even better products in 2024. Look to see some helpful AI agents from major tech companies. We’ll also see competition between companies who benefit as implementers and consumers, as they convert AI agent work into business value that they can pass downstream to their customers.</li><li><strong>AI Agent powered devices:</strong> Finally, expect to see some physical devices become more capable and interactive due to AI agent technology. Smart devices will become smart 2.0. (The cynic in me predicts they will be 2.0 times more expensive as well.) For example, I currently have a smart vacuum that cleans some rooms in my house. I don’t talk to it now because it’s not THAT smart. And it doesn’t really talk to me either, other than reporting errors. With AI agent technology I may be able to tell a smart 2.0 vacuum, “hey vac, can you clean this room in an hour, I’m busy here right now”. Or “can you push all the toys there into a pile to help me clean up, and then vacuum around them?” I’m just ideating here. Maybe the first widely available AI-agent-backed physical devices won’t be vacuums, but I’m quite certain they are coming. I’m excited to see them, use them, and understand their capabilities and limitations.</li></ol><p>Did I say 12 predictions? As I was doing final edits and thinking deeper, a few more ideas crossed my mind. I just couldn’t resist that imaginative urge. These next ideas may not surface in 2024, but it will be interesting to see if they show up.</p><ol><li><strong>AI agent assisted malware</strong>: Computer viruses evolve in so many ways, so it stands to reason we’ll see one that is AI-agent-powered. Also, AI agents are coming to identity theft. We may need better ways to spot them, as they will be more convincing than many human criminals out there. To be clear, I’m not predicting that these agents spawn on their own, but rather that human actors will create them.</li><li><strong>Self-destructing AI agent</strong>: We have to create this, because it sounds cool. But seriously, there may be applications where an on-demand API doesn’t make sense, and an AI agent needs to be downloaded to a local device. Once a task is completed successfully, the agent can self-destruct to free up space. Another application may be to enforce a license agreement, and keep customers paying periodically.</li><li><strong>Self-improving AI agent</strong>: There are various ways to improve LLM responses — few-shot prompting, fine-tuning, RAG, RLHF — but these improvements generally have a manual implementation component. A self-improving AI agent will recognize when it has learned something new that is worth internalizing; it will be able to automatically tune it’s own LLM model(s) to improve its native performance.</li><li><strong>AI agent scientists</strong>: In contrast to an AI agent with a business goal, an AI agent scientist may have a learning mandate, coupled with access to tools and data. The agent would devise hypotheses, and test them. Maybe the agent will even be able to request access to new tools. Connected to a real lab (in say materials science, or biotech), this could supercharge experimentation and R&amp;D.</li></ol><p>What about you? What ideas do you have about where AI agents are headed? I’d love to hear about it in the comments.</p><h3>Conclusion</h3><p>LLM apps stole the show in 2023. I anticipate AI Agents will steal the show in 2024. Right now many agents (and LLMs for that matter) are like toys–fun to play with, but not necessarily useful in the real world. However, the building blocks are already here, and we’re figuring out how to put them together. Companies building these toys will graduate soon, releasing production-grade AI agents that are sophisticated, and capable of working in real-life scenarios. Like it or not, AI agents will become part of our experience.</p><p>Thinking about the future with AI agents can bring both a sense of excitement and terror at the same time. The future is coming so quickly, and it is hard to keep up. The power that AI agents bring to perform tasks and complete work represents great opportunity for many people and businesses. We will be able to do more, create more, and enjoy more with less effort. However, these innovative new ideas will undoubtedly disrupt business as we know it, and all of us will need to learn and adapt to the new technology.</p><h3>About the Author</h3><p>Daniel Dowler has consulted on industry projects in healthcare, financial services, entertainment, software, and retail with some of the best companies in the world. He is an AI/ML Principal for Eviden, leading exciting technical implementations of AI/ML. He enjoys using the best technology to solve problems, connecting the dots from the data, to the math, to the business.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a85b8fb1702d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/the-ai-agents-are-coming-a85b8fb1702d">The AI Agents are Coming</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 3] How AI is Changing the IoT Based Predictive Maintenance: Inference, Evaluation, and…]]></title>
            <link>https://medium.com/eviden-data-science-community/part-3-how-ai-is-changing-the-iot-based-predictive-maintenance-inference-evaluation-and-d3db4cf8f263?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/d3db4cf8f263</guid>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[predictive-maintenance]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sandeep Singh]]></dc:creator>
            <pubDate>Tue, 07 Nov 2023 13:39:53 GMT</pubDate>
            <atom:updated>2023-11-07T13:39:53.247Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Part 3] How AI is Changing the IoT Based Predictive Maintenance: Inference, Evaluation, and Optimization</h3><p><em>Contributors</em> <a href="https://medium.com/u/fc3ec864a0ec"><em>Anurag Bhatia</em></a>, <a href="https://medium.com/u/c30bacea13e4">Saurabh</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4AmekjuuWabLyd5Z" /><figcaption>Photo by <a href="https://unsplash.com/@dizzyd718?utm_source=medium&amp;utm_medium=referral">Drew Dizzy Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This section is the third—and final—part of our <a href="https://medium.com/@asksandeep/part-1-how-ai-is-changing-the-iot-based-predictive-maintenance-an-introduction-bcde20fab230">predictive maintenance series</a>. We will quickly go through inference, evaluation, and optimization possibilities.</p><p><strong>Outlining Inference</strong></p><p>An inference pipeline needs to support both historical (i.e., batch) as well as live (i.e., online) inference.</p><p>Batch inference on historical data (sometimes referred to as back-testing) is necessary to decide whether model performance is up to the mark such as running harness for result comparison and applying optimization (e.g., good enough to be deployed in production). Another aspect of the model is production generating predictions (whether and when to alarm) on live data e.g., the requirement of real-time alerts capability).</p><p><strong>Building Evaluation Strategy</strong></p><p>This section will help to formulate the metrics required and steps to move forward on the evaluation techniques.</p><blockquote>Standard metrics</blockquote><p>Challenge: Since it’s not a typical classification or a regression problem, there are no widely agreed upon, industry-wide off-the-shelf performance metrics.</p><p>Solution: We need to frame our predictive maintenance problem as a classification one to use the usual model performance metrics.</p><blockquote>Steps involved</blockquote><p><strong><em>Step 1:</em></strong> Decide the appropriate length of the lead_time (i.e., the time gap between the timestamp of a model alarm and that of the downtime—if any—after the alarm) depending on the use case involved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/0*UpVUmH_2SRYAcfVo" /><figcaption>A predicted observation (<a href="https://www.h2o.ai/blog/ai-driven-predictive-maintenance-with-h2o-hybrid-cloud/">Source</a>)</figcaption></figure><p>Implication: If lead_time is too small (e.g., 10 minutes) it is likely not useful since there will not be enough time for maintenance engineers to go out in the field to check and do any necessary repair work.</p><p><strong><em>Step 2: </em></strong>Categorize each alarm (i.e., anomaly) into whether or not it was useful.</p><p>True Positive: An alarm that was followed by an actual downtime within the lead_time. This can also be called “correct alarms.”</p><p>False Positive: An alarm that was <em>not</em> followed by an actual downtime within the lead_time. This can also be called “false alarms.”</p><p><strong><em>Step 3: </em></strong>Categorize each outage (i.e., downtime) into whether it was caught beforehand by the model prediction (alarm) or not (missed).</p><p>True Negative: A case when neither a downtime happened nor was there a corresponding model alarm to begin with.</p><p>False Negative: A situation where the downtime did happen, but the model failed to catch it (i.e., no prior alarm went out).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/0*WFr-YJj75nrWDbTf" /><figcaption>confusion matrix</figcaption></figure><p>Having followed these steps, we can now pick and choose one or more of the following model performance metrics relevant for classification:</p><ul><li>Confusion matrix</li><li>Precision</li><li>Recall</li><li>f1-score</li></ul><blockquote>Custom metrics</blockquote><p>Depending on the use-case and input data, we might need to explore custom metrics. Here is a relevant example from a less technical and more business perspective:</p><p>Cost-Benefit Analysis: Without getting into the technicalities of machine learning and its related jargon, the business audience will be more interested in answering the following questions:</p><p>a) What exactly is the <strong>value add</strong> here that comes from the solution being implemented? (e.g., What percentage of the total downtimes in a month was the model able to predict beforehand? Accordingly, how many did it miss?)</p><p>Implication: If the model is not catching enough proportion of the downtimes, then it is not adding value to begin with.</p><p>b) At what <strong>cost</strong> does this value add come?</p><p>To be more specific, how many times does the maintenance engineer need to go out in a day (for example) to respond to every alarm?</p><p>Implications:</p><p>Even if the model is catching most—or all — of the downtimes but alerting several times a day, the cost may be prohibitive. Also, there is a real danger that, over time, end users will lose faith in the model predictions and stop taking the alarms seriously. Hence, a custom metric like “number of alarms per day (or hour)” becomes relevant in such situations.</p><p>Let’s assume the lead_time was 5 days, the number of alarms per day, in this case, should be &lt; 0.2 (i.e. ⅕) to be any better than just a random guess.</p><p>Reason: Each alarm will get a time slot of five days to check whether there was a downtime for that specific machine or not and accordingly be called correct and false. This means one alarm at a regular interval every five days will indeed catch all downtimes, but that is just a random baseline—and the machine learning model has to be judged in that context.</p><p><strong>Here are the optimization techniques for such outcomes</strong></p><p>The custom metric of “number of alarms per day” is necessary—but not necessarily sufficient in all situations. Even if the overall average number of alarms is low (which is good), the frequency of the alarm may be disproportionately high on some days while non-existent on most other days:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/0*0OPhM0m78uDGXi13" /><figcaption>series of no. of alarms (not optimized)</figcaption></figure><p>Another pitfall to look out for is that the threshold set by the model might be too low and hence alarming a little too often:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/0*X3R4EaJlZ4C7D3Yj" /><figcaption>observation score and threshold plot</figcaption></figure><p>In such cases, it is important to check the underlying distribution of reconstruction errors in our anomaly detection model. The threshold for alarming is often set (e.g., by formulas like “mean + 3-times standard_deviation”) assuming that the underlying distribution is normal (Gaussian) in shape, but this assumption may not hold true in some cases. Here is a good example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*4CmrgkMhxXV8HEBr" /><figcaption>error distribution</figcaption></figure><p>In other words, it’s a bit like the Tweedie distribution but certainly not Gaussian. In such cases, we either have the option of choosing another way to arrive at the threshold or force the underlying error distribution to first become closer to a Gaussian one and thereafter proceed with the same way/formula to set the threshold as mentioned above. Here is an example of this threshold optimization approach by changing the error distribution (i.e., selectively choosing some error values while ignoring others):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BUF-BGn0yLGI00qg" /><figcaption>Distribution transformation</figcaption></figure><p>Since the new distribution is relatively much more (though still not completely) in sync with the assumption used while calculating the threshold, now the threshold value, as well as the alarms’ frequency, are likely to become more optimized and meaningful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*NzwfP0SYUwszOQXy" /><figcaption>Observation score and threshold plot</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/0*hDOFGJ69oXI343rg" /><figcaption>Optimized alerts</figcaption></figure><p><strong><em>Note</em></strong>: Alarm frequency has come down significantly as a result of threshold optimization. In this specific case, there was no adverse impact on downtimes predicted while the improvement (in terms of reduction in false alarms) was significantly higher.</p><h3>Conclusion</h3><p>With the optimization strategy of the IoT-based predictive maintenance model, I am concluding this discussion. Now we are at the stage where we have an end-to-end understanding of predictive maintenance capability building — from the data ingestion to the failure alerts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3db4cf8f263" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/part-3-how-ai-is-changing-the-iot-based-predictive-maintenance-inference-evaluation-and-d3db4cf8f263">[Part 3] How AI is Changing the IoT Based Predictive Maintenance: Inference, Evaluation, and…</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 2] How AI is Changing the IoT Based Predictive Maintenance: Datasets and ML Models]]></title>
            <link>https://medium.com/eviden-data-science-community/part-2-how-ai-is-changing-the-iot-based-predictive-maintenance-datasets-and-ml-models-c64bf71b0a38?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/c64bf71b0a38</guid>
            <category><![CDATA[predictive-maintenance]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[manufacturing]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sandeep Singh]]></dc:creator>
            <pubDate>Tue, 07 Nov 2023 13:39:34 GMT</pubDate>
            <atom:updated>2023-11-07T13:39:34.603Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Contributors</em> <a href="https://medium.com/u/fc3ec864a0ec"><em>Anurag Bhatia</em></a>, <a href="https://medium.com/u/c30bacea13e4">Saurabh</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZxOV1GcjvhNdWfZa" /><figcaption>Photo by <a href="https://unsplash.com/@jorgedevs?utm_source=medium&amp;utm_medium=referral">Jorge Ramirez</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This is the second part of our <a href="https://medium.com/@asksandeep/part-1-how-ai-is-changing-the-iot-based-predictive-maintenance-an-introduction-bcde20fab230">predictive maintenance series</a>. In this article, we will take a quick snapshot of the datasets one might have to deal with and possible ML models.</p><p><strong>A Variety of Datasets</strong></p><p>In this section, we have broken down the datasets by the type of analysis on the possible entities.</p><blockquote><em>Sensor Analysis</em>: The following factors need to be taken into consideration in analyzing the sensor behavior:</blockquote><blockquote>Total number of sensors</blockquote><p>Implication:</p><ul><li>Can we build sensor-specific models?</li></ul><blockquote>Mapping of sensors to specific locations (i.e., issues/groups)</blockquote><p>Implications:</p><ul><li>If the sensor count is too high, think about focussing on group-based modeling.</li><li>If the group count is also high, think about group-biasing (i.e., prioritize those groups which are more prone to outages).</li></ul><blockquote>Whether sensors are digital or analog in nature?</blockquote><p>Implications:</p><ul><li>Transitions and latency are important for digital sensors while trend and time-series analysis can be more relevant for analog sensors.</li><li>For digital sensors, every transition from 0 to 1 and vice-versa can be used for feature extraction.</li></ul><blockquote>Within each location, which sensors are expected to be in sync with each other?</blockquote><p>Implication:</p><ul><li>Monitoring lag (if any) between in-sync sensors can be used for building latency models (e.g., Unusual lag at some timestamp could be a red flag for anomalous behavior for that part of the machine).</li></ul><blockquote>Do the sensors keep emitting and sending data even when the machine is not being operated (e.g., during the night when the factory is closed)?</blockquote><p>Implications:</p><ul><li>Is the data during non-operational hours relevant for any modeling exercise?</li><li>What if the working hours start from 9 AM and we have included a 3-hour rolling mean step as part of our feature engineering?</li></ul><blockquote><em>Downtime Analysis</em>: As an initial exercise to understand the data in the context of the business problem, here are some initial questions to look for answers to</blockquote><blockquote>What is the long-term trend of downtimes (i.e., outages)? Is the number of downtimes going up/down or consistently staying within a range? Is it too volatile?</blockquote><blockquote>Is there any seasonality to the data (e.g., monthly, quarterly, etc.)?</blockquote><p>Example of seasonality: snapshot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/0*8lTiR1FbUdfEpSTp" /></figure><p>Implication:</p><p>Some months could easily be more vulnerable to machine faults than others. Depending on the industry, there are usually specific periods in a calendar year when the machine’s utilization is at full capacity, while they are under-utilized at other times (e.g., the summer season for rides in amusement parks, harvest season in case of agriculture, etc.). So, the key question from a modeling perspective is: can we apply time-series methods and algorithms to build models (e.g., ARIMA)?</p><blockquote>Can we ascertain what caused each downtime to begin with? Was it preventable? Or was it due to outside factors beyond our control (e.g., weather)?</blockquote><p>Implication: It is important to focus only on those downtimes that can be attributed to maintenance-related issues and thus, could have been detected beforehand.</p><blockquote>Are the downtimes spread all over the place? Or are they concentrated in some pockets (i.e., groups) that seem more vulnerable than others (e.g., more frequent downtimes)?</blockquote><p>Machine-wise example of hours lost in repair:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/0*SWuoWlugQgANXB1W" /><figcaption>sample data description snapshot</figcaption></figure><p>Implication: Some machines seem more vulnerable to breakdowns than others. Can we apply the 80:20 rule? Should we apply group bias?</p><blockquote>What is the extent of disruption for different types of downtimes (e.g., Work on the machine is resumed in &lt; 10 minutes for some problems, while it takes &gt; 3 hours to get the repair done in case of another set of issues)?</blockquote><p>Implication: Assuming the business goal is to reduce downtime hours (thereby preventing revenue loss), it’s better to focus on those problems that are either recurring or take a lot more time to repair and resume operations.</p><p>Snapshot of distribution-plot of downtimes (x-axis) (i.e., various up-times lost due to faults)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*glTmopCFofikXkjO" /><figcaption>Distribution of failures by duration</figcaption></figure><p><em>Implication</em>: Not all types of faults mean the same in terms of up-times. Some faults are often repaired within minutes while a few others are more likely to take hours to do so. The focus needs to be prioritized accordingly.</p><blockquote><em>Telemetry Analysis</em>: IoT mnemonics often send out telemetry messages that might be stored in another database or accessible through another API. Here are some factors to keep in mind while analyzing telemetry data:</blockquote><blockquote>Frequency of emitted messages (e.g., the usual number of messages emitted every hour while the machine is in operation)</blockquote><p>Rate of telemetry messages emitted:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/0*AQw3U9A7w0Q9K9-Q" /><figcaption>count of telemetry messages</figcaption></figure><p><em>Implication</em>: The rate of messages emitted usually stays confined within a range, but it might display spikes on certain occasions. In such cases, it is important to check whether there is a correlation between the time period of those spikes and machine outages (i.e., Can the unusually high number of telemetry messages be considered a signal while building predictive maintenance models?).</p><blockquote>Carefully analyze the schema of the telemetry message. For example, is there a color code column in the schema based on different problems at the part of the machine where the sensor is located?</blockquote><blockquote>What is the nature of the telemetry data?</blockquote><p>Implication: Can NLP or text-similarity measures be used for the analysis of the text in telemetry data?</p><p><strong>The Kind of Models We Can Build</strong></p><blockquote>Statistical Model (Percentile)</blockquote><p>A simple statistical model based on the distribution of the aggregated sensor transitions. There are two major steps used to produce the training artifacts that can be further used for inference:</p><ul><li>Create the model from the original aggregated sensor dataset and identify the type of distribution (e.g., normal, chi-square, binomial, Poisson, and Tweedie).</li><li>Tweek the identified distribution to the normal distribution if the distribution is not normal. The example below states the distribution found as bimodal, and it requires a modification for the upper bound threshold value.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/0*nykPq1mqYudzSI0C" /><figcaption>Bi-modal distribution on single sensor behavior</figcaption></figure><p>A mean value in the distribution above gives the middle value, which divides the bimodal distribution into two unimodal distributions. In the figure below the unimodal distribution on the right side is picked and used as a normally distributed dataset for anomaly detection. The threshold value is calculated, keeping precision and recall in sync. The value of the threshold lies two or three standard deviations away from the mean value to the right of the unimodal distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/0*fgz7Oaxuop3ztkCF" /><figcaption>Sliced unimodal distribution on single sensor behavior</figcaption></figure><blockquote>Classification Model (XGBoost)</blockquote><p>Following the <a href="https://xgboost.readthedocs.io/en/latest/">documentation</a> from XGBoost, it is an optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable. It implements machine learning algorithms under the <a href="https://en.wikipedia.org/wiki/Gradient_boosting">Gradient Boosting</a> framework. XGBoost provides a parallel tree boosting (aka, GBDT, GBM) that solves many data science problems in a fast and accurate way. The same code runs on major distributed environments (i.e., Hadoop, SGE, MPI) and can solve problems beyond billions of examples.</p><p>In the case of a predictive maintenance XGboost classifier, strategic groupings of the sensor value output predict failure. It is not easy to identify the relevant feature that correlates the independent variables (sensor data) to the dependent variables (failures).</p><p>The strategic groupings are created to solve the purpose of feature space expansion and create new opportunities for feature engineering. Such grouping requires intensive exploratory data analysis at the initial stage that can be further automated based on results.</p><p>Each group is created to generate a single trained model for a single area of the problem. For example, pressure sensors considered in a single group handle problems related to pressure maintenance. Similarly, heat sensor groups are created to handle heat maintenance problems. These models alert if they observe anomalous behavior in each particular and for the entire group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/0*A_GeL29H2CMGZut7" /><figcaption>Sensors behavior with predicted and actual failures</figcaption></figure><p>The figure above shows a representative aggregated dataset with alarms marked in grey bars and actual failures marked in red bars. If these grey bars start appearing a few days before actual failures, they are considered true positive alarms. Bars in grey that come without red bars following are considered false positive alarms.</p><blockquote>Anomaly Detection Model (FB Prophet)</blockquote><p>The prophet is an open-source API for handling time-series in its <a href="https://facebook.github.io/prophet/">documentation</a> and is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality—plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend and typically handles outliers well.</p><p>In the case of predictive maintenance case studies, the model can be built on two features for each sensor: one is date timestamp [ds] and the other is sensor output by aggregated transitions. We fit the model by instantiating a new object. Any settings to the forecasting procedure are passed into the constructor as such (e.g., daily_seasonality=True, weekly_seasonality=True, yearly_seasonality=True). Then you call its method and pass in the historical data frame. Fitting should take 1–5 mins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/0*DWV8SrSPK8WB0A7_" /><figcaption>A single sensor example using the prophet model</figcaption></figure><blockquote>Anomaly Detection Model (Autoencoder)</blockquote><p>An encoder is a feedforward, fully connected neural network that compresses the input into a latent space representation and encodes the input as a compressed representation in a reduced dimension. The compressed input is the distorted version of the original input.</p><p>A decoder is also a feedforward network like the encoder and has a similar structure to the encoder. This network is responsible for reconstructing the input back to the original dimensions from the output generated by the encoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cc1VuV1hwJ3YCdAE" /><figcaption>Encoder/Decoder architecture</figcaption></figure><p><strong>Summary</strong></p><p>In summary, most of the datasets consist of sensor transitions (e.g., on/off or sensor telemetry and maintenance log), and the most successful way of coming up with an alerting solution is based on anomaly detection.</p><p><strong>What’s Left?</strong></p><p>The final section will be helpful to conclude the predictive maintenance capability building.</p><p><a href="https://medium.com/@asksandeep/part-3-how-ai-is-changing-the-iot-based-predictive-maintenance-d3db4cf8f263">[Part 3] How AI is Changing the IoT Based Predictive Maintenance: Inference, Evaluation, and Optimization</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c64bf71b0a38" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/part-2-how-ai-is-changing-the-iot-based-predictive-maintenance-datasets-and-ml-models-c64bf71b0a38">[Part 2] How AI is Changing the IoT Based Predictive Maintenance: Datasets and ML Models</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 1] How AI is Changing the IoT-Based Predictive Maintenance: An Introduction]]></title>
            <link>https://medium.com/eviden-data-science-community/part-1-how-ai-is-changing-the-iot-based-predictive-maintenance-an-introduction-bcde20fab230?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/bcde20fab230</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[predictive-maintenance]]></category>
            <category><![CDATA[manufacturing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sandeep Singh]]></dc:creator>
            <pubDate>Tue, 07 Nov 2023 13:39:01 GMT</pubDate>
            <atom:updated>2023-11-07T13:39:01.322Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Contributors</em> <a href="https://medium.com/u/fc3ec864a0ec"><em>Anurag Bhatia</em></a>, <a href="https://medium.com/u/c30bacea13e4">Saurabh</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WP6klazCV44NcoPw" /><figcaption>Photo by <a href="https://unsplash.com/@thisisengineering?utm_source=medium&amp;utm_medium=referral">ThisisEngineering RAEng</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Before we start discussing the role of artificial intelligence (AI) in IoT-based predictive maintenance, let me briefly introduce where and why there is a need for it.</p><p><strong>Where It All Started</strong></p><p>Manufacturing is a unique industry, one where many companies don’t know their actual infrastructure maintenance cost. They deal with the risk of unforeseen downtimes despite doing preventive maintenance. Therefore, industries have been increasingly relying on data science models to understand their operational behavior and the effects of preventive maintenance on expected lost costs.</p><p>Preventive maintenance was a technique previously used to prevent upcoming failures that force organizations into trade-off situations. But due to continuous evolution, machine learning (ML) models allow organizations to break the trade-off situation barriers by avoiding the unplanned downtimes and minimizing planned ones.</p><p><strong>Predictive Maintenance Problem Formulation Strategies</strong></p><p>A predictive maintenance problem can be formulated in many ways, but the following seems to be building blocks in problem formulation and outlining.</p><blockquote>Failure status could be observed using classification and anomaly detection algorithms that run on raw and feature engineered datasets from sensors.</blockquote><blockquote>Datasets used in these systems need to go through a rigorous exploratory data analysis process and were shaped over time by feedback from maintenance engineers.</blockquote><blockquote>Final alerts can be shown in the dashboard or sent over email/SMS to the facility maintenance personnel.</blockquote><p><strong>The Complexity of the Predictive Maintenance Problem</strong></p><p>Complexity is a multi-faceted phenomenon, involving a variety of features including disorder, nonlinearity, and self-organization. The complexity measurement parameters are listed below in order (with the probable outcomes they may change) but are mostly measurable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*LhbCvFSAZx1akRpQKIMiHQ.png" /><figcaption>List of conditions and products of complexity</figcaption></figure><p>Entities, in our case, could be sites, site locations, or sensors.</p><p>Transactions can be from the sensor output, maintenance logs, and telemetry records.</p><p>Disorders could be location-based, operational, maintenance driven, or lack of maintenance, monitoring misconfiguration, due to parts failure.</p><p>The system under maintenance can be of various types such as mechanical, electrical, software, or hardware.</p><p>A client’s (whether internal or external) role is an important condition in various areas of problem complexity such as domain knowledge, alert relevance, and system usability.</p><p>To measure complexity, we need to apply the conditions defined above and analyze the iterations used to achieve the robustness, acceptability, and modularity of the application. This is achieved on optimized computing resources.</p><p><strong>What’s Next?</strong></p><p>Let’s take a deep dive into the datasets, models, inference, evaluation, and optimization in the upcoming stories:</p><p>[<a href="https://medium.com/@asksandeep/part-2-how-ai-is-changing-the-iot-based-predictive-maintenance-datasets-and-ml-models-c64bf71b0a38">Part 2] How AI is Changing the IoT Based Predictive Maintenance: Datasets and ML Models</a></p><p>[<a href="https://medium.com/@asksandeep/part-3-how-ai-is-changing-the-iot-based-predictive-maintenance-d3db4cf8f263">Part 3] How AI is Changing the IoT Based Predictive Maintenance: Inference, Evaluation, and Optimization</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcde20fab230" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/part-1-how-ai-is-changing-the-iot-based-predictive-maintenance-an-introduction-bcde20fab230">[Part 1] How AI is Changing the IoT-Based Predictive Maintenance: An Introduction</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Do your EDA notebooks use containerize deployment?]]></title>
            <link>https://medium.com/eviden-data-science-community/do-your-eda-notebooks-use-containerize-deployment-e25f5eeb0d11?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/e25f5eeb0d11</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Sandeep Singh]]></dc:creator>
            <pubDate>Tue, 07 Nov 2023 13:38:44 GMT</pubDate>
            <atom:updated>2023-11-07T13:38:44.435Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2gFr7DlrYqGkED26" /><figcaption>Photo by <a href="https://unsplash.com/@carrier_lost?utm_source=medium&amp;utm_medium=referral">Ian Taylor</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Exploratory Data Analysis (EDA) refers to the critical process of performing initial investigations on the data with the help of summary statistics and graphical representations. The most common discoveries made are about the completeness, patterns, and anomalies. Sometimes it is also used to test the hypothesis and assumption made on the dataset.</blockquote><p><strong>Why it is important to organize EDA efforts?</strong></p><p>Solving a complex machine-learning problem requires a massive amount of effort in data investigation. These efforts require a lot of code to summarize and preprocess the data. You probably want this process to be as portable as possible. In other words, it can be run as many times as you like, even on different machines on-premises or cloud.</p><p>Unfortunately, it is often seen the code works fine on a local machine and customized virtual environment but gives errors during runtime. It can be due to different versions of libraries installed on the host machine.</p><p>To deal with this problem a combination of docker, terraform, and GitHub actions will help to complete the CI/CD loop.</p><p><strong>Let’s summarize the tool support</strong></p><blockquote>Docker</blockquote><p><a href="https://www.docker.com/">Docker</a> is an open-source platform that allows you to develop, build, ship and run and manage isolated applications. The principle is to develop an application that contains the written code and all the context to run the code dependencies and their versions. For example, when you wrap your application with all its context, you build a Docker image, which can be saved in your local repository, in the Docker Hub or GCP artifact registry.</p><p>To get started with Docker, please, check this<a href="https://www.docker.com/get-started"> documentation</a>.</p><blockquote>GitHub Actions</blockquote><p>GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that helps to Automate your build, test, and deployment pipeline. Actions workflows to be triggered when an event occurs in your repository build and test every code change, or deploy merged pull requests to production. Workflows contain one or more <em>jobs</em> that can run in sequential order or parallel as per the definition.</p><p>To get started with GitHub Actions, please, check this <a href="https://docs.github.com/en/actions/quickstart">documentation</a></p><blockquote>Terraform</blockquote><p>Terraform is an Infrastructure as a code (IAC) open-source software application that lets you define both cloud and on-premises resources in configuration files that you can version reuse, and share. A consistent workflow like GitHub Actions is used to provision and manage all of your infrastructures throughout its lifecycle as defined in the human-readable configuration files. Terraform can manage low-level components like containers, compute, storage, and networking resources, as well as high-level details like DNS entries and SaaS features.</p><p>To get started with Terraform on GCP, please, check this <a href="https://developer.hashicorp.com/terraform/tutorials/gcp-get-started">documentation</a>.</p><blockquote>Google Cloud Platform (GCP)</blockquote><p>We will be considering GCP to elaborate on the example of deploying containerized notebooks. Mainly required services for this deployment are Vertex-AI managed notebooks or user-managed notebooks and access to Google Artifact Registry.</p><p>Finally, A service account for the GitHub Actions workflow will be required with appropriate permission to access all listed GCP services.</p><p><strong>Containerize and deploy</strong></p><p>The success of containerized deployment depends on how well the files are organized. and how consistent workflows are written.</p><p>First of all, You have to prepare a list of notebooks that will contain the different EDA modules, This will also help you to analyze the dependencies required to run the EDA modules. Now let’s add the identified dependencies to the requirements.txt file which will allow us to install all the python modules needed to make our EDA work. The best way to write requirements.txt is to tag the dependencies with the version and make the file useful when writing the Dockerfile.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/12403669d8bb87da30b683e00ce762d1/href">https://medium.com/media/12403669d8bb87da30b683e00ce762d1/href</a></iframe><p>Write the Dockerfile with the base image, The base image could be a pre-build image or a custom image. Also, The container instructions for example installing the python modules and copying the EDA notebooks to the image.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d203f39a751922f3ce77227813b446a1/href">https://medium.com/media/d203f39a751922f3ce77227813b446a1/href</a></iframe><p>Create the terraform modules and environment scripts with configuration to create either user-managed notebooks or google managed notebooks with a custom container image, essential virtual machine, and network configuration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6bd46577f1c5b36d007742b517a1d5ec/href">https://medium.com/media/6bd46577f1c5b36d007742b517a1d5ec/href</a></iframe><p>Write the GitHub actions workflow to push the container image to the google artifact registry. Lastly, add the terraform workflow for provisioning the resources using the configuration file with the definition of terraform module for google notebooks runtime (google managed notebooks) or google notebooks instance (user-managed notebooks) using the docker image from the artifact registry.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c098041b5b353cd1a3148c426467b5c/href">https://medium.com/media/9c098041b5b353cd1a3148c426467b5c/href</a></iframe><blockquote>One important thing to note here google managed notebooks comes with the auto idle shutdown option.</blockquote><p>A similar effort will be required if the same EDA is needed to deploy on a Streamlit app! Of course, there will be additional services will be required on GCP.</p><p>Enjoy Exploring!</p><p><strong>References</strong></p><ol><li><a href="https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/notebooks_instance">https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/notebooks_instance</a></li><li><a href="https://github.com/marketplace/actions/hashicorp-setup-terraform">https://github.com/marketplace/actions/hashicorp-setup-terraform</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e25f5eeb0d11" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/do-your-eda-notebooks-use-containerize-deployment-e25f5eeb0d11">Do your EDA notebooks use containerize deployment?</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[E-Commerce with Generative AI using OpenAI and Azure]]></title>
            <link>https://medium.com/eviden-data-science-community/e-commerce-with-generative-ai-using-openai-and-azure-941c55a3acbd?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/941c55a3acbd</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[ecommerce-web-development]]></category>
            <category><![CDATA[semantic-kernel]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Dave Arlin]]></dc:creator>
            <pubDate>Fri, 03 Nov 2023 19:56:27 GMT</pubDate>
            <atom:updated>2023-11-03T19:56:27.070Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6lsHcUpQX_ZGGckH" /></figure><h3>Azure OpenAI and Semantic Kernel</h3><p>In my initial experience with <a href="https://openai.com/product/chatgpt">ChatGPT</a> and its paid version, ChatGPT Plus, I was astounded by their ability to produce code based on my prompts. These prompts varied from planned product requirements to informal brainstorming about web design aesthetics. I managed to develop about 80% of an <a href="https://dotnet.microsoft.com/en-us/apps/aspnet">ASP.NET</a> Web microservice architecture complete with comprehensive Authentication &amp; Identity management and a frontend Blazor application with a modern look and feel. The remaining 20% involved a combination of some challenges with the tool and my own expertise to fulfill the tasks. You can refer to <a href="https://medium.com/@davearlin/building-an-oauth2-enabled-web-application-with-chatgpt-and-gpt-4-fdeb7e9251fa">my earlier blog post</a> where I detail this journey.</p><p>Once I gained access to the <a href="https://platform.openai.com/docs/api-reference">OpenAI API</a>, the prospect of using this capability through custom application code was exciting. While my initial interactions were somewhat challenging, my attention was soon drawn to Microsoft’s open-source <a href="https://github.com/microsoft/semantic-kernel">Semantic Kernel</a> (SK).</p><p>Semantic Kernel describes itself as a “lightweight SDK facilitating the integration of AI <a href="https://en.wikipedia.org/wiki/Large_language_model">Large Language Models</a> (LLMs) with traditional programming languages.” In essence, it simplifies coding with LLMs such as GPT 3.5 &amp; 4 using this library. As of the time of writing, Semantic Kernel SDK is only available for Python &amp; C#.</p><h3>Virtual Grocer Assistant Demo</h3><p>With the collaboration of my colleagues, we dedicated about a week to constructing a demonstration making use of this library, coupled with an LLM and some Azure services, to display the potential of such an architecture in an e-commerce scenario. Our demonstration was coined the “Virtual Grocer Assistant”.</p><p>Our goal was to empower users to locate their desired shopping items via a text-based input powered by a chat like interface. For instance, the user could type: “I wish to purchase olive oil, eggs, and milk.” and the response would present the available items in the store, offering the option to add these items to their shopping cart. Additionally, we desired to incorporate a fun feature: recipe suggestions. Given the immense training data of LLMs, fulfilling such requests became significantly more feasible.</p><p>Let’s explore some example prompts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LUuIH81CEXyRfvQJJVBz8g.png" /><figcaption>Request a single grocery item</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjZ8cUcvxch3Voxnabzc0g.png" /><figcaption>Request multiple grocery items</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQm934g4qQH4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQm934g4qQH4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQm934g4qQH4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/13991817bbc83d920a9170c32ee6ab47/href">https://medium.com/media/13991817bbc83d920a9170c32ee6ab47/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fl5ExblwlEqw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dl5ExblwlEqw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fl5ExblwlEqw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6379acdae532b09090f7a50a1347e9c/href">https://medium.com/media/d6379acdae532b09090f7a50a1347e9c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fq8uYW_GVM4U%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dq8uYW_GVM4U&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fq8uYW_GVM4U%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0343d9c26970ee89cd6cd3616dca079/href">https://medium.com/media/d0343d9c26970ee89cd6cd3616dca079/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FJvZ52vCn43c%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DJvZ52vCn43c&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FJvZ52vCn43c%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9cb5c7afed0cf2c1b4ec663b31e9acd1/href">https://medium.com/media/9cb5c7afed0cf2c1b4ec663b31e9acd1/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQSyK9ZjFwXs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQSyK9ZjFwXs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQSyK9ZjFwXs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca635898aa67de81122933c02c15279b/href">https://medium.com/media/ca635898aa67de81122933c02c15279b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EU9a5fOHhIuWaewPjk9Odw.png" /><figcaption>Request something that has nothing to do with groceries or recipes</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WaxLrSy4MPMTvBBgcuBd7w.png" /><figcaption>Request a grocery item and something unrelated to groceries or recipes</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQvs5GJOYD0k%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQvs5GJOYD0k&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQvs5GJOYD0k%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/356985b5fe620906b7305e5e6fac0118/href">https://medium.com/media/356985b5fe620906b7305e5e6fac0118/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*snJnUMIIaGRXU54rNMTKrg.png" /><figcaption>Multilingual support (Spanish) — “I want to buy olive oil”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_HTZeJnmNuIJrPec9_cGg.png" /><figcaption>Multilingual support (Arabic) — “I want to buy salsa”</figcaption></figure><h3>Architecture</h3><p>Now let’s unpack the architecture we used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MwPgv1Ml8KL9TOLTdTkviw.png" /><figcaption>High level architecture diagram and flow</figcaption></figure><p>We opted for <a href="https://dotnet.microsoft.com/en-us/apps/aspnet/web-apps/blazor">Blazor .NET 7 WASM</a> as our frontend technology, hosted on an ASP.NET Web API. Then, we employed <a href="https://azure.microsoft.com/en-us/products/cognitive-services/openai-service/?ef_id=_k_CjwKCAjwpayjBhAnEiwA-7enayh2QBr9sqisDW9Dn1shUm34eXUyMqC92P-LduvHp0aQtmrBC2gD2BoCEFQQAvD_BwE_k_&amp;OCID=AIDcmm5edswduu_SEM__k_CjwKCAjwpayjBhAnEiwA-7enayh2QBr9sqisDW9Dn1shUm34eXUyMqC92P-LduvHp0aQtmrBC2gD2BoCEFQQAvD_BwE_k_&amp;gad=1&amp;gclid=CjwKCAjwpayjBhAnEiwA-7enayh2QBr9sqisDW9Dn1shUm34eXUyMqC92P-LduvHp0aQtmrBC2gD2BoCEFQQAvD_BwE">Azure’s OpenAI service</a>, a managed platform for OpenAI’s LLMs, accessible via an API Key. The consumption charge is based on the number of tokens used. You can find more details about the pricing here: <a href="https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/">https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/</a></p><p>Tokens, as defined by Microsoft, are “the fundamental units of text or code that an LLM AI employs to process and generate language.” Both the user’s input and the LLM’s output text are divided into these tokens for processing. GPT-4, which is the latest version at the time of writing, has made significant strides in processing human languages. In this particular demo, we used the “text-davinci-003” LLM.</p><p>We also incorporated <a href="https://azure.microsoft.com/en-us/products/search">Azure Cognitive Search</a> to manage our product inventory. In a real-world setting, we might also use another persistent store like a relational database (Postgres / SQL Server, etc) for additional inventory information. With a structured response from our LLM, we could construct a search query for our Cognitive Search database to retrieve the available products in stock.</p><p>Lastly, we utilized <a href="https://azure.microsoft.com/en-us/products/storage/blobs/?ef_id=_k_CjwKCAjwpayjBhAnEiwA-7enay6Scg0OUR-iGxxGj_l4oHFYFfddc5bCkNYUtPXOh9BT_34mTxQ0fxoCUIgQAvD_BwE_k_&amp;OCID=AIDcmm5edswduu_SEM__k_CjwKCAjwpayjBhAnEiwA-7enay6Scg0OUR-iGxxGj_l4oHFYFfddc5bCkNYUtPXOh9BT_34mTxQ0fxoCUIgQAvD_BwE_k_&amp;gad=1&amp;gclid=CjwKCAjwpayjBhAnEiwA-7enay6Scg0OUR-iGxxGj_l4oHFYFfddc5bCkNYUtPXOh9BT_34mTxQ0fxoCUIgQAvD_BwE">Azure Blob Storage containers</a> to house our product image data, which is then referenced by our frontend.</p><h3>Prompt Engineering</h3><p>Working with Large Language Models (LLMs) requires a shift in mindset — mastering Prompt Engineering is crucial. Ensuring that you’ve clearly set the expectations for the LLM before providing input to it is pivotal; otherwise, coding based on its response could prove challenging.</p><p>In our demonstration, we direct our LLM via our semantic skill to anticipate either recipes, ingredients, or a combination of both, and to deliver a well-structured JSON response. We then convert a portion of this response into a <a href="http://lucenetutorial.com/lucene-query-syntax.html">Lucene query</a>, which we run against our Azure Cognitive Service index containing our inventory data. This process enables us to merge the vast knowledge of the LLM with our private data to provide a relevant response to the end user.</p><p>Token constraints can be a tough nut to crack. The Davinci model we used allows roughly 4000 tokens per response. If a response exceeds this limit, it’s truncated. Although there are methods to retrieve the subsequent set of tokens (response text), it’s an aspect to bear in mind and, in most situations, probably limit. This ensures you don’t end up delivering a 100-page response to the end user and avoid overspending on consumption. It’s also important to note that models may count tokens differently.</p><p>Now let’s look at the ASP.NET Web API side of the code. Initially, it calls Azure OpenAI through the Semantic Kernel library. Apart from the end user’s prompt, we also pass our semantic and native skills to facilitate the processing of the request. The semantic skill is forwarded to the Semantic Kernel library to formulate the prompt sent to the text-davinci-003 model. Following the model’s response, which includes some JSON, our native skill is invoked. At this juncture, we transform the JSON portion of the response into a fuzzy search Lucene query for use in calling Cognitive Search. Lastly, we combine the result from Cognitive Search and the unstructured response from the LLM to create our own structured response sent back to the client.</p><p>Here’s the prompt we utilized in our “PersonalShopper” semantic skill. With more practice and the introduction of newer, more advanced models, we anticipate honing this further.</p><pre>For each item in the input, classify it as a shopping-list-item or a recipe. Examples:<br>I want a chocolate cake | shopping-list-item<br>I want oil | shopping-list-item<br>I&#39;d like a chocolate cake | shopping-list-item<br>I want to make chocolate cake | recipe<br>I want to bake a chocolate cake | recipe<br>I&#39;d like to make my own chocolate cake | recipe<br>I need some milk | shopping-list-item<br>I need chocolate cake | shopping-list-item<br>I want to buy salsa | shopping-list-item<br>I want pre made chocolate cake | shopping-list-item<br>I want cookies from the bakery | shopping-list-item<br><br>If nothing at all in the input can be classified as a shopping-list-item, recipe, or something you would buy at a grocery store, say something humorous in response to the input, then say you&#39;re sorry this is a grocery store.  But, if at least one thing in the input can be classified, then it is OK.<br><br>For a recipe, find the ingredients for the a popular recipe for that item and show the result as a JSON object with the following properties:<br>- RecipeName<br>- RecipeUrl<br>- RecipeDescription<br>- Ingredients<br>  - Name<br>  - Quantity<br>- Directions<br><br>Note, the Directions should be an array.<br><br>For a shopping-list-item, show the result as a JSON object with the following properties:<br>- Name<br><br>All results should be returned in a JSON array.<br><br>++++<br><br>{{$input}}<br><br>++++</pre><p>Let’s look at an example of the raw response returned from the model when typing in <em>“I want to make chocolate cake and buy salsa”</em>.</p><pre><br><br>Sorry, this is a grocery store, so we can&#39;t help you with flying to the moon. Here are the results for the items you mentioned:<br>[<br>  {<br>    &quot;RecipeName&quot;: &quot;Chocolate Cake&quot;,<br>    &quot;RecipeUrl&quot;: &quot;https://www.allrecipes.com/recipe/10813/best-chocolate-cake/&quot;,<br>    &quot;RecipeDescription&quot;: &quot;This is a very easy and delicious chocolate cake you can make in no time. It&#39;s moist and rich, and the flavor is out of this world!&quot;,<br>    &quot;Ingredients&quot;: [<br>      {<br>        &quot;Name&quot;: &quot;all-purpose flour&quot;,<br>        &quot;Quantity&quot;: &quot;2 cups&quot;<br>      },<br>      {<br>        &quot;Name&quot;: &quot;granulated sugar&quot;,<br>        &quot;Quantity&quot;: &quot;2 cups&quot;<br>      },<br>      {<br>        &quot;Name&quot;: &quot;baking soda&quot;,<br>        &quot;Quantity&quot;: &quot;1 teaspoon&quot;<br>      },<br>      {<br>        &quot;Name&quot;: &quot;salt&quot;,<br>        &quot;Quantity&quot;: &quot;1 teaspoon&quot;<br>      },<br>      {<br>        &quot;Name&quot;: &quot;unsweetened cocoa powder&quot;,<br>        &quot;Quantity&quot;: &quot;3/4 cup&quot;<br>      },<br>      {<br>        &quot;Name&quot;: &quot;vegetable oil&quot;,<br>        &quot;Quantity&quot;: &quot;1 cup&quot;<br>      },<br>      {<br>        &quot;Name&quot;: &quot;buttermilk&quot;,<br>        &quot;Quantity&quot;: &quot;1 cup&quot;<br>      },<br>      {<br>        &quot;Name&quot;: &quot;eggs&quot;,<br>        &quot;Quantity&quot;: &quot;2&quot;<br>      },<br>      {<br>        &quot;Name&quot;: &quot;vanilla extract&quot;,<br>        &quot;Quantity&quot;: &quot;2 teaspoons&quot;<br>      },<br>      {<br>        &quot;Name&quot;: &quot;boiling water&quot;,<br>        &quot;Quantity&quot;: &quot;1 cup&quot;<br>      }<br>    ],<br>    &quot;Directions&quot;: [<br>      &quot;Preheat oven to 350 degrees F (175 degrees C). Grease and flour two 9 inch round pans.&quot;,<br>      &quot;In a large bowl, mix together the flour, sugar, baking soda, salt and cocoa.&quot;,<br>      &quot;In a separate bowl, combine the oil, buttermilk, eggs and vanilla. Mix into the dry ingredients until just blended.&quot;,<br>      &quot;Stir in the boiling water last. Batter will be thin. Pour evenly into the prepared pans.&quot;,<br>      &quot;Bake 30 minutes in the preheated oven, or until a toothpick inserted into the center of the cake comes out clean.&quot;<br>    ]<br>  },<br>  {<br>    &quot;Name&quot;: &quot;salsa&quot;<br>  }<br>]</pre><p>In this case, we got a semi-structured response with some text at the top and a JSON object under it. Future versions of this will allow the entire response to always be fully structured.</p><p>Next our native C# “QueryBuilderSkill” is called to form a fuzzy search Lucene query:</p><pre>    [SKFunction(&quot;BuildQuery&quot;)]<br>    [SKFunctionName(&quot;BuildQuery&quot;)]<br>    public string BuildQuery(string item)<br>    {<br>        var jsonStartIndex = item.IndexOf(&quot;[&quot;);<br>        if (jsonStartIndex == -1)<br>        {<br>            jsonStartIndex = item.IndexOf(&quot;{&quot;);<br>        }<br><br>        var itemJson = item.Substring(jsonStartIndex, item.Length - jsonStartIndex);<br>        var unstructuredModelResponse = jsonStartIndex == 0 ? string.Empty : item.Substring(0, jsonStartIndex).Replace(&quot;Here are your results:&quot;, string.Empty).Trim();<br><br>  var json = JsonNode.Parse(itemJson);<br><br>        var subQueries = json switch<br>        {<br>            JsonObject obj =&gt; BuildQueryForObject(obj),<br>            JsonArray arr =&gt; BuildQueryForArray(arr),<br>            _ =&gt; Enumerable.Empty&lt;string&gt;()<br>        };<br><br>        var recipes = json switch<br>        {<br>            JsonObject obj =&gt; new List&lt;Recipe&gt; { BuildRecipeForObject(obj) },<br>            JsonArray arr =&gt; BuildRecipesForArray(arr),<br>            _ =&gt; new List&lt;Recipe&gt;()<br>        };<br><br>        recipes.RemoveAll(r =&gt; string.IsNullOrEmpty(r.Name));<br>        var serializedRecipes = JsonConvert.SerializeObject(recipes);<br>        var query = string.Join(&quot; or &quot;, subQueries.Distinct(StringComparer.InvariantCultureIgnoreCase));<br><br>        return $&quot;{unstructuredModelResponse}{StringExtensions.ModelResponseDelimiter}{serializedRecipes}{StringExtensions.ModelResponseDelimiter}{query}&quot;;<br>    }</pre><p>Here’s the Lucene syntax outputted from our native skill.</p><pre>&quot;all-purpose flour&quot;~ or &quot;granulated sugar&quot;~ or &quot;baking soda&quot;~ or &quot;salt&quot;~ or &quot;unsweetened cocoa powder&quot;~ or &quot;vegetable oil&quot;~ or &quot;buttermilk&quot;~ or &quot;eggs&quot;~ or &quot;vanilla extract&quot;~ or &quot;boiling water&quot;~ or  or &quot;salsa&quot;~</pre><p>Our Cognitive Search index is populated with the following product catalog data:</p><pre>{<br> &quot;value&quot;: [<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;1&quot;,<br>   &quot;name&quot;: &quot;Organic Unbleached White All-Purpose Flour&quot;,<br>   &quot;aliases&quot;: [&quot;white flour&quot;, &quot;all-purpose flour&quot;, &quot;organic flour&quot;, &quot;unbleached flour&quot;, &quot;baking flour&quot;],<br>   &quot;size&quot;: &quot;5 lb&quot;,<br>   &quot;image_name&quot;: &quot;flour.jfif&quot;,<br>   &quot;cost&quot;: 6.79<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;2&quot;,<br>   &quot;name&quot;: &quot;Premium Pure Cane Granulated Sugar&quot;,<br>   &quot;aliases&quot;: [&quot;granulated sugar&quot;, &quot;white sugar&quot;, &quot;sweetener&quot;, &quot;baking sugar&quot;],<br>   &quot;size&quot;: &quot;4 oz&quot;,<br>   &quot;image_name&quot;: &quot;sugar.jfif&quot;,<br>   &quot;cost&quot;: 4.79<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;3&quot;,<br>   &quot;name&quot;: &quot;Cocoa Powder Unsweetened&quot;,<br>   &quot;aliases&quot;: [&quot;cocoa powder&quot;, &quot;chocolate powder&quot;, &quot;baking cocoa&quot;, &quot;unsweetened cocoa&quot;],<br>   &quot;size&quot;: &quot;10 oz&quot;,<br>   &quot;image_name&quot;: &quot;cocoa-powder.jfif&quot;,<br>   &quot;cost&quot;: 7.29<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;4&quot;,<br>   &quot;name&quot;: &quot;Baking Powder&quot;,<br>   &quot;aliases&quot;: [&quot;rising agent&quot;, &quot;leavening agent&quot;, &quot;baking ingredient&quot;],<br>   &quot;size&quot;: &quot;8.1 oz&quot;,<br>   &quot;image_name&quot;: &quot;baking-powder.jpg&quot;,<br>   &quot;cost&quot;: 2.49<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;5&quot;,<br>   &quot;name&quot;: &quot;Acme Pure Baking Soda&quot;,<br>   &quot;aliases&quot;: [&quot;sodium bicarbonate&quot;, &quot;baking soda&quot;, &quot;cleaning agent&quot;, &quot;leavening agent&quot;],<br>   &quot;size&quot;: &quot;1 lb&quot;,<br>   &quot;image_name&quot;: &quot;baking-soda.jpg&quot;,<br>   &quot;cost&quot;: 1.29<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;6&quot;,<br>   &quot;name&quot;: &quot;Acme Natural Sea Salt&quot;,<br>   &quot;aliases&quot;: [&quot;sea salt&quot;, &quot;cooking salt&quot;, &quot;table salt&quot;, &quot;seasoning&quot;],<br>   &quot;size&quot;: &quot;26 oz&quot;,<br>   &quot;image_name&quot;: &quot;salt.jpg&quot;,<br>   &quot;cost&quot;: 3.53<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;7&quot;,<br>   &quot;name&quot;: &quot;DeLallo Instant Espresso Powder&quot;,<br>   &quot;aliases&quot;: [&quot;espresso powder&quot;, &quot;coffee powder&quot;, &quot;instant coffee&quot;, &quot;coffee flavor&quot;],<br>   &quot;size&quot;: &quot;1.94 oz&quot;,<br>   &quot;image_name&quot;: &quot;espresso-powder.webp&quot;,<br>   &quot;cost&quot;: 6.99<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;8&quot;,<br>   &quot;name&quot;: &quot;Acme 2% Reduced Fat Milk&quot;,<br>   &quot;aliases&quot;: [&quot;2% milk&quot;, &quot;reduced fat milk&quot;, &quot;low fat milk&quot;, &quot;dairy&quot;],<br>   &quot;size&quot;: &quot;1 gal&quot;,<br>   &quot;image_name&quot;: &quot;milk.jfif&quot;,<br>   &quot;cost&quot;: 2.79<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;9&quot;,<br>   &quot;name&quot;: &quot;Acme Extra Virgin Olive Oil&quot;,<br>   &quot;aliases&quot;: [&quot;olive oil&quot;, &quot;cooking oil&quot;, &quot;extra virgin olive oil&quot;, &quot;EVOO&quot;],<br>   &quot;size&quot;: &quot;32 oz&quot;,<br>   &quot;image_name&quot;: &quot;olive-oil.jpg&quot;,<br>   &quot;cost&quot;: 13.99<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;10&quot;,<br>   &quot;name&quot;: &quot;Acme Large Brown Eggs&quot;,<br>   &quot;aliases&quot;: [&quot;eggs&quot;, &quot;large eggs&quot;, &quot;brown eggs&quot;, &quot;protein&quot;],<br>   &quot;size&quot;: &quot;12 ct&quot;,<br>   &quot;image_name&quot;: &quot;eggs.jfif&quot;,<br>   &quot;cost&quot;: 1.89<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;11&quot;,<br>   &quot;name&quot;: &quot;Vanilla Extract&quot;,<br>   &quot;aliases&quot;: [&quot;vanilla&quot;, &quot;baking extract&quot;, &quot;flavor extract&quot;, &quot;vanilla flavor&quot;],<br>   &quot;size&quot;: &quot;4 oz&quot;,<br>   &quot;image_name&quot;: &quot;vanilla-extract.jfif&quot;,<br>   &quot;cost&quot;: 14.77<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;12&quot;,<br>   &quot;name&quot;: &quot;Gilbert&#39;s All Natural Caprese Chicken Sausage&quot;,<br>   &quot;aliases&quot;: [&quot;sausage&quot;, &quot;chicken&quot;, &quot;breakfast sausage&quot;, &quot;caprese&quot;],<br>   &quot;size&quot;: &quot;4 ct / 10 oz&quot;,<br>   &quot;image_name&quot;: &quot;chicken-sausage.webp&quot;,<br>   &quot;cost&quot;: 3.99<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;13&quot;,<br>   &quot;name&quot;: &quot;Acme Organic Baby Spinach&quot;,<br>   &quot;aliases&quot;: [&quot;spinach&quot;, &quot;greens&quot;, &quot;salad greens&quot;, &quot;vegetable&quot;, &quot;leafy greens&quot;],<br>   &quot;size&quot;: &quot;5 oz&quot;,<br>   &quot;image_name&quot;: &quot;baby-spinach.jpg&quot;,<br>   &quot;cost&quot;: 3.49<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;14&quot;,<br>   &quot;name&quot;: &quot;Acme Shredded Mild Cheddar Cheese&quot;,<br>   &quot;aliases&quot;: [&quot;cheddar cheese&quot;, &quot;shredded cheese&quot;, &quot;mild cheddar&quot;, &quot;cheese&quot;],<br>   &quot;size&quot;: &quot;8 oz&quot;,<br>   &quot;image_name&quot;: &quot;cheddar-mild-cheese.jfif&quot;,<br>   &quot;cost&quot;: 2<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;15&quot;,<br>   &quot;name&quot;: &quot;Acme Salsa Verde Medium&quot;,<br>   &quot;aliases&quot;: [&quot;salsa&quot;, &quot;medium salsa&quot;, &quot;verde salsa&quot;, &quot;green salsa&quot;, &quot;Mexican sauce&quot;],<br>   &quot;size&quot;: &quot;16 oz&quot;,<br>   &quot;image_name&quot;: &quot;medium-salsa.jfif&quot;,<br>   &quot;cost&quot;: 3.02<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;16&quot;,<br>   &quot;name&quot;: &quot;Acme Salsa Mild&quot;,<br>   &quot;aliases&quot;: [&quot;salsa&quot;, &quot;mild salsa&quot;, &quot;chunky salsa&quot;, &quot;Mexican sauce&quot;],<br>   &quot;size&quot;: &quot;16 oz&quot;,<br>   &quot;image_name&quot;: &quot;mild-salsa.jfif&quot;,<br>   &quot;cost&quot;: 3.49<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;17&quot;,<br>   &quot;name&quot;: &quot;Medium Ripe Avocado&quot;,<br>   &quot;aliases&quot; : [&quot;ripe avocado&quot;, &quot;medium avocado&quot;, &quot;avocado&quot;],<br>   &quot;size&quot;: &quot;1 ct&quot;,<br>   &quot;image_name&quot;: &quot;large-avocado.jfif&quot;,<br>   &quot;cost&quot;: 0.99<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;18&quot;,<br>   &quot;name&quot;: &quot;Acme Pure Ground Black Pepper&quot;,<br>   &quot;aliases&quot; : [&quot;mccormick pepper&quot;, &quot;ground pepper&quot;, &quot;black pepper&quot;],<br>   &quot;size&quot;: &quot;3 oz&quot;,<br>   &quot;image_name&quot;: &quot;black-pepper.jfif&quot;,<br>   &quot;cost&quot;: 5.49<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;19&quot;,<br>   &quot;name&quot;: &quot;Acme Organic Cilantro&quot;,<br>   &quot;aliases&quot; : [&quot;organic cilantro&quot;, &quot;simple cilantro&quot;, &quot;cilantro&quot;],<br>   &quot;size&quot;: &quot;0.5 oz&quot;,<br>   &quot;image_name&quot;: &quot;cilantro.jfif&quot;,<br>   &quot;cost&quot;: 2.29<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;20&quot;,<br>   &quot;name&quot;: &quot;Roma Tomato&quot;,<br>   &quot;aliases&quot; : [&quot;italian tomato&quot;, &quot;red tomato&quot;, &quot;tomato&quot;, &quot;tomatoes&quot;, &quot;roma tomatoes&quot;],<br>   &quot;size&quot;: &quot;1 ct&quot;,<br>   &quot;image_name&quot;: &quot;roma-tomato.jfif&quot;,<br>   &quot;cost&quot;: 0.45<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;21&quot;,<br>   &quot;name&quot;: &quot;Acme Petite Canned Diced Tomatoes&quot;,<br>   &quot;aliases&quot; : [&quot;red tomatoes&quot;, &quot;diced tomatoes&quot;, &quot;canned tomatoes&quot;],<br>   &quot;size&quot;: &quot;14.5 oz&quot;,<br>   &quot;image_name&quot;: &quot;diced-canned-tomatoes.jfif&quot;,<br>   &quot;cost&quot;: 1.00<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;22&quot;,<br>   &quot;name&quot;: &quot;Jalapeno Peppers&quot;,<br>   &quot;aliases&quot; : [&quot;jalapeno&quot;, &quot;hot pepper&quot;, &quot;jalapeno pepper&quot;, &quot;jalapeno peppers&quot;],<br>   &quot;size&quot;: &quot;1 ct&quot;,<br>   &quot;image_name&quot;: &quot;jalapeno-pepper.jfif&quot;,<br>   &quot;cost&quot;: 0.22<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;23&quot;,<br>   &quot;name&quot;: &quot;Jumbo Red Onions&quot;,<br>   &quot;aliases&quot; : [&quot;red onions&quot;, &quot;jumbo onions&quot;, &quot;onion&quot;, &quot;onions&quot;],<br>   &quot;size&quot;: &quot;1 ct&quot;,<br>   &quot;image_name&quot;: &quot;jumbo-red-onion.jfif&quot;,<br>   &quot;cost&quot;: 0.70<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;24&quot;,<br>   &quot;name&quot;: &quot;Garlic&quot;,<br>   &quot;aliases&quot; : [&quot;fresh garlic&quot;, &quot;garlic clove&quot;, &quot;garlic cloves&quot;],<br>   &quot;size&quot;: &quot;1 ct&quot;,<br>   &quot;image_name&quot;: &quot;garlic.jfif&quot;,<br>   &quot;cost&quot;: 0.59<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;25&quot;,<br>   &quot;name&quot;: &quot;Acme Chili Powder&quot;,<br>   &quot;aliases&quot; : [&quot;chili powder&quot;, &quot;smart chili&quot;],<br>   &quot;size&quot;: &quot;1.98 oz&quot;,<br>   &quot;image_name&quot;: &quot;chili-powder.jfif&quot;,<br>   &quot;cost&quot;: 1.25<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;26&quot;,<br>   &quot;name&quot;: &quot;McCormick Ground Cumin&quot;,<br>   &quot;aliases&quot; : [&quot;mccormick cumin&quot;, &quot;ground cumin&quot;],<br>   &quot;size&quot;: &quot;1.5 oz&quot;,<br>   &quot;image_name&quot;: &quot;ground-cumin.webp&quot;,<br>   &quot;cost&quot;: 3.79<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;27&quot;,<br>   &quot;name&quot;: &quot;Large Limes&quot;,<br>   &quot;aliases&quot; : [&quot;green limes&quot;, &quot;lime&quot;, &quot;lime juice&quot;],<br>   &quot;size&quot;: &quot;1 ct&quot;,<br>   &quot;image_name&quot;: &quot;large-lime.jfif&quot;,<br>   &quot;cost&quot;: 0.59<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;28&quot;,<br>   &quot;name&quot;: &quot;Honeycrisp Apple&quot;,<br>   &quot;aliases&quot; : [&quot;sweet apple&quot;, &quot;apple&quot;],<br>   &quot;size&quot;: &quot;1 ct&quot;,<br>   &quot;image_name&quot;: &quot;honeycrisp-apple.jfif&quot;,<br>   &quot;cost&quot;: 1.50<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;29&quot;,<br>   &quot;name&quot;: &quot;Acme Apple Juice&quot;,<br>   &quot;aliases&quot; : [&quot;juice&quot;, &quot;apple juice&quot;],<br>   &quot;size&quot;: &quot;64 fl oz&quot;,<br>   &quot;image_name&quot;: &quot;apple-juice.jfif&quot;,<br>   &quot;cost&quot;: 2.69<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;30&quot;,<br>   &quot;name&quot;: &quot;Fresh Atlantic Salmon Whole Fillet Farm Raised&quot;,<br>   &quot;aliases&quot; : [&quot;salmon fillet&quot;, &quot;atlantic salmon&quot;, &quot;salmon&quot;, &quot;salmon fillets&quot;],<br>   &quot;size&quot;: &quot;1 ct&quot;,<br>   &quot;image_name&quot;: &quot;fresh-atlantic-salmon.jpg&quot;,<br>   &quot;cost&quot;: 24.98<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;31&quot;,<br>   &quot;name&quot;: &quot;Bakery Fresh Chocolate Cake&quot;,<br>   &quot;aliases&quot; : [&quot;cake&quot;, &quot;chocolate cake&quot;, &quot;bakery cake&quot;, &quot;bakery chocolate cake&quot;, &quot;whole chocolate cake&quot;, &quot;premade cake&quot;, &quot;premade chocolate cake&quot;],<br>   &quot;size&quot;: &quot;6 in / 16.7 oz&quot;,<br>   &quot;image_name&quot;: &quot;bakery-chocolate-cake.jfif&quot;,<br>   &quot;cost&quot;: 5.00<br>  },  <br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;32&quot;,<br>   &quot;name&quot;: &quot;Bakery Fresh Oatmeal Raisin Cookies&quot;,<br>   &quot;aliases&quot; : [&quot;cookies&quot;, &quot;oatmeal raisin&quot;, &quot;raisin cookies&quot;, &quot;bakery cookies&quot;, &quot;oatmeal cookies&quot;],<br>   &quot;size&quot;: &quot;16 ct&quot;,<br>   &quot;image_name&quot;: &quot;oatmeal-raisin-cookies.jfif&quot;,<br>   &quot;cost&quot;: 4.49<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;33&quot;,<br>   &quot;name&quot;: &quot;Bakery Fresh Chocolate Chip Cookies&quot;,<br>   &quot;aliases&quot; : [&quot;cookies&quot;, &quot;chocolate chip&quot;, &quot;chocolate chip cookies&quot;, &quot;chocolate cookies&quot;, &quot;bakery cookies&quot;],<br>   &quot;size&quot;: &quot;16 ct&quot;,<br>   &quot;image_name&quot;: &quot;chocolate-chip-cookies.jfif&quot;,<br>   &quot;cost&quot;: 4.49<br>  },<br>  {<br>   &quot;@search.action&quot;: &quot;upload&quot;, <br>   &quot;id&quot;: &quot;34&quot;,<br>   &quot;name&quot;: &quot;Betty Crocker Super Moist Chocolate Fudge Cake Mix&quot;,<br>   &quot;aliases&quot; : [&quot;chocolate cake mix&quot;, &quot;chocolate fudge mix&quot;, &quot;cake mix&quot;, &quot;fudge cake&quot;, &quot;moist cake&quot;, &quot;moist chocolate cake&quot;],<br>   &quot;size&quot;: &quot;15.25 oz&quot;,<br>   &quot;image_name&quot;: &quot;betty-crocker-super-moist-chocolate-fudge-cake.jfif&quot;,<br>   &quot;cost&quot;: 1.79<br>  }<br> ]<br>}</pre><p>Here is the full ASP.NET Web API endpoint logic:</p><pre>[HttpPost]<br>public async Task&lt;ChatMessage&gt; Post([FromBody]ChatPrompt prompt)<br>{<br>  var skContext = await _semanticKernel.RunAsync(prompt.Prompt!, _invSkill[&quot;PersonalShopper&quot;], _querySkill[&quot;BuildQuery&quot;]);<br>  <br>  if (skContext.ErrorOccurred)<br>  {<br>      return new ChatMessage<br>      {<br>          PreContent = skContext.Result,<br>          IsError = true<br>      };<br>  }<br>  <br>  //Parse out both the recipe collection and lucene query from the LLM model + native QueryBuilder skill response <br>  var skContextResultParts = skContext.Result.Split(StringExtensions.ModelResponseDelimiter, StringSplitOptions.RemoveEmptyEntries);<br>  <br>  var unstructuredResponseExists = skContextResultParts.Length == 3;<br>  <br>  Recipe[] recipes;<br>  string? query;<br>  string unstructuredResponse = null;<br>  <br>  if (unstructuredResponseExists)<br>  {<br>      unstructuredResponse = skContextResultParts[0];<br>      recipes = JsonConvert.DeserializeObject&lt;Recipe[]&gt;(skContextResultParts[1]);<br>      query = skContextResultParts[2];<br>  }<br>  else<br>  {<br>      recipes = JsonConvert.DeserializeObject&lt;Recipe[]&gt;(skContextResultParts[0]);<br>      query = skContextResultParts[1];<br>  }<br>   <br>  //Call Cognitive Search passing in the lucene query<br>  var cog = new CogSearch(_config);<br>  var products = cog.SearchProducts(query);<br>  <br>  //Construct response object for our consumer<br>  return new ChatMessage<br>  {<br>      PreContent = unstructuredResponse,<br>      Products = products,<br>      Recipes = recipes,<br>      IsUser = false,<br>      RecipeContent = recipes?.Any() ?? false ?<br>          &quot;Here are some recipe details&quot; :<br>          string.Empty,<br>      InventoryContent = products?.Any() ?? false ?<br>          &quot;These are items we have in stock related to your ask.&quot; :<br>          &quot;We don&#39;t have any of the required ingredients in stock.&quot;<br>  };<br>}</pre><p>At the end, we return a <strong>ChatMessage</strong> that our Blazor frontend can parse for rendering.</p><h3>Alternatives</h3><p>While, I made use of exclusively Microsoft and .NET technologies, there are other options out there. These tools at the time of this demo were the easiest ones for me to get started with. I could’ve used an Angular or React frontend, but sharing common objects between Blazor and my ASP.NET Web API made me go that route.</p><p>We tried some of the <a href="https://platform.openai.com/docs/models">other available models</a> in Azure OpenAI such as gpt-3.5 and text-ada-001. I did not get access to gpt-4 yet which I am eager to try. Google has their <a href="https://ai.google/discover/palm2">PaLM 2</a> model which would be interesting to work with in the future.</p><p>As for the Generative AI platform, there are other options and their capabilities are constantly getting improved. <a href="https://cloud.google.com/blog/products/ai-machine-learning/create-generative-apps-in-minutes-with-gen-app-builder">GCP’s Generative AI App Builder</a> looks interesting, but I have not had a chance to try it or get access to it yet. I don’t know much about <a href="https://aws.amazon.com/bedrock/">Amazon Bedrock</a> at the moment. There is also <a href="https://huggingface.co/">HuggingFace</a> and <a href="https://python.langchain.com/en/latest/index.html">LangChain</a> which look to be promising on the open source side.</p><h3>Conclusion</h3><p>In essence, the combined power of OpenAI, Semantic Kernel, and Azure’s services led to the creation of a robust, AI-driven “Virtual Grocer Assistant”. The AI effectively processed user requests, from simple grocery items to complex recipe suggestions, demonstrating the potential of this technology in revolutionizing the e-commerce landscape. As we continue to build upon this foundation, we anticipate a future where AI integration becomes even more seamless and efficient, heralding a new era in digital user experiences.</p><p>For those interested in seeing some additional examples using Semantic Kernel in Python or C#, check out their repo here: <a href="https://github.com/microsoft/semantic-kernel">https://github.com/microsoft/semantic-kernel</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=941c55a3acbd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/e-commerce-with-generative-ai-using-openai-and-azure-941c55a3acbd">E-Commerce with Generative AI using OpenAI and Azure</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an OAuth2 Enabled Web Application with ChatGPT and GPT 4]]></title>
            <link>https://medium.com/eviden-data-science-community/building-an-oauth2-enabled-web-application-with-chatgpt-and-gpt-4-fdeb7e9251fa?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/fdeb7e9251fa</guid>
            <category><![CDATA[bing]]></category>
            <category><![CDATA[blazor]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[oauth2]]></category>
            <dc:creator><![CDATA[Dave Arlin]]></dc:creator>
            <pubDate>Fri, 03 Nov 2023 19:55:47 GMT</pubDate>
            <atom:updated>2023-11-03T19:55:47.867Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-UjO1qgOf-LmeGqE" /><figcaption>Robot/Human Nightclub Image Generated by Bing / DALL-E</figcaption></figure><h3>Off to the Nightclub with ChatGPT</h3><p>Over the past week, I found myself staying up late at night while I embarked on a journey to build a simple yet functional platform using Microsoft’s <a href="https://learn.microsoft.com/en-us/aspnet/core/security/authentication/identity?view=aspnetcore-7.0&amp;tabs=visual-studio">Identity platform</a>, which I’ve named “Clubbing with ChatGPT.” This platform features a web UI that simulates a user attempting to enter a nightclub. My objective was to leverage Microsoft’s Identity platform, specifically IdentityServer4, to achieve this. The experiment aimed to have <a href="https://chat.openai.com/">ChatGPT</a> with <a href="https://openai.com/research/gpt-4">GPT4</a> generate three distinct applications:</p><ol><li>An <a href="https://dotnet.microsoft.com/en-us/apps/aspnet/apis">ASP.NET Web API</a> functioning as an <a href="https://openid.net/connect/">OpenID Connect</a> Identity Server, serving custom login pages based on the <a href="https://oauth.net/2/">OAuth2</a> client seeking authorization.</li><li>An ASP.NET Web API serving as a Resource API, providing “Strobe Light” data to consumers upon request, as long as their <a href="https://jwt.io/">JWT</a>, created by the aforementioned Identity Server, contained the necessary scope.</li><li>An <a href="https://dotnet.microsoft.com/en-us/apps/aspnet/web-apps/blazor">ASP.NET Blazor</a> WASM application that simulates the user being outside the nightclub. Upon attempting to “enter,” users are prompted to log in. Once authorized, they are transported to a page representing the club’s dance floor. This application also features two additional pages, one for exclusive “VIP” users and another for the “Club Manager.” Access to these pages requires specific user roles. Moreover, the top navigation bar’s color changes according to the user’s role.</li></ol><p>The results were not entirely perfect. I think it’s safe to say that ChatGPT won’t complete 100% of all your work for you, but it sure can help.</p><p>I divided the above applications up into three separate prompts within their own context in ChatGPT.</p><h3>Prompt 1 — Identity Server</h3><blockquote>Build me a central Identity Server auth service using ASP.NET Web API.</blockquote><blockquote>· Use only the built in templates from the dotnet CLI.</blockquote><blockquote>· This service will be the central identity platform and use the OpenID Connect standard to supply access tokens to various clients.</blockquote><blockquote>· This service will use EntityFramework Migrations and SQL Server. The appsettings.Development.json should contain a connection string pointing to “localhost” and using a trusted connection and trusting the server certificate.</blockquote><blockquote>· This app must run on https at port 5001. Insecure http is not allowed. Update the properties.json to accommodate this.</blockquote><blockquote>· This app will use SCSS for styling, I already have the necessary components installed to compile SCSS to CSS.</blockquote><blockquote>· This application will not have a Startup.cs, instead it will have a Program.cs using top level statements.</blockquote><blockquote>· This service will begin by allowing two OAuth2 clients, “chatgpt-clubbin-web-client” and “chatgpt-clubbin-mobile-client”. They will both use the authorization code grant type with PKCE.</blockquote><blockquote>· When an auth challenge is presented to a client using the “chatgpt-clubbin-web-client”, the user should be redirected to a custom login page. Any other OAuth2 clients, including the “chatgpt-clubbin-mobile-client” should use the default IdentityServer4 provided login UI.</blockquote><blockquote>· The “chatgpt-clubbin-web-client” login page should be styled to with a full screen background image. The login form should be horizontally and vertically centered. The login form should have an “&lt;h2&gt;” that says “Are you on the list?” with a username and password textbox underneath it with black background, white borders and white text. There should be a light gray watermark for username and password and no labels for those textboxes. All this content should appear within a translucent &lt;div&gt; that is dark gray.</blockquote><blockquote>· A custom “club-access” scope should be associated with both OAuth2 clients in addition to the default “openid” and “profile” and “offline_access” scopes.</blockquote><blockquote>· These clients and scopes should be seeded in the database, not in memory.</blockquote><blockquote>· Cors origin for the “chatgpt-clubbin-web-client” will be “<a href="https://localhost:6001/">https://localhost:6001</a>” and also stored in the database.</blockquote><blockquote>· The following users should be seeded in the database. Make sure the password policy allows for these shortened passwords.</blockquote><blockquote>1. username: “john”, password: “doe”, first name: “John”, last name: “Doe”, role: “Club Attendee”</blockquote><blockquote>2. username: “jane”, password: “doe” — first name: “Jane”, last name: “Doe”, role: “Club Attendee”</blockquote><blockquote>3. username: “mrbig”, password: “boss”, first name: “Big”, last name: “Boss”, role: “Club Manager”</blockquote><blockquote>4. username: “vip”, password “clubber”, first name: “VIP”, last name: “Clubber”, role: “VIP”</blockquote><blockquote>· There will be future Resource APIs created later that will need to validate the JWTs issued from this service. The first of these will run at “<a href="https://localhost:5002/">https://localhost:5002</a>”</blockquote><p>In certain instances, I provided very specific instructions, such as using EF Migrations, SQL Server, particular users, roles, port numbers, OAuth2 clients, and scopes. I also detailed the exact text I wanted displayed on the login form. For those who have worked with the Identity Platform, it’s no secret that there have been numerous changes to the IdentityServer4 library over the years. While it remains open source, it’s now also a commercial product under Duende Software, which is the sole official source for ongoing support. Additionally, Microsoft’s identity libraries employ this library and add a few extra features.</p><p>ChatGPT struggled with differentiating between the required references and writing the necessary code for a custom login page tailored to a specific OAuth2 client. Ultimately, I had to take the generated code, refactor, and rewrite a significant portion of it. It’s worth noting that ChatGPT’s training data only extends until 2021, which led to some outdated suggestions, such as referencing .NET 6 framework libraries or older .NET Core templates that are no longer current.</p><p>Nevertheless, ChatGPT was useful in providing a general sense of what needed to be done and gave me <strong>13 Steps</strong>, in very specific detail,<strong> </strong>to do so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*bIdTYrDFHoAtREn5mkh56A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*NzJMGTFlVcqNr7wp32RrMA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*E7cdSEm0jjN6J74n-B9IOw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*rNh-XzRg1yiKWbDDL8ijvA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*hGYHR200efTost5MF-HMrQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*bLbfEd9-GXMpzZAAeUbVYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*GoQ8mVBvQG5LBjBtLEwCUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*zdKvOuJhop--tjTndzs33g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*GhrBcLvngMfIkiJpYH0oPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*l7RJ_uiFcjjDzgdjgU7yRw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*16ORfa-3EZ-7Q_Jgaxc73Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*DNpmWBkGMRGBwdwAF1N08w.png" /><figcaption>The 13 step response from my first prompt</figcaption></figure><p>I needed to expand on my initial prompt with a few more specific instructions regarding the custom login page for a particular OAuth2 client. During this process, it became evident that certain aspects of my original prompt were overlooked or disregarded, such as the SCSS generation I had mentioned and assumptions about the existence of specific files that were never present. I also encountered a few runtime errors and sought ChatGPT’s assistance to address them. Surprisingly, ChatGPT was quite adept at helping me resolve these issues. In most instances, the problem stemmed from missing registrations in the Program.cs class for certain types used throughout the code. It’s intriguing that ChatGPT knew how to rectify these problems, yet it failed to recognize that they would arise when it initially provided the code.</p><h3>Prompt 2 — Resource API</h3><blockquote>Create an ASP.NET Web API project named “Acme.StrobeLight.API”. Provide the full source code, styling and command line commands with the following details. If this does not fit in a single response, provide the subsequent prompts for me to accomplish this.</blockquote><blockquote>· This service will be a Resource API that needs to validate JWTs issued by an existing Identity Server called “Acme.IdentityServer” which is an ASP.NET Web API using IdentityServer4. Acme.IdentityServer runs at <a href="https://localhost:5001.">https://localhost:5001.</a></blockquote><blockquote>· This service will run on https locally at <a href="https://localhost:5002/">https://localhost:5002</a></blockquote><blockquote>· This service will provide data to a Blazor web application running at <a href="https://localhost:6001/">https://localhost:6001</a></blockquote><blockquote>· This service will not have a Startup.cs, instead it will have a Program.cs using top level statements.</blockquote><blockquote>· This service will use EntityFramework Migrations and SQL Server. The appsettings.Development.json should contain a connection string pointing to “localhost” and using a trusted connection and trusting the server certificate.</blockquote><blockquote>· The following “StrobeLight” data should be seeded in the database.</blockquote><blockquote>name: Moving Head Light Beams, quantity: 4, maxQuantity: 5</blockquote><blockquote>name: Pro Strike Array 4-LED Strobe / Blinder Effect Light, quantity: 2, maxQuantity: 4</blockquote><blockquote>name: Chauvet DJ Helicopter Q6 RGBW Beam/Strobe/Laser, quantity: 2, maxQuantity 3</blockquote><blockquote>· This service should expose the necessary REST endpoints that allow clients to update the quantity of any existing “StrobeLight” records. This quantity may not be less than 0 or greater than the “maxQuantity” value defined for the StrobeLight being updated.</blockquote><blockquote>· There should also be an endpoint to retrieve the full list of all StrobeLight entities. Users may not add or remove new StrobeLight entities at this time.</blockquote><p>I’ll leave out the responses from ChatGPT and just show some of the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cgdwQsL9o0LXjgyUu3m2bg.png" /><figcaption>Acme.StrobeLight.API project structure created by ChatGPT’s response</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AvEEyqFkZUtOoWX_zjfLSg.png" /><figcaption>Swagger UI showcasing the two endpoints created for the Acme.StrobeLight.API</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*SzgQVlQHZqogoR0iEQvzCA.png" /><figcaption>The “StrobeLightDb” database created via EF Migrations and code provided by ChatGPT’s response</figcaption></figure><p>The development of this particular component was significantly smoother, as it involved a relatively simple and straightforward REST API. This API exposed only two endpoints: one for retrieving a list of all Strobe Lights and another for updating the quantity of a specific strobe light.</p><p>To ensure proper implementation, I only needed to send a few additional prompts. These guided ChatGPT in adding the correct CORS policy for my frontend application and incorporating the “club-access” scope, which was essential for validating the JWT sent by my frontend application.</p><h3>Prompt 3 — Web Application</h3><blockquote>Create a Blazor WASM standalone project named “ChatGPT Clubbing”. Provide the full source code, styling and command line commands with the following details. If this does not fit in a single response, provide the subsequent prompts for me to accomplish this.</blockquote><blockquote>· SCSS will be used for styling the HTML. I already have the necessary tools set up to auto compile scss to css.</blockquote><blockquote>· This application should use the .NET 7 framework</blockquote><blockquote>· This application will run on https at <a href="https://localhost:6001/">https://localhost:6001</a></blockquote><blockquote>· This application will not have a Startup.cs, instead it will have a Program.cs using top level statements.</blockquote><blockquote>· All pages should have their content horizontally and vertically centered with a black background, white borders and white text. All text should be white, buttons should be borderless with white text. When a user hovers over a button, the background color of the button should turn a dark gray. Any content on a page should be contained in a translucent &lt;div&gt; with a dark gray background.</blockquote><blockquote>· The home page will be titled “Outside the Club”. This page should allow anonymous users to access it. This page will have a full screen video taking up the background. This page should additionally have text that says “You’re outside the club. If you’re on the list, head on in!” Under this text should be a button that says “Come In”.</blockquote><blockquote>· When the user clicks the “Come In” button, it should navigate to a page named “Club Floor”. This page will require the user to be authorized. If the user is not authorized, they should be navigated to a login form handled by an existing ASP.NET Web API Identity Server called “Acme.IdentityServer” using OpenID Connect.</blockquote><blockquote>· The identity authority Web API’s root url is <a href="https://localhost:5001/">https://localhost:5001</a>. This application will be authenticated via an OAuth2 client named “chatgpt-clubbin-web-client” using the authorization code grant and passing in the “club-access” scope. There are certain roles that can be returned from this identity authority, they are “Club Attendee”, “VIP”, and “Club Manager”.</blockquote><blockquote>· Once logged in and authorized, the user will be redirected to the “Club Floor” page.</blockquote><blockquote>· The “Club Floor” page will have a video that takes up the whole screen. The text on this screen should display a few paragraphs about having fun at the club and enjoying the excellent music and cocktails. Under this text should be a list of “Strobe Lights” including their name and quantity. These can be listed in a simple table with columns for “Name” and “Quantity” with a sub header above the table named “Check out the awesome lighting!”. This data will be retrieved from an already existing “Acme.StrobeLight.API” resource API at <a href="https://localhost:5002/">https://localhost:5002</a> from an endpoint named “/strobelights” using an HTTP GET and only passing in the JWT from the Acme.IdentityServer as a bearer token.</blockquote><blockquote>· When a user is authenticated, there should be a navigation bar that appears at the top of the “ChatGPT Clubbin” app. This navigation bar should have a dark gray background with white text and white link styles. When you hover over a link, it should give the link text a slightly lighter gray background than the top navigation background color. This navigation bar should not appear if the user is not authenticated.</blockquote><blockquote>· The navigation bar should have a link that says “Go Outside” aligned to the left with a margin of 10px from the left border. This should navigate to the “Outside the Club” page.</blockquote><blockquote>· The navigation bar should have a link that says “Club Floor” 10 px from the left of the “Go Outside” link that navigates to the “Club Floor” page.</blockquote><blockquote>· The navigation bar should have a “Call it a Night” link at the far right of the top nav with a 10px margin from the right border. This should log the user out. Then the user will be navigated to the “Outside the Club” screen.</blockquote><blockquote>· When a “VIP” user logs in, the nav bar should instead have a dark gold color background This user will have an additional link 10px to the right of the “Club Floor” link called “VIP Area”. This link will take this user to a new page called “Behind the Ropes”. This page will have a different video in the background and different text containing some paragraphs about the perks of getting VIP access inside of a typical night club such as bottle service and a spectacular view.</blockquote><blockquote>· When a “Club Manager” user logs in, the nav bar should have a platinum background color. This user will have an additional link 10px to the right of the “Club Floor” link called “Back Office”. This link will take this user to a new page called “Back Office”. This page will contain a list of all available “Strobe Lights” obtained through the “/strobelights” endpoint on the “Acme.StrobeLight.API” api.</blockquote><blockquote>· In the “backoffice” page, the user will also be able to change the quantity of each strobe light via another endpoint defined in the existing “Acme.StrobeLight.API” at the /strobelights endpoint via an HTTP Put, passing in the StrobeLight Id and Quantity. The user may not pass in a value that exceeds the MaxQuantity property defined for each strobe light, returned from the previous call to “/strobelights” via the HTTP Get. Include client side validation in the “backoffice” page. In the table of the list of strobe lights, one of the columns should use a textbox instead of a label to display the Quantity. Under this text box should be a smaller label that displays what the MaxQuantity is with text like “Max: [MaxQuantity]” where MaxQuantity is returned from the HTTP Get to /strobelights.</blockquote><blockquote>· As soon as the user starts to make changes, a “Save” checkbox button should appear to the right of this table as well a “Cancel” x button to its right. Changes to this list should not be submitted to the Strobe Light API until the end user clicks the “Save” button for each row. Once the changes are persisted to the database, the “Save” and “Cancel” button should no longer appear for that row. If the user clicks the “Cancel” button, any changes the end user made to that row should revert back to the original list when first loaded. Then, both the “Save” and “Cancel” buttons should disappear. When the “Save” operation succeeds, text should appear where the “Save” and “Cancel” buttons were that says “You got it boss!”. This text should disappear after 5 seconds.</blockquote><p>I must admit that creating this project with ChatGPT wasn’t without its challenges. I had to follow up with numerous prompts and even restart the process about 10 times, fine-tuning my communication with ChatGPT. On more than one occasion, ChatGPT informed me that my request was overly complicated. Initially, I tried to combine all three application requests into a single prompt, thinking that approach would be more efficient. However, I discovered that dividing them into three separate prompt contexts yielded better results.</p><p>Ultimately, by combining ChatGPT’s responses and making some of my own adjustments, I managed to achieve most of what I had initially set out to accomplish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*soUL_04ZATqPNHES4qZpjA.png" /><figcaption>Clubbing with ChatGPT Home Page — Image Courtesy of Bing Image Creator w/DALL-E</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l-fmNYUfOd-TteUTEnB5ag.png" /><figcaption>Custom Login Page</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tDhWncikheorZ_tfmTP9iQ.png" /><figcaption>Club Floor Page with a standard user</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pwaSz649ufrNgShbQ9jAKQ.png" /><figcaption>VIP Nav Bar</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KRVDzV3VS07dg4_bgKoow.png" /><figcaption>Club Manager Nav Bar</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gfSc_Pg-QPSNMSbWf3hW6g.png" /><figcaption>The Back Office page</figcaption></figure><p>No, ChatGPT did not find these images for me, but it did provide the placeholder video &amp; image URLs for me to fill in. I did some basic Google image searching and also tried to see what Bing Image Creator and DALL-E could get me.</p><p>This final component of the experiment was both remarkable and chaotic. It was interesting to observe ChatGPT’s guidance on creating the “.razor” pages and their corresponding CSS/SCSS, as I had requested. The AI did a great job with the full-screen background image/video styling and the navbar theme. While it was initially puzzled by my “Club Manager” admin functionality, I managed to clarify most of it with subsequent prompts. I had to manually tweak a fair amount around the networking side. But, on the UI side of the Back Office page form, ChatGPT even implemented a 5-second delay to display the text confirming my save succeeded as I requested.</p><p>The messy part involved numerous incorrect references, specifically around the Identity libraries. There were also several updates to files like “App.razor,” and custom implementations of “AuthenticationStateProvider.” Things became overly complicated pretty quick. ChatGPT appeared uncertain about the files generated by the initial “dotnet new” commands on the built-in ASP.NET Web API template and seemed to overlook my explicit mention of “<a href="https://learn.microsoft.com/en-us/dotnet/csharp/whats-new/tutorials/top-level-statements">Top Level Statements</a>” in Program.cs. Consequently, I had to carry out substantial refactoring, and ChatGPT offered numerous apologies with new suggestions. Some of these worked, some didn’t.</p><p>In short, no it didn’t give me 100% of all the code I needed to accomplish my full ask, but it probably gave me about 80%. It encouraged me to think more critically critical and taught me to ask better questions. At times, I felt like I relied less on my own problem-solving skills while tweaking ChatGPT’s responses than I would have when working on and debugging code myself. However, I’m still learning how to harness ChatGPT’s potential effectively.</p><p>My grand plan was to store my ChatGPT prompts in source control as an even higher level of “code”. However, I discovered that even when resetting the prompt context, the responses could vary dramatically, with different libraries, file names, and approaches to the same task. This unpredictability is both fascinating, maddening, and slightly terrifying, so I abandoned that idea.</p><p>I’m eager to hear if others have had similar experiences while working on complex projects and if anyone has discovered tips or tricks for obtaining better or more consistent results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdeb7e9251fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/building-an-oauth2-enabled-web-application-with-chatgpt-and-gpt-4-fdeb7e9251fa">Building an OAuth2 Enabled Web Application with ChatGPT and GPT 4</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ChatGPT Plus w/ GPT4-High Level Data Center to Azure Migration Proposal Experiment]]></title>
            <link>https://medium.com/eviden-data-science-community/chatgpt-plus-w-gpt4-high-level-data-center-to-azure-migration-proposal-experiment-a12faa71c78b?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/a12faa71c78b</guid>
            <category><![CDATA[work-productivity]]></category>
            <category><![CDATA[gpt-4]]></category>
            <category><![CDATA[technology-consulting]]></category>
            <category><![CDATA[ai-assistant]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Dave Arlin]]></dc:creator>
            <pubDate>Fri, 03 Nov 2023 19:54:09 GMT</pubDate>
            <atom:updated>2023-11-03T19:54:09.608Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ll be sharing an experiment I conducted using the capabilities of <a href="https://openai.com/blog/chatgpt-plus">ChatGPT Plus</a>, powered by the cutting-edge <a href="https://openai.com/research/gpt-4">GPT-4 AI model</a>. I’ll dive deeper into various intriguing use cases I’ve explored with this technology.</p><p>My curiosity was piqued when I discovered that <a href="https://openai.com/blog/chatgpt">ChatGPT</a> can adopt a multitude of personas. So, I put it to the test as an “Azure Cloud Architect AI Assistant” and observed its responses. It’s interesting to note the potential variation in its behavior depending on the initial instructions provided.</p><p>One standout feature of ChatGPT that I truly appreciate is its ability to store chat and context history from previous conversations. This allows me to pick up a conversation right where I left off, be it a week or a month later, with ChatGPT retaining the entire thread for seamless continuity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7GuVVOVz3Qtt6llkuGj_DA.png" /><figcaption>Showcasing the ability to resume previous context history from previous conversations</figcaption></figure><p>Don’t be too fixated on the “Data Warehouse Dating Scene.” I was trying to coax ChatGPT into crafting a witty data architecture joke involving Snowflake, BigQuery, Redshift, and Synapse. And yes, it can attempt humor as well!</p><h3>DataCenter to Azure Migration Proposal Accelerator</h3><p>After introducing ChatGPT as an “Azure Cloud Architect AI Assistant,” I proceeded to describe my current situation and requested a plan, architecture diagram, and Terraform code tailored to my needs. Admittedly, ChatGPT didn’t accomplish everything in my initial request. I eagerly await the day when I can effortlessly have it generate an initial Visio, Lucidchart, Draw.io (or any preferred tool) file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*FGIBBkuVY_IIv3Tz86DGYA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*zP1-BOnST-5-ipnLZjjT2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*vXDBzcKD_bhSsBH5guRT7Q.png" /><figcaption>My first multi-step command</figcaption></figure><p>Next, I simply asked it to create a DR plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*VrwhhXA_YK4JJA1PPVAiBA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*2jt62IzDb_ZYH7Pg63qGig.png" /></figure><p>Next, I informed ChatGPT that I work for a technology consulting firm (which is true) and requested that it create a potential staffing model based on this information, incorporating the <a href="https://www.scrum.org/learning-series/what-is-scrum">Agile Scrum</a> methodology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*Jmfuy62VBksRJTYgaNnHkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*HDKGbhpeiZGX85NdbtjjVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*R7s0ne2juJVEk5jnRMy3mA.png" /><figcaption>Not a bad first a take at a staffing model!</figcaption></figure><p>Finally, I tried to get it to (again) give me some kind of architecture diagram. I had seen from another <a href="https://www.linkedin.com/pulse/gpt-4-secret-weapon-cutting-edge-software-development-andy-abbott/">blog series</a> by my old friend / boss / technical mentor <a href="https://twitter.com/asabbott?lang=en">Andy Abbott</a> an example of this so knew it could do it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*iQp_fSc77l_GxPn9JtTD9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*VQbut7yKzGy89YjlphU_uA.png" /><figcaption>ASCII diagram for high-level reference architecture</figcaption></figure><p>Is the generated output more helpful than the summary provided earlier? Maybe not, but it’s intriguing to see that ChatGPT managed to create a visual representation of an architecture diagram nonetheless.</p><p>In conclusion, engaging in more in-depth conversations with ChatGPT can yield additional insights for crafting a “mock” proposal. While it doesn’t do all the work, and some editing may be necessary to add context, there’s no denying that it significantly accelerates such tasks. Moreover, it prompts me to consider potential questions and motivates me to learn more about products or topics I’m less familiar with. I strongly recommend exploring this tool to enhance your productivity and gain valuable insights.</p><p>For reference, I’m using the $20/month ChatGPT Plus subscription from OpenAI. Alternatively, you can experiment with the Bing chatbot, which is currently free. Although it doesn’t save conversations, you can retain them as long as your browser tab remains open. Depending on Microsoft’s ongoing rollout, <a href="https://www.bing.com/">Bing</a> users may have access to either the GPT-3.5 or GPT-4.0 model. I haven’t tried Google’s <a href="https://bard.google.com/">Bard</a> yet but am eager to see how it performs in the future.</p><p>I would love to hear feedback or stories from others who have experimented with ChatGPT or similar AI technologies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a12faa71c78b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/chatgpt-plus-w-gpt4-high-level-data-center-to-azure-migration-proposal-experiment-a12faa71c78b">ChatGPT Plus w/ GPT4-High Level Data Center to Azure Migration Proposal Experiment</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gen AI Series: Alternatives]]></title>
            <link>https://medium.com/eviden-data-science-community/gen-ai-series-alternatives-c71778bf029?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/c71778bf029</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Nicholas Beaudoin]]></dc:creator>
            <pubDate>Tue, 27 Jun 2023 18:45:01 GMT</pubDate>
            <atom:updated>2023-06-27T18:48:26.153Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is a continuation of the Gen AI series with particular attention to Large Language Models (LLM). This article discusses alternatives to Gen AI and the questions organizations should be asking before implementing a Gen AI capability. The first question organizations should ask themselves when evaluating ML solutions is “Are there simpler solutions available to me?” Like most organizations, you are probably getting bombarded with decks that have an image of a robot looking at a computer as “slide 1.” (Because nothing signals AI knowledge better than a pic of a Terminator-style robot. Please see the image below). These decks look savvy, dazzling, and full of incredible action-word jargon. But does your organization really need powerful LLM tools? Do you really need that proverbial Ferrari to run errands around town when a Honda Civic can get you to the grocery store in the same amount of time and with much less overhead?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*36T9Ak8WCTzmnZ05" /><figcaption>99% guarantee that a Gen AI slide deck will have a similar robot on “slide 1”. Notice the AI action words! <a href="https://www.analyticsvidhya.com/blog/2023/03/an-introduction-to-large-language-models-llms/">Source</a></figcaption></figure><p><strong>Why not “Gen AI All things?”</strong></p><p>LLMs such as Google’s <a href="https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html">PaLM</a>, Azure’s <a href="https://openai.com/blog/chatgpt">ChatGPT</a>, and AWS JumpStart’s use of <a href="https://cohere.com/">Cohere</a>, <a href="https://huggingface.co/">HuggingFace</a>, and <a href="https://www.ai21.com/studio?utm_source=google-search&amp;utm_medium=cpc&amp;utm_campaign=Studio_Search_US_EN_Singup_Branded&amp;ad_set_name=AI21_Studio&amp;utm_term=ai21%20labs&amp;gad=1&amp;gclid=CjwKCAjwkeqkBhAnEiwA5U-uM_yZ6ZEmmvOkeNQLIqw9gWTXgR_1o5aojvUKOHg-hAy1QvZ8DrEZgxoCEFAQAvD_BwE">AI21</a>’s models make using Gen AI appear straightforward and simple. Especially when using a managed service that allocates computed resources, ties in storage, seamlessly links IAM and security credentials, and lays template code to put pipelines in place. Managed services are fantastic! But managed services are expensive. If you are a smaller business and your use case can be deployed on a traditional ML solution, then it may be time to evaluate your options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*roMInYYvbaVwa6eS" /></figure><p>There are terrific resources for text extraction, classification with text inputs, and other solutions leveraging LLMs. For example, AWS offers a <a href="https://aws.amazon.com/bedrock/">Bedrock</a>, which is using an LLM under the hood to classify text. Time will tell if these models outperform more traditional document intelligence solutions like Optical Character Recognition (OCR) coupled with a classification layer using <a href="https://medium.com/@nick.s.beaudoin/searching-for-lost-nuclear-bombs-bayes-theorem-in-action-bf4671ea035">Naive Bayes</a> or a tree-based solution like <a href="https://xgboost.readthedocs.io/en/stable/">XGBoost</a>. Or if AWS’s <a href="https://aws.amazon.com/comprehend/">Comprehend</a> platform for document intelligence is a better solution to using AWS Bedrock. As companies adopt one or the other, more specific use cases will evolve to which platform, traditional ML or managed LLM services, are better suited.</p><p>These are the questions that organizations should be thinking about when deciding whether to leverage an LLM over traditional machine learning models:</p><ol><li>How risk-averse is your organization?</li><li>What is the model’s latency?</li><li>What is the marginal cost per inference call?</li><li>Understand the difference between inference for traditional ML vs LLMs</li><li>Is explainability important to you?</li><li>What is the scope of the AI application?</li><li>What is the time to value?</li></ol><p><strong>Can I solve this problem with a traditional ML approach?</strong></p><p>It is important to define why your organization is using machine learning to solve a business use case. In the world of machine learning, the risk is often associated with the degree one’s model gets answers wrong. This article is not geared to a deep dive into the trade-offs between recall and precision, but for those interested in these topics where false negatives and false positives drive decision-making, I highly recommend the blog series from Dr. Jason Brownlee on “<a href="https://machinelearningmastery.com/">Machine Learning Mastery</a>.”</p><p>If a model’s accuracy is paramount to your organization and are leading a risk-averse solution, such as making healthcare decisions that impact people’s lives, then your decision-making will be different than an organization that is more comfortable with risk. Companies that work in fraud detection for credit cards tend to be comfortable with a model that throws false positives, allowing the company to reach out to alert users where there is little downside other than an annoying text. Whereas companies that deal with HIPAA-related information, such as health insurance, will be more prone to avoid false positives that can have life-impacting results if a client is notified.</p><p><strong>What is the model’s latency?</strong></p><p>When evaluating ML solutions, how quickly do you need to get responses from the user? I recall from a client engagement in the entertainment industry, a VP saying that 5 seconds is an eternity for a user to wait for a dashboard to load. Websites that leverage HTML, CSS, and JavaScript can return an API call in nanoseconds, ensuring that the ML model’s inference time (the time it takes to send the user-generated input to the model weights, via API), calculates a prediction, and send it back to the frontend for the user. This takes time. Time makes the model look ineffective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*XrFwGCHSXmcGGcox" /><figcaption>Did someone say “latency”? <a href="https://variety.com/2016/film/reviews/zootopia-film-review-1201703504/">Source</a></figcaption></figure><p>More complex models require longer time at inference, thus longer wait times for clients. If the model is producing predictions in timed intervals (batch processing, aka offline inference) then this doesn’t matter. You can schedule the batch predictions to run during low-traffic periods such as automating actions during graveyard shifts. But if your predictions need to happen in real-time (online inference), then model latency matters. To further add to this dilemma, the longer the text output is in an LLM, the longer the wait for results. Again, more complex models take more time. Your organization needs to determine how time is factored into the user experience.</p><p>Currently, there is little information from companies developing LLMs as to how Service-Level Agreement (SLA) will be produced to give explicit promises to latency. At the time of this writing, <a href="https://huyenchip.com/2023/04/11/llm-engineering.html">OpenAI</a> has not provided an SLA for their GPT-powered models.</p><p><strong>What is the marginal cost per inference call?</strong></p><p>Each time the user calls a model’s API, it costs money. If you are hitting an API that is a simple ML algorithm, such as a tree-based model like random forest or XGBoost, then the cost is minimal. You simply have to take the model weights and perform a simple calculation to get the prediction. If you are using a clustering algorithm that needs to use the entire dataset at the point of inference (called a “greedy” algorithm), then your costs will be higher since you are scanning the dataset at each inference point.</p><p>If you are calling an LLM, then you could be paying as high as $0.25 for each inference call. Adding to that, with prompt chaining, many inference calls are a collection of lower-level API calls, acting in an additive manner to create one larger call to the model. Include prompt management for in-context learning, such as <a href="https://python.langchain.com/docs/get_started/introduction.html">LangChain</a>, throw in vector database storage using open-source solutions such as <a href="https://zilliz.com/what-is-milvus?utm_source=google&amp;utm_medium=cpc&amp;utm_campaign=WP_Milvus_Search&amp;utm_content=WP_Milvus_TKAG&amp;utm_term=milvus&amp;utm_campaign=WP_Milvus_Search&amp;utm_source=adwords&amp;utm_medium=ppc&amp;hsa_acc=3636806625&amp;hsa_cam=20033301076&amp;hsa_grp=151285141111&amp;hsa_ad=656191017493&amp;hsa_src=g&amp;hsa_tgt=kwd-2942447257&amp;hsa_kw=milvus&amp;hsa_mt=e&amp;hsa_net=adwords&amp;hsa_ver=3&amp;gad=1&amp;gclid=CjwKCAjwkeqkBhAnEiwA5U-uM5EfuqmUfG2iaiQxzzAFQLMk0JCR1i_YEdx4qyYqUraGwiZMK5E0nhoCDPkQAvD_BwE">Zilliz</a> (Milvus), and <a href="https://www.trychroma.com/">ChromaDB</a>, paid solutions such as <a href="https://www.pinecone.io/">PineCone</a> and <a href="https://www.deeplake.ai/">DeepLake</a>, or managed solutions like GCP <a href="https://cloud.google.com/vertex-ai/docs/matching-engine/overview">Matching Engine</a> and AWS’s <a href="https://aws.amazon.com/kendra/">Kendra</a>, then these all add up.</p><p><strong>Understand the difference between inference for traditional ML vs LLMs</strong></p><p>As previously stated, when a traditional classification ML model is pushed to production, the model weights are saved in a serialized file, such as a <a href="https://docs.python.org/3/library/pickle.html">pickle file</a>. These files are stored in a place that is easily accessible for the model to access and run in a reproducible manner, such as a <a href="https://www.docker.com/">Docker</a> container. When the model is called for prediction, the inputs go through a data transformation that allows the data to “talk” to the model weights. The model (algorithm) then multiplies those weights by the transformed user inputs. Once those calculations are complete, then the model sends those results back to the user interface, normally through a <a href="https://www.ibm.com/topics/rest-apis#:~:text=the%20next%20step-,What%20is%20a%20REST%20API%3F,representational%20state%20transfer%20architectural%20style.">RestAPI</a>. The cost for the inference call is straightforward: data transforms, weight calculations, and data ingress/egress fees.</p><p>Traditional supervised ML models have different behavior at the point of inference that can have different costs. When users call the API via a frontend interface, the data still needs to go through a transformation so the data inputs to the model can interact with the algo’s calculations. The divergence between classifications is there are no model weights sitting in a file for the model to hit. Furthermore, unsupervised ML models are known to be greedy, meaning they need to take all the training data, calculate the clusters/groupings, and then spit out a prediction. By scanning the data and recalibrating groupings, the new prediction takes more time, and thus, more cost. The predictions are sent back to the user interface through the API and presented to the user. These affect the cost of the prediction returned.</p><p>Large Language Models (LLMs) typically are trained for days, weeks, and months depending on the size of the training dataset. (For more information on training datasets used in LLM architecture, please refer to the following <a href="https://medium.com/@nick.s.beaudoin/large-language-models-101-a64847e17a3f">Medium</a> post). When the user inputs a prompt for prediction, the LLM has a few moving parts that diverge from traditional methods. The user prompt text is sent to an embedding model. This model takes the word strings and converts them to character tokens. These tokens are in the form of float numbers nested between 0 and 1. The embeddings are then compared to the vector database embeddings where the trained model artifacts are located. Often referred to as semantic space, the vector database holds embeddings in the form of vectors which can be queried to look for similarities by leveraging a clustering algorithm most notably <a href="https://ignite.apache.org/docs/latest/machine-learning/binary-classification/ann#:~:text=An%20approximate%20nearest%20neighbor%20search,good%20as%20the%20exact%20one.">Approximate Nearest Neighbors </a>(ANN). <em>Approximate</em> is used instead of explicit because scanning the entire vector database for exact matches would not be time and cost-efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QcozcJEofqz2MtRP" /><figcaption>Semantic space (vector DB) looks like it came from <em>Tron. </em><a href="https://twitter.com/Adarsh____gupta/status/1652543746364567554">Source</a></figcaption></figure><p>Once a match is found in this semantic space, a result is produced to the model. The model then decides which response to select based on a predicted probability score. If the user is chaining prompts together, such as through LangChain, the model will store these prompts as “context” to call in local memory, similar to how the vector database returns results. Once the model has returned predictions, they are returned to the user in the form of text. Since the LLM’s inference architecture is more complex and the training process more costly, the marginal cost of inference is much higher.</p><p><strong>Is explainability important to you?</strong></p><p>Some businesses are required to explain how models work, specifically those in the insurance and financial sectors. With new laws developing around both traditional and generative ML, model explainability is an area that many companies are keenly interested in. However, there are businesses that may not need explainability, particularly when it comes to decisions that don’t directly affect people’s lives. Model explainability becomes more difficult as model complexity increases, often convoluting the cost of model development through new explainability tools. This should be an area that the business should be aware of.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/0*Q5jPKUMycT7qwsfI" /><figcaption>Model explainability on some LLMs. <a href="https://pharmafactz.com/list-of-black-box-warnings/">Source</a></figcaption></figure><p><strong>What is the scope of the AI application?</strong></p><p>Bringing an LLM to production takes time, similar to a traditional machine learning solution. When done, it is just a minor part of a well-oiled process within an MLOps pipeline (machine learning operations). Building both traditional and Gen AI solutions within an MLOps infrastructure allows for ease of scaling, repeatability of results, fast iterations for data scientists to experiment, and the lowering of technical debt. However, MLOps take time. Organizations should understand the type of solution they want to build first. The following are product scopes that need to be determined before building the traditional ML or Gen AI solution. Needless to say, one should always have a well-defined process for adding features to the scope of the AI application to avoid scope creep.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*5Xfv0VVVouXCS9JE" /><figcaption>Maintain clear definitions of what the product will look like before the build phase to avoid scope creep. <a href="https://brainleaf.com/blog/management/what-is-scope-creep/">Source</a></figcaption></figure><ul><li><em>Proof of Concept:</em> The POC is a toy example of a working AI solution. Normally this runs in a local environment, such as on a data scientist’s local browser window, and lives in either a Jupyter Notebook or an undeployed snippet of code. The POC shows general functionality but will lack many of the features that are desired for organizational deployment.</li><li><em>Minimal Viable Product: </em>The MVP is a deployable version of the POC. The MVP should be used for gathering buy-in in an organization to sponsor the development of a fully-built solution that can be taken to production. Key elements in an MVP are the ability for users to interact with it in the organization’s cloud environment, local network, or VPC.</li><li><em>Soft Launch Product:</em> Soft launches are when a development team pre-selects a group of users to test the application in an organization’s environment. Sometimes referred to as the Quality Assurance (QA) portion of the development, the QA process is a great time to collect user feedback and improve on the application. The time this takes can range from days to months.</li><li><em>Deployed:</em> If following proper MLOps guidelines, the AI application should be deployable via a simple RestAPI. However, security and infrastructure teams need to be actively aiding in the deployment of the application since they will be the gatekeepers to future use.</li></ul><p>Needless to say, each of these steps takes time. There are areas that further complicate timelines by how involved the data science portion, infrastructure development, data engineering tasks, and organizational governance constraints are. These can include, but are not limited to, the following:</p><ul><li>Data engineering (ETL, processing, querying, prompt engineering, prompt templates, feature engineering)</li><li>Feature store curation</li><li>Modeling (traditional ML algorithms, fine-tuning LLMs, hyperparameter tuning)</li><li>Repository management</li><li>Container registration and orchestration</li><li>Model housing (model registration and tracking)</li><li>Model performance tracking</li><li>Governance templates</li><li>Infrastructure as Code (IaC) components</li><li>Pipeline orchestration</li><li>Security: API endpoint and VPC network (subnets per IPs)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*DCs7bWanH7VUCz-X" /><figcaption>Where does MLOps fit? <a href="https://neptune.ai/blog/mlops-tools-platforms-landscape">Source</a></figcaption></figure><p><strong>What is the time to value?</strong></p><p>Taking into consideration the previous scale of your organization’s AI application, how long does it need to run to yield results? Going further, how long will it take before those results yield value for the organization? Normally, time-to-value encompasses two areas:</p><ol><li>Time to train</li><li>Time to results</li></ol><p>Time to train refers to how long the model takes to complete its training and tuning operations and be ready for use. Time to results refers to how long the model needs to sit in production to gather user data and feedback, in the form of API calls, to provide business value. For example, when using cloud provider AI platforms, such as Google’s <a href="https://cloud.google.com/solutions/retail-product-discovery">Discovery Retail API</a> (a real-time recommendation engine), the AI app needs to be collecting user data for at least <a href="https://cloud.google.com/retail/docs/user-events">one month</a> to yield accurate predictions. Furthermore, the model needs to train for <a href="https://cloud.google.com/retail/docs/manage-models#:~:text=The%20Retail%20API%20uses%20the,continues%20to%20work%20as%20usual.">2 to 4 days</a> before being deployed. Many of these larger cloud-based models will require specific time frames to collect enough user data to make predictions. On the other hand, smaller ML models can be trained in minutes, but with far less accuracy and for smaller use cases. Depending on whether your organization is deploying a robust, managed cloud AI solution through AWS, Google, or Microsoft, or creating the infrastructure for deployment in-house, time-to-value calculations should be taken into account well before work begins.</p><p><strong>Last Word</strong></p><p>Deciding when and where to use Gen AI, particularly LLMs, is a large task for most organizations. Companies need to think about a variety of different factors when considering the gains from LLMs. While not all use cases are appropriate for LLMs, there are certain areas that shine when leveraging Gen AI. Most notably, those that involve language and text inputs. These areas are developing at a rapid pace and the benefits, often at scale, that are occurring in the textual space are at a logarithmic scale.</p><p>Special thanks to <a href="https://www.linkedin.com/in/anurag-bhatia/">Anurag Bhatia</a>, <a href="https://www.linkedin.com/in/steven-pais-4045483/">Steven Pais</a>, and <a href="https://www.linkedin.com/in/rishi-sheth10/">Rishi Sheth</a> for the technical feedback during the writing process.</p><p><em>Author’s note: This article was not written by Gen AI.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c71778bf029" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/gen-ai-series-alternatives-c71778bf029">Gen AI Series: Alternatives</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gen AI Series: Bias]]></title>
            <link>https://medium.com/eviden-data-science-community/gen-ai-series-bias-2964a2b4daa3?source=rss----fb6ceb507922---4</link>
            <guid isPermaLink="false">https://medium.com/p/2964a2b4daa3</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[bias-in-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nicholas Beaudoin]]></dc:creator>
            <pubDate>Mon, 19 Jun 2023 19:20:48 GMT</pubDate>
            <atom:updated>2023-06-13T16:56:10.170Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past few months, there have been many explanations of the pros and cons of generative AI (Gen AI). From grandiose visions of AI overlords harnessing the powers of SkyNet, to the less doomsday-like approaches of embracing the free market and letting innovation lead the way. The call for a 6-month moratorium on Gen AI has been at the forefront of our 24-hour news cycle. Comparisons to the Ford Model T affecting replacing the job of the poor horseshoe blacksmith come to mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mh7MB4hDDMef06P-vWYR_w.png" /><figcaption>The call for a recent 6-month pause has gained <a href="https://www.google.com/search?q=ai+6+month+pause+news&amp;rlz=1C5CHFA_enUS972US972&amp;hl=en&amp;tbm=isch&amp;sxsrf=APwXEdcr2Et5cnKvkJM55e_lIwKZTywe8Q:1686674151181&amp;source=lnms&amp;sa=X&amp;ved=2ahUKEwjT3Ift1sD_AhUFO0QIHeIdA6YQ0pQJegQIBhAI&amp;biw=1440&amp;bih=718&amp;dpr=2#imgrc=da6fphHGE_3pMM">momentum</a></figcaption></figure><p>However, the foray into a complete AI narrative is not monolithic, it does not pervade our global narrative as a fight between good and evil. Depending on one’s technical, economic, and political leanings, there seems to be a narrative that fits each flavor of thought. Regulate, demote, stop, pause, inspect. These are all partial solutions to a much more complex problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fhK1-MZhfohB_xV1TFgiJQ.png" /><figcaption>Gen AI is not a go or no-go question, there is nuance and dialogue needed in the <a href="https://www.youtube.com/watch?v=BY9KV8uCtj4">AI community</a></figcaption></figure><p>Gen AI is exciting and companies that are implementing it will no doubt provide a better and more timely service to their customers. This article is not about the benefits that any individual company seeks to gain, but rather, a community-level dialogue to the areas where Gen AI is apt to bring debate. The following areas dive into eight sections that I label <strong>BADASSES</strong>. These include:</p><ul><li><strong>B</strong>ias</li><li><strong>A</strong>lternatives</li><li><strong>D</strong>isinformation</li><li><strong>A</strong>ttribution</li><li><strong>S</strong>ecurity</li><li><strong>S</strong>iloing</li><li><strong>E</strong>xplainability</li><li><strong>S</strong>ensationalism</li></ul><p>As we learn more about the new technologies that present themselves each day, the new models, architecture, and infrastructure landscapes, understanding these 8 areas will become increasingly important for companies that are looking to leverage Gen AI gains. Various domains will touch upon different areas of these threads. Those Gen AI solutions directly affect people’s lives, regardless of the domain they fall into. The government domain (politics and national security) will worry more about disinformation, while academia will need to be more aware of attribution and sensationalism. My hope is that by touching these 8 threads, a more coherent understanding of Gen AI Ms in modern society will become more clear.</p><p><strong>Let’s Dig In!</strong></p><h3><strong>Bias</strong></h3><p>Bias is present in all elements of society. Our personal preferences and past experiences culminate into predisposed notions of what is correct. These preferences make their way into the types of news we follow, the clothing we wear, the information we absorb through social media, and the thoughts we have on how society should look and sound. These collective preferences have created the modern internet, embodied in the text and images that are stored on websites. Data is collected at a mass scale for the training of Gen AI solutions (see the ChatGPT model summary later). It is these preferences that are embodied in the Gen AI models and are returned as bias.</p><p>Large-scale data collection is nothing new. We have heard for years of the National Security Agency’s (NSA) ability to collect and store data. 10 years ago, it was reported that the NSA “touches” over <a href="https://medium.com/r?url=https%3A%2F%2Farstechnica.com%2Finformation-technology%2F2013%2F08%2Fthe-1-6-percent-of-the-internet-that-nsa-touches-is-bigger-than-it-seems%2F%23%3A%7E%3Atext%3DWhile%2520the%2520NSA%2520may%2520%2522touch%2Ctap%2520points%2520to%2520do%2520so.">29 petabytes of data daily</a>. In fact, their Utah data center is <a href="https://nsa.gov1.info/utah-data-center/">reported</a> to be able to store a yottabyte (one billion trillion megabytes). One can only assume that has increased at a logarithmic pace. In the private sector, most companies that have high processing capabilities collect data at an astonishing higher rate than before. The world’s largest retailer (WalMart), with over 20,000 stores worldwide, generates <a href="https://www.forbes.com/sites/bernardmarr/2017/01/23/really-big-data-at-walmart-real-time-insights-from-their-40-petabyte-data-cloud/?sh=161e0f366c10">2.5 petabytes of data</a> each hour! Data-centric companies are using data at a rate never before seen in historical records. So how does large-scale data collection affect the development of Gen AI?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BbbNRWwQUlm6YbmS" /><figcaption>National Security Administration <a href="https://medium.com/r?url=https%3A%2F%2Fnsa.gov1.info%2Futah-data-center%2F">Utah Data Center</a></figcaption></figure><p>As a technical background, ChatGPT-3 leverages Common Crawl, a 321 terabyte (TB) collection of raw text from the internet. However, this data tends to be skewed towards <a href="https://stanford-cs324.github.io/winter2022/lectures/data/">younger users</a> from developed countries. GPT-2 relied heavily on <a href="https://stanford-cs324.github.io/winter2022/lectures/data/">Reddit data</a>, which according to a <a href="https://dl.acm.org/doi/pdf/10.1145/3442188.3445922">2016 Pew Internet Research</a> survey is 67 percent men of which 64 percent of those men are between the ages of 18 and 29. The GPT framework utilizes Wikipedia, where only 15 percent of contributors are female. Finally, WebText, another key input to the GPT models, is trained on news sources, Wikipedia, and sources of fiction. A recent study by <a href="https://arxiv.org/pdf/2009.11462.pdf">Gilmen et al. 2020</a> found that a sampling of this text had a toxicity score of equal to or more than <a href="https://arxiv.org/pdf/2009.11462.pdf">50 percent</a>. All this is important because it lays the foundation for how we view bias and the data it lives.</p><p>The issue of bias in ML is not anything new. Take for example Tay, a fun-loving Bot that Microsoft released in 2016. Tay was trained on public Twitter posts. Tay was supposed to emulate the nature of society and how we interact with Twitter. It took less than 24 hours for Tay to be a fun, upbeat friend who sends you cool tweets to a racist, Fascist-loving Bot from hell. (see below for how it began and how it <a href="https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist">ended</a>). Tay taught the ML community that training data matters, and simply throwing the eternity of the public internet at an algorithm is bound to find the worst corners of it to represent if left tweaked by engineers.</p><p>We have seen this story play out, from Microsoft’s unwittingly <a href="https://towardsdatascience.com/racist-data-human-bias-is-infecting-ai-development-8110c1ec50c">fumble</a> with Tay. <a href="https://artsandculture.google.com/story/joy-buolamwini-examining-racial-and-gender-bias-in-facial-analysis-software-barbican-centre/BQWBaNKAVWQPJg?hl=en">Dr. Joy Buolamwini</a> of MIT has done tremendous research and education in this field, promoting a more inclusive use of AI technologies. Frequently bringing to light the inherent racial bias that ML systems contain. What does it mean when AI systems from leading companies fail to correctly classify the faces of Oprah Winfrey, Michelle Obama, and Serena Williams but can classify white celebrities with <a href="https://time.com/5520558/artificial-intelligence-racial-gender-bias/">shocking precision</a>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oF4YgwdfksgL-qWuhfIGIA.png" /><figcaption><a href="https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist">Tay</a></figcaption></figure><p>Bloomberg recently released a <a href="https://www.bloomberg.com/graphics/2023-generative-ai-bias/?leadSource=uverify%20wall">report</a> that took a deep-dive approach to StabilityAI’sgenerative AI text-to-image creation, <a href="https://stability.ai/stablediffusion">Stable Diffusion</a>. The one-sentence summary of the inherent biases is telling:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0auN_Uh_MnxpFL4E" /></figure><blockquote><em>“The world according to Stable Diffusion is run by White male CEOs. Women are rarely doctors, lawyers or judges. Men with dark skin commit crimes, while women with dark skin flip burgers.”</em></blockquote><p>The Bloomberg <a href="https://www.bloomberg.com/graphics/2023-generative-ai-bias/?leadSource=uverify%20wall">report</a> analyzed 5,000 AI-generated faces that represent 16 different types of jobs, from janitor to CEO. They compared these images using a skin-tone index that allows for six different ratings (Fitzpatrick Skin Scale), from light to dark. These ratings were compared across industries to the statistics reported by the US Bureau of Labor Statistics. When compared to the actual distribution of assumed skin tones from US government statistics, the results are shocking to the level of bias in Stable Diffusion’s generated images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ato2vHhdPNfDwcAb" /></figure><blockquote><em>“When prompted to generate images of a “terrorist,” the model consistently rendered men with dark facial hair, often wearing head coverings — clearly leaning on stereotypes of Muslim men. According to a </em><a href="https://www.gao.gov/assets/gao-17-300.pdf"><em>2017 report</em></a><em> from the Government Accountability Office, radical Islamic extremists committed 23 deadly terrorist attacks on US soil since Sept. 11, 2001 — but far-right extremists, including White supremacists, committed nearly three times as many during the same time frame.”</em></blockquote><p>The bias doesn’t only affect ethnicity. As seen below, using the Bureau of Labor Statistics data, Stable Diffusion tends to skew its results to generate more women for lower social status jobs (dishwasher, cashier, housekeeper) and lean more towards men for positions that have higher education and social ladder requirements.</p><p>But there are solutions to this problem, as Dr. Buolamwini describes in her “Safe Face Pledge” whereby companies promise to evaluate the data they train models on for racial biases. For Gen AI to build on the work that pioneers in racial equality and justice movements have moved through society, they must address the data they train on. Take for example areas in large language models, a type of Gen AI. As evidenced by a recent <a href="https://arxiv.org/abs/2203.15556">Chinchilla paper</a>, tuning an LLM algorithm can only get you so far, the real improvement comes from more training data. For companies training LLMs to rely solely on the gigabytes they throw at LLMs won’t be enough. It will take a concerted effort to give these models the right data to improve their performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I0oMZQW-UHz_IEr8WeqT2w.png" /><figcaption>Wrong type of <a href="https://animals.sandiegozoo.org/animals/chinchilla">Chinchilla</a></figcaption></figure><p>What is the solution to overcoming this deluge of training data bias in the LLMs that are deployed today?</p><ol><li>Commit to ethical training data curation</li><li>Join open communities</li><li>Document dataset curation</li><li>Release training dataset samples to the public</li><li>Hold data in escrow accounts</li></ol><p><strong>Commit to ethical training data curation</strong></p><p>Ethical training data curation borrows from academic sampling methodologies frequently found in public opinion research and survey analytics. By leveraging bedrock sampling methodologies from leaders in the political science academic arena, notably Leslie Kish, Ph.D., a more robust sampling framework can be put in place to better represent underrepresented groups. The world of political opinion survey research has leveraged these methodologies for half a century to create accurate samples of the American electorate. By looking into the pioneering work of people like Stan Greenberg who was Bill Clinton’s pollster Stan Greenberg (or his organization Greenberg Quinlan Rosner) or Jan van Lohuizen and Fred Steeper, George W Bush’s <a href="https://www.brookings.edu/articles/words-vs-deeds-president-george-w-bush-and-polling/">pollsters</a>, we can borrow from methodologies that have proven successful in understanding the American electorate. These methodologies, as it comes to choosing data sampling allocations and distributions can help create a more equitable and accurate data representation. When combined with model parameters and data size recommendations from the Chinchilla paper, gains to LLM accuracy, capabilities, and costs can go hand-in-hand with a more diverse and ethically appropriate data collection.</p><p><strong>Join open communities</strong></p><p>Joining open communities where data collection ideas are shared is critical. Open collaboration in the 1980s for nuclear deterrence proved fruitful in helping overcome some of the largest stockpiling drawdowns in human history of fissile materials. A similar approach to open collaboration, even when competing in the traditional Prisoner’s Dilemma constraint where the optimal choice is to cheat, will help in curating better datasets. Open participation, open sharing of code (ie, Reagan’s “Trust but verify” approach) can lead to more robust training data sets.</p><p><strong>Document dataset curation</strong></p><p>Any software engineer or machine learning engineer knows that proper documentation of code is worth its “weight” in gold. The same holds for how data is curated. Just as DevOps and MLOps filled the gaps in software development and ML model infrastructure, respectively, so too does a new era of data documentation. This documentation should answer four key questions: where, when, how, and why. Where was the data collected, the duration of collection, the methods, such as filtering and permissions used, and why it was collected?</p><p><strong>Release training dataset samples to the public</strong></p><p>AI companies should share a random sample of training data for a proper audit of their processes. The algorithm for this sampling methodology should be an openly agreed upon consent that allows for the input of multiple leading, and independent, companies that can offer clarity and statistical best practices to sampling methodologies used. Furthermore, allowing these datasets to be evaluated by the open community such as Kaggle has helped engage machine learning practitioners.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/0*8Z_2p9qJlT2yMQzH" /></figure><p><strong>Hold data in “escrow” accounts</strong></p><p>AWS JumpStart recently launched LLM management and productionalization that allows data and model weights to the law in “escrow.” Similar to a buyer and seller putting their money in an escrow account to ensure the proper functioning of a home sale. Since users are concerned about companies training their future models on their input prompts, and companies not wanting their model weights <a href="https://www.theverge.com/2023/3/8/23629362/meta-ai-language-model-llama-leak-online-misuse">leaked to the public</a> (LLaMA from Facebook), this concept has gained traction. AWS allows the user to place their data on AWS-managed infrastructure while the model weights, for example, AI21 and Cohere model weights, are stored on a similar stored capacity on AWS. At the point of inference, the model weights are called in the escrow account and compared against the user input data. This allows the flexibility to use the model vehicle keeping the user data private and the model weights the IP of the company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5GiYwVdUeInxh4lQ" /></figure><p><a href="https://www.realtor.com/advice/buy/understand-the-basics-of-escrow/">Source</a></p><p><strong>Where do we go from here?</strong></p><p>Bias in data collection impacts LLMs at an astonishing rate. While there are specific actions that can be taken immediately, there is a regulation that is pending that will attempt to draft the future of Gen AI. Most notably in the EU and US Congress. Time will tell where Gen AI is headed, but if GDPR and CCPA are any indicators, more regulation is around the corner. In the meantime, there remain areas of additional exploration. The idea of human-in-the-loop actions that allow for more human involvement in the data curation process, stress-testing the datasets for undisclosed bias, and other creative means of data discovery in the model-building process will help organizations uncover new areas for growth.</p><p><strong>Author’s note</strong>: Special thanks goes out to Steven Pais, Rishi Sheth, and Claire Salling for their technical review and sound boarding as this draft moved from PROD to DEV :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2964a2b4daa3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eviden-data-science-community/gen-ai-series-bias-2964a2b4daa3">Gen AI Series: Bias</a> was originally published in <a href="https://medium.com/eviden-data-science-community">Eviden Data Science and Engineering Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>