<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Carolyn Saplicki on Medium]]></title>
        <description><![CDATA[Stories by Carolyn Saplicki on Medium]]></description>
        <link>https://medium.com/@c.saplicki?source=rss-ee180b77a57f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*s-Mvi81deEzdDeYOuDJAVg.jpeg</url>
            <title>Stories by Carolyn Saplicki on Medium</title>
            <link>https://medium.com/@c.saplicki?source=rss-ee180b77a57f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:21:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@c.saplicki/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Myth of Perfect AI: Why Real Value Starts With Imperfection]]></title>
            <link>https://medium.com/@c.saplicki/the-myth-of-perfect-ai-why-real-value-starts-with-imperfection-4361f5e39e6d?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4361f5e39e6d</guid>
            <category><![CDATA[adoption]]></category>
            <category><![CDATA[fear-of-failure]]></category>
            <category><![CDATA[value-creation]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[time-management]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Tue, 09 Dec 2025 16:28:24 GMT</pubDate>
            <atom:updated>2025-12-09T16:28:24.835Z</atom:updated>
            <content:encoded><![CDATA[<p>Here’s a truth I’ve learned after a year architecting AI solutions: most people don’t care how advanced your model is — they care how often it messes up. One mistake, and trust evaporates. And that’s often hard to reconcile when the manual processes they’ve relied on for decades are slow, inconsistent, and full of errors.</p><p>This isn’t a technical problem. It’s a psychological one. And if you want AI to stick, you have to shift end-user mindsets — not just fine-tune LLM weights.</p><p>As a Data Science Architect, I’ve spent the past year helping organizations adopt AI across messy, deeply human workflows. Leaders talk excitedly about innovation and transformation, but the real friction shows up with the end users — the people who actually live inside the process every day. Many expect AI to be flawless, almost magical, because that’s how it’s marketed. Yet their current workflows are anything but flawless.</p><h4>The Real Problem Isn’t Accuracy — It’s Time</h4><p>Most teams focus the AI conversation on accuracy: <em>How often does the model get it wrong? What if it fails on this one strange PDF?</em></p><p>End users think these are the questions they should be asking — not because they’re wrong, but because they don’t know how to frame the conversation around value or gain. But accuracy isn’t the real pain point. <strong>Time is.</strong></p><p>Take a simple example: extracting information from PDFs. I recently worked on a project where engineers spent an enormous amount of time doing exactly that — manually transferring data from PDFs into Excel. In that engagement, this single task consumed 60% of an engineer’s year. With AI assistance, that workload dropped to ~5% of their time.</p><p>That’s not incremental improvement. <strong>That’s transformation.</strong></p><p>And as <a href="https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai">McKinsey</a> puts it, “AI only makes an impact in the real world when enterprises adapt to the new capabilities that these technologies enable.” In other words, success isn’t just about model performance — it’s about redesigning the workflow around what AI makes possible, and accepting that the new workflow will come with new, but far smaller, imperfections.</p><h3>Adoption Happens When You Build With Users, Not For Them</h3><p>The latest <a href="https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai">State of AI report</a> shows a clear pattern: while AI usage is widespread, value capture isn’t. Companies are experimenting, piloting, dabbling. But few are unlocking enterprise-level impact.</p><p>Why? Because value only appears when humans actually <strong>use</strong> the tool.</p><p>And humans don’t adopt perfection — they adopt usefulness.</p><p>Here are the tips and tricks I recommend to help shift the conversation and truly get AI adoption:</p><ol><li>Bring users into brainstorming.</li><li>Ask what slows them down.</li><li>Watch how they <em>actually</em> do the work.</li><li>Let them co-design version 1.0.</li></ol><p>When users feel the tool is built <em>for them</em>, they tolerate imperfections. When the tool feels imposed, they reject anything less than flawless automation.</p><p>At my company, we always start with MVPs. We treat AI engagements not as the end state, but as the beginning — the first shared blueprint for version 2.0, 3.0, and beyond.</p><h4>Embrace Imperfection</h4><p>AI doesn’t have to be perfect to be transformative. It has to be <em>adopted</em>. And adoption comes from shifting the frame away from fear — fear of errors, fear of change, fear of new workflows — and toward value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4361f5e39e6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gemini 4 Prompt Engineering!]]></title>
            <link>https://medium.com/@c.saplicki/gemini-4-prompt-engineering-c232fa62b856?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c232fa62b856</guid>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[semantics]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[refinement]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Mon, 16 Dec 2024 15:47:16 GMT</pubDate>
            <atom:updated>2024-12-16T16:02:51.498Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s no secret that prompts greatly change output with respect to GenAI projects. Using different words or phrases like try to or ensure can vastly change the output of a LLM. Rhetoric plays a vital role when crafting a prompt, especially in a GenAI system where structured or consistent output is desired.</p><p>Recently, I worked on a challenging GenAI project centered around code transformation, aiming to expedite developers’ workflow during this process. To understand how to tackle this problem using GenAI, my team and I needed to understand developers’ approaches to code transformations. Our team regularly met with developers to hear how they approach code transformations. In this context, a developer is someone who is hands-on-keyboard editing the code to generate a transformation. This knowledge proved invaluable in comprehending the steps involved in code transformation, a highly subjective task. For instance, a developer could start editing layout or logic.</p><p>Drawing on the developers’ expertise, I crafted an initial prompt that resonated with me. However, the output I received was horrendous. Instead of improving the code, the AI produced something even worse. How could this be? I was using the developers secret recipe to create code in a beautifully written prompt.</p><p>After rewriting, editing, and retrying my prompt with no improvement, I started to wonder if there was a better way. Could Gemini help me? Or maybe I was tired from rephrasing the same instructions over and over again…</p><p>I fed my prompt into Gemini and first asked where it was confusing. I obtained results like the order of operations and specific word choices were vague. This initial feedback gave me a fresh perspective.</p><p>By feeding my prompt into a large language model like Gemini, I gained clarity on the model’s interpretation of the task and its recommended steps. The feedback was surprisingly diverse, ranging from suggestions for alternative wording to a blunt “omit this task” or a simple “looks great!” This exercise provided invaluable insights into the nuances of communicating effectively with large language models.</p><p>These interactions remind me of the fundamental differences in how humans and AI models process information and approach tasks. While we might intuitively grasp complex or ambiguous instructions, AI models require explicit and precise directions. They also reinforce my belief in the potential of AI to augment human tasks rather than fully replace them.</p><p>Try these prompts the next time you want to better understand your prompt or need help refining it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*sHVpGDg1c9sbqbP2moi1yQ.png" /></figure><ul><li><em>“Read over my prompt. What is the task behind my prompt? Is there anything that looks confusing?”</em></li><li><em>“What are the steps in this prompt?”</em></li><li><em>“Can you rephrase this prompt in a clearer way?”</em></li><li><em>“Are there any unnecessary or redundant parts in this prompt?”</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c232fa62b856" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Swift Sentiment Analysis]]></title>
            <link>https://medium.com/@c.saplicki/swift-sentiment-analysis-f93c06ef30a7?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f93c06ef30a7</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Thu, 18 Jul 2024 13:44:24 GMT</pubDate>
            <atom:updated>2024-07-18T13:51:02.937Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://en.wikipedia.org/wiki/Sentiment_analysis">Sentiment analysis</a> is a powerful tool for business to gain insights from text data. This can unlock understanding about customers emotions and opinions. By understanding these hidden insights, businesses can improve customer satisfaction, develop better products, and gain a competitive edge.</p><p>There are many tools to investigate sentiment analysis. In this blog, I’ll investigate two methods: Traditional NLP and Generative AI (GenAI) to produce sentiment scores. I use the first few lines from Taylor Swift’s song <a href="https://www.youtube.com/watch?v=sRxrwjOtIag"><em>All Too Well (10 Minute Version)</em></a>.</p><pre>I walked through the door with you, the air was cold<br>But somethin&#39; &#39;bout it felt like home somehow<br>And I left my scarf there at your sister&#39;s house<br>And you&#39;ve still got it in your drawer, even now</pre><p>Although song lyrics often have hidden layers of meaning, I’ll analyze these lines on a surface level to understand the sentiment by sentence. This can provide insights into the overall tone and potential emotions in the song.</p><h3><strong>NLP</strong></h3><p>To demonstrate sentiment analysis through a traditional NLP model, I use <a href="https://cloud.google.com/natural-language?hl=en">Google’s NLP API</a>. The NLP API does multiple things such as sentiment analysis, entity recognition, entity sentiment analysis, and other text annotations. Here, I use sentiment analysis capability through the <strong>analyzeSentiment</strong> method. I formatted the responses from this model into a table for readability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ujmiwwyRzj8uVyo9qVVJyA.png" /></figure><p>We will focus our analysis on <strong>sentiment score</strong>. Sentiment score is between -1.0 (negative sentiment) and 1.0 (positive sentiment). <a href="https://cloud.google.com/natural-language/docs/basics#interpreting_sentiment_analysis_values">Here</a> is more information on interpreting sentiment analysis values.</p><p>The overall document sentiment score is 0.139. These responses are reliable. The API cannot parse out sentiment by speaker, but this can be done with post-processing.</p><h3><strong>GenAI Approach</strong></h3><p>I utilize <a href="https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemini-1.5-pro-preview-0409?project=ai-ml-team-sandbox">Gemini 1.5 Pro</a> for sentiment analysis. Gemini 1.5 Pro is a foundation model that performs well at a variety of multimodal tasks such as visual understanding, classification, summarization, and creating content from image, audio and video.</p><h4>Prompt 1:</h4><blockquote>Create a json of the sentiment in each line.</blockquote><blockquote>Use this text:</blockquote><blockquote><a href="https://genius.com/24526500/Taylor-swift-all-too-well-10-minute-version-taylors-version-from-the-vault/I-walked-through-the-door-with-you-the-air-was-cold-but-somethin-bout-it-felt-like-home-somehow">I walked through the door with you, the air was cold</a></blockquote><blockquote><a href="https://genius.com/24526500/Taylor-swift-all-too-well-10-minute-version-taylors-version-from-the-vault/I-walked-through-the-door-with-you-the-air-was-cold-but-somethin-bout-it-felt-like-home-somehow">But somethin’ ‘bout it felt like home somehow</a></blockquote><blockquote>And I left my scarf there at your sister’s house</blockquote><blockquote>And you’ve still got it in your drawer, even now</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hdoA1OiVzmEIrc1Kkdurgg.png" /></figure><p>This prompt does not result in a sentiment analysis that is as granular as desired (numeric values). Let’s try a similar prompt and specify a scale for our sentiment.</p><h4>Prompt 2:</h4><blockquote>Create a json of the sentiment in each line. <strong>Sentiment ranges from [-1,1].</strong></blockquote><blockquote>Use this text:</blockquote><blockquote><a href="https://genius.com/24526500/Taylor-swift-all-too-well-10-minute-version-taylors-version-from-the-vault/I-walked-through-the-door-with-you-the-air-was-cold-but-somethin-bout-it-felt-like-home-somehow">I walked through the door with you, the air was cold</a></blockquote><blockquote><a href="https://genius.com/24526500/Taylor-swift-all-too-well-10-minute-version-taylors-version-from-the-vault/I-walked-through-the-door-with-you-the-air-was-cold-but-somethin-bout-it-felt-like-home-somehow">But somethin’ ‘bout it felt like home somehow</a></blockquote><blockquote><a href="https://genius.com/24388439/Taylor-swift-all-too-well-10-minute-version-taylors-version-from-the-vault/And-i-left-my-scarf-there-at-your-sisters-house-and-youve-still-got-it-in-your-drawer-even-now">And I left my scarf there at your sister’s house</a></blockquote><blockquote><a href="https://genius.com/24388439/Taylor-swift-all-too-well-10-minute-version-taylors-version-from-the-vault/And-i-left-my-scarf-there-at-your-sisters-house-and-youve-still-got-it-in-your-drawer-even-now">And you’ve still got it in your drawer, even now</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NfTbnSbgtsetG913Ssu5mw.png" /></figure><p>I was pleasantly surprised at the output of the model. Each sentence has a sentiment score. With some post-processing, the overall document sentiment score is 0.25.</p><p><em>Note: Sentiment analysis using a generative AI model will need prompt tuning for desired sentiment score output (numeric, categorical) and output structure.</em></p><h3><strong>Comparison</strong></h3><p>I reformatted both outputs to be easily comparable. Here, we investigate sentiment score in particular.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rD0MGDZDTMeYQCPT6Qzgqg.png" /></figure><p>In general, the scores for these lines are not very different. For the most part, they are both positive or negative. If there is a difference in sign, it is almost negligible as these score show neutral sentiment. The scores from the NLP API seem much more precise. More nuances may be captured with the NLP API.</p><h3>Song Sentiments...</h3><p>In the first four lines, we are already taken on a ride of the singer’s sentiment, showing emotional shifts. This rapid changing from negative to positive sentiment highlights nostalgia. Sentiment analysis allows us to delve even deeper, analyzing each sentence to gain a more nuanced understanding of the song’s emotional core. The graph below utilizes the NLP scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*o2QrR7dctsdpIZG6nFyWrg.png" /><figcaption>NLP Scores</figcaption></figure><h3>Model Sentiments…</h3><p>For high-level sentiment insights, GenAI is a great first step to investigate documents. However, I am more inclined to continue to use NLP methods for analysis of text with respect to sentiment. The NLP model is more precise and consistent. For instance, with the NLP model, the responses will always be the same. GenAI models have the ability to change responses. This makes it more difficult to consistently compare sentiment across different tasks.</p><p>Additionally, we can use NLP API documentation to answer question as to the integrity of the model and output. With respect to the GenAI output, our initial prompt only reported sentiment in 3 buckets: positive, neutral and negative. Our second prompt response had numeric values; however, it is unclear how these numbers are crafted. I am not as confident in the ability to compare the results of sentiment analysis across multiple documents for granular insights.</p><h3><strong>Conclusion</strong></h3><p>Overall, I am excited by all the advances in GenAI; however, I am a strong believer in traditional AI techniques. In my example, NLP API provides great sentence by sentence sentiment analysis that is easy to understand and has a great baseline for other downstream tasks. For true, intricate sentiment understanding, I’d recommend using a dedicated sentiment analysis model as opposed to a GenAI model where sentiment may be inconsistent. For quick sentiment insights, I would use GenAI at a high level (positive, neutral, negative).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f93c06ef30a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consistency Is Key!]]></title>
            <link>https://medium.com/@c.saplicki/consistency-is-key-df4247fb89a6?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/df4247fb89a6</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[transcription]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[cosine-similarity]]></category>
            <category><![CDATA[consistency]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Tue, 25 Jun 2024 13:25:19 GMT</pubDate>
            <atom:updated>2024-06-25T18:39:00.708Z</atom:updated>
            <content:encoded><![CDATA[<p>When working on projects using Generative AI (GenAI), it’s important to consider how generated content is being evaluated. Often, a multifaceted solution is needed for comprehensive evaluation. These can include traditional statistical methods and incorporating GenAI models. Recently, I worked on productionizing audio-to-text transcriptions. In order to deliver a truly comprehensive solution, I investigated the evaluation method often referred to as consistency. This evaluation method was one of a few monitoring methods.</p><p>In this blog, you’ll learn about use cases for speech-to-text APIs and the evaluation of these transcriptions. I’ll also briefly touch upon the need for methods outside of consistency.</p><h3><strong>Use Case</strong></h3><p>Just last week, I had to call a retailer to learn about how to return an oversized item. Luckily, they had home pick-up. This customer service call, like many, was recorded. This is often for legal compliance or training purposes.</p><p>Suppose the audio from my call was transcribed into text. This text or transcript could be utilized for easier reference, legal compliance, improved training, and data analysis. Take my specific example. The retailer could use transcripts from all their callers to gain insights into customer needs, agent performance and common issues. This data can be used to improve products and develop internal training.</p><h3><strong>Models</strong></h3><p>Now that we understand the value of transcribing phone calls, let’s talk about some of the models we might use for transcription. With the GenAI boom, there are many models that we can utilize when it comes to transcription. I have investigated quite a few.</p><p><strong>Speech-to-Text APIs</strong>: A Speech-to-Text API is a tool that can enable the conversion of spoken audio into written text.</p><ul><li><a href="https://cloud.google.com/speech-to-text?hl=en">Googles Chirp</a>: Google Chirp is a specific type of Speech-to-Text API that understands over 100 languages. While still under development, it offers high accuracy for speech recognition.</li><li><a href="https://openai.com/index/whisper/">OpenAI Whisper</a>: OpenAI Whisper is a Speech-to-Text API that understands English and other languages. Whisper has high accuracy and is open-source.</li></ul><p><strong>Speaker Diarization</strong>: Speaker diarization is the process of automatically figuring out who is speaking in an audio recording.</p><ul><li><a href="https://github.com/pyannote/pyannote-audio">PyAnnote</a>: Pyannote is an open-source Python toolkit for speaker diarization. It uses deep learning to identify who is speaking in an audio recording, and offers pre-trained models or customization options. The downside of this model is that it subsequently needs a speech-to-text model. Pyannote has a pipeline framework to enable transcription with speaker diarization.</li></ul><p><strong>Generative AI</strong>: Generative AI is a type of artificial intelligence that can create new data, like text, images, or even code. There are many GenAI models. For this blog, we’ll just investigate Google’s Gemini models.</p><ul><li><a href="https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemini-1.5-pro-preview-0409?project=ai-ml-team-sandbox">Gemini 1.5 Pro</a>: Gemini 1.5 Pro is a foundation model that performs well at a variety of multimodal tasks such as visual understanding, classification, summarization, and creating content from image, audio and video.</li><li><a href="https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemini-1.5-flash-preview-0514?project=ai-ml-team-sandbox">Gemini 1.5 Flash</a>: Gemini Flash 1.5 is a variant of Google’s large language model, Gemini, specifically designed for speed and efficiency. It is designed for high-volume, high-frequency tasks where cost and latency matter.</li></ul><p>Overall, many models are suitable for different use cases. For ease of use and interpretability, I heavily worked with Gemini. For the purpose of this blog, I will be comparing Gemini 1.5 Pro and Gemini 1.5 Flash for speech-to-text transcription.</p><h3><strong>Transcriptions</strong></h3><p>In order to generate transcriptions, I utilized Google’s Gemini models. You can easily try these models in VertexAI Studio. I used the <a href="https://cloud.google.com/python/docs/reference/aiplatform/latest/vertexai">Python SDK</a>. I uploaded and called both Gemini models on a mock call between a customer and a customer service agent. The call was about 3 minutes long. I utilized the same prompt for each model.</p><p>Here is the prompt I used:</p><p><em>“Transcribe this audio file with speaker diarization, identifying the customer and agent speakers. Include timestamps at the beginning of each speaker change in the format [HH:MM:SS] SpeakerName: Text. After the initial transcription, please review the transcript to ensure all speaker changes are identified and formatted correctly.”</em></p><p>Here are the first few lines from each call:</p><p><strong>Gemini 1.5 Pro</strong></p><blockquote>[00:00:00] Agent: This call is now being recorded. Rachel Flowers, how may I help you?</blockquote><blockquote>[00:00:04] Customer: I bought flowers online for my wife, and it turns out they shipped the wrong colour.</blockquote><blockquote>[00:00:12] Agent: Oh I am so sorry sir.</blockquote><p><strong>Gemini 1.5 Flash</strong></p><blockquote>[00:00:00] Automated: This call is now being recorded.</blockquote><blockquote>[00:00:04] Automated: Rachel Flowers. How may I help you?</blockquote><blockquote>[00:00:09] Customer: I bought flowers online for my wife and it turns out they shipped the wrong color.</blockquote><blockquote>[00:00:18] Agent: Oh, I am so sorry sir.</blockquote><h3><strong>Evaluation</strong></h3><p>There are many ways to evaluate the transcription. Evaluation methods change based on the scale of your transcriptions.</p><h4>One Model: Few Transcripts</h4><p>Here, we can manually read the transcripts or compare a human-generated transcript to one created by a model.</p><ul><li><strong>Manual Read</strong>: One of the simplest ways to evaluate if your audio file was correctly transcribed is to read the text while listening to the audio. Here, you can get a good understanding of the accuracy of your model. This is, however, time intensive.</li><li><strong>Human-Generated Text</strong>: Another way you can understand and evaluate the accuracy of your transcription is to generate text of the audio file. Here, you can directly compare the text and use metrics like <a href="https://en.wikipedia.org/wiki/Word_error_rate">Word Error Rate (WER)</a> to evaluate the accuracy of the transcription. Again, this is time intensive and may not allow for discrepancies on how text is transcribed. For instance, suppose a speaker says “um” or stutters. This may not be captured by WER or direct comparison, leading to misleading accuracy scores. <em>Note: there are many other ways to compare texts outside of WER.</em></li></ul><p>These methods focused on investigating a transcription with respect to one model and few transcriptions. Human intervention is heavily needed.</p><h4>One Model: Many Transcripts</h4><p>To evaluate many transcripts, you can utilize other models with similar performance to ensure the transcripts have the same information from the initial audio. This method is called consistency.</p><ul><li><strong>Consistency</strong>: Consistency is a more complex way to understand and evaluate your transcriptions. Consistency compares transcriptions from different models through downstream tasks like embedding comparison. The general idea of consistency is when you have models that generate fairly accurate transcriptions, downstream tasks should result in similar results. When results differ, human intervention is needed to update models (or prompts).</li></ul><h3>Consistency Example</h3><p>Here are the general steps of consistency with the transcription example:</p><ol><li><strong>Transcribe the audio using multiple models.</strong></li></ol><p>In our example, I used Gemini 1.5 Pro and Gemini 1.5 Flash. See Transcriptions.</p><p><strong>2. Utilize downstream tasks to evaluate the similarity of the transcriptions.</strong></p><p>Here, I investigate <a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a> and a downstream GenAI task.</p><ul><li>For Cosine Similarity, I use t<a href="https://en.wikipedia.org/wiki/Word_embedding">ext embedding</a> (chunk each transcript and create a text embedding for each chunk) and then calculate cosine similarity for each chunk. If the cosine similarity is consistently high throughout the text, it suggests both models agree on the content, indicating a likely accurate transcription. If there are drops in cosine similarity, it highlights the need for a closer look.</li><li>For GenAI, using both transcripts, ask the same question(s) to a GenAI model and compare the responses. Assuming both models transcribed the audio well, they should have similar results.</li></ul><h4><strong>Consistency: Cosine Similarity</strong></h4><p>Here, I plotted the cosine similarity score for each embedded chunk of text</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KHWeAhsWfnsGtLly" /></figure><p>The average cosine similarity score is 0.898. This score indicates a high degree of similarity between the two transcribed texts.</p><p>What does this tell us? Both models were able to pull out similar transcriptions from the original audio file.</p><p>These metrics can be collected and monitored over time to understand your production model’s transcription integrity. Thresholds can be created and transcriptions can be flagged for further investigation.</p><p><em>Note: The transcripts above do have slight differences. These are why the cosine scores are not equal to 1. Additionally, the models I am comparing are Gemini 1.5 Pro and Gemini 1.5 Flash. It is not surprising the transcripts of this audio are similar.</em></p><h4><strong>Consistency: Downstream Models</strong></h4><p>To evaluate the similarity of generative AI responses, I utilized Gemini 1.5 Pro and the prompt <em>“Using this transcript, what color flowers did the man want?”.</em></p><p>Using both transcriptions I got the same answer:</p><blockquote>The man wanted red roses.</blockquote><p>What does this tell us? Both models were able to accurately pull out the same information from the original audio file. This highlights the quality of the original transcriptions.</p><p>These outputs can be collected and monitored over time to understand your production model’s transcription integrity. Discrepancies in downstream responses can be flagged for further investigation.</p><p><em>Note: Downstream tasks with GenAI models should incorporate questions that are simple and complex.</em></p><p>There are frameworks available to streamline this process such as <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/models/side-by-side-eval">AutoSxS</a>. AutoSxS is a tool within Vertex AI that compares two generative AI models’ responses to the same prompt. It uses an automated rater to determine which model generates a better response, similar to how a human would judge them.</p><h3><strong>Conclusion</strong></h3><p>Evaluating output from GenAI models is a difficult task. Often, you will have to use multiple approaches to ensure robust results. While consistency provides some great insights into how models are transcribing texts, it is not the only way to evaluate transcriptions. However, consistency can <em>consistently</em> allow for a quick understanding of whether vetted models are getting similar results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df4247fb89a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MAS AI/ML Modernization Accelerator: Air Compressor Use Case]]></title>
            <link>https://medium.com/ibm-data-ai/mas-ai-ml-modernization-accelerator-air-compressor-use-case-2dd05b27b800?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dd05b27b800</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[air-compressor]]></category>
            <category><![CDATA[industry]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[maximo-application-suite]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Tue, 09 Jan 2024 15:48:31 GMT</pubDate>
            <atom:updated>2024-01-09T16:51:19.611Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://medium.com/u/ee180b77a57f">Carolyn Saplicki</a><em>, IBM Data Scientist</em></p><p>Industries are constantly seeking innovative solutions to maximize efficiency, minimize downtime, and reduce costs. One groundbreaking technology that has emerged as a game-changer is <a href="https://www.ibm.com/products/maximo/asset-performance-management">asset performance management</a> (APM) <a href="https://www.ibm.com/topics/artificial-intelligence">artificial intelligence</a> (AI). However, embarking on the journey of implementing artificial intelligence (AI) in your asset performance management strategy can be both exciting and daunting. While the benefits of AI are widely recognized, getting started in this field comes with its own set of challenges.</p><p>Enter <a href="https://www.ibm.com/products/maximo">IBM Maximo® Application Suite</a> (MAS), a powerful set of applications used for asset monitoring, management, predictive maintenance, and reliability planning. MAS enables companies to anticipate equipment failures, mitigate risks, and optimize their maintenance processes. Many businesses are in different stages of their MAS AI/ML modernization journey. Recognizing this, we have created “on-ramps”, designed to simplify the process of integrating AI into maintenance operations on MAS. This solution can be found on RedHat Marketplace.</p><p>In this blog, we delve into 4 different “on-ramps” we created in a <a href="https://marketplace.redhat.com/en-us/products/ibm-maximo-detection-and-prediction-for-air-compressors">MAS Accelerator</a> to offer a straightforward path to harnessing the power of AI in MAS, wherever you may be on your MAS AI/ML modernization journey. These “on-ramps” are exploring ML/AI capabilities by leveraging out of the box models, migrating third-party models (custom), migrating third-party models (SROM-enhanced) and integrating third-party models. Each of these accelerators leverages state-of-the-art algorithms and machine learning techniques to identify anomalies accurately and in real-time.</p><h3><strong>Background</strong></h3><p>Our “on-ramps” accelerator focuses on industrial air compressors which are often considered the workhorse of a manufacturing plant. Industrial-grade air compressors are designed to provide a steady flow of compressed air for long periods of time and can take fluctuating surges in use typical in major manufacturing plants.</p><p>Air compressors are used for various purposes, such as powering pneumatic tools, providing compressed air for industrial processes, and supplying air for HVAC systems. Industries that use air compressors include manufacturing, automotive, construction, and energy.</p><p>The frameworks for each of these “on-ramps” can be utilized for other assets. All data scientists could leverage our patterns during an engagement. These patterns ensure consistency, efficiency, and collaboration among data science teams, making the MAS AI/ML modernization process smoother and scalable. We are leveraging Air Compressors data, but the solutions are generalizable. This means they can be applied to various industries and use cases beyond Air Compressors.</p><h3><strong>Maximo Application Suite</strong></h3><p><a href="https://www.ibm.com/products/maximo">IBM Maximo® Application Suite</a> (MAS) is a comprehensive application suite designed to empower businesses with efficient asset management and maintenance capabilities. With Maximo, businesses can effectively manage the entire lifecycle of their assets, from planning and procurement to maintenance and retirement. With its flexible architecture, scalability, and industry-specific solutions, MAS empowers businesses across sectors to achieve improved operational efficiency, cost savings, and enhanced asset reliability.</p><h3><strong>MAS AI/ML Modernization Accelerator “On-Ramps”</strong></h3><p>We have created a total of 4 different “on-ramps”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*g-1eiWPAdjmUkHf0x4qEEw.png" /><figcaption>MAS AI Modernization “On-Ramps”</figcaption></figure><h4><strong>Solution 1: Explore AI/ML in MAS</strong></h4><p>This data science solution predicts failure date, failure probability and anomalies in your assets. These problems can be solved by leveraging the out of the box (OOTB) solutions provided by MAS.</p><h4><strong>Solution 2: Migrate 3rd party models to MAS (Custom Model)</strong></h4><p>This data science solution predicts anomalies in air compressor assets using an <a href="https://en.wikipedia.org/wiki/Isolation_forest">isolation forest</a> model. An isolation forest model is an unsupervised machine learning algorithm used for anomaly detection. It isolates anomalies by randomly partitioning the data into subsets. The goal is to use air compressor features in the isolation forest model to detect when an anomaly occurs.</p><h4><strong>Solution 3: Migrate 3rd party models to MAS (SROM-Enhanced Model)</strong></h4><p>This data science solution predicts anomalies in air compressor assets using all variables through the Smarter Resource and Operations Management (SROM) toolkit. The SROM toolkit that is supplied with Maximo Production Optimization On-Premises provides a consistent programming model, a scalable execution model for model training, hyperparameter tuning (HPT) and the solution deployment, and the industry-specific templates and tooling for frequently occurring use cases.</p><p>Using SROM, we create multiple models and pick the best performing one to be deployed, which in this case is a One Class SVM model. Isolation Forest, Nearest Neighbor Anomaly Model, Anomaly Robust PCA, Neural Network NSA, Random Partition Forest, Gaussian Graphical Model are some of the other models that are considered by SROM pipeline to pick the best performing model. In this use case, we used the unsupervised machine learning approach to detect anomalies in the sensors data. This means that there is no need to have labelled data from your client.</p><h4><strong>Solution 4: Integrate 3rd party models with MAS</strong></h4><p>This data science solution predicts anomalies in air compressor assets using an external model. These models can be deployed on client’s on-prem or any cloud platform. Through Watson Studio, we create a Python wrapper function to get results from the deployed models and integrate the model within Watson Machine Learning.</p><h3><strong>Accelerator Steps</strong></h3><p>Each of our “on-ramps” consists of three key steps designed to help businesses thrive in their AI/ML modernization journey:</p><ul><li><strong>MAS Data Preparation</strong></li><li><strong>MAS Asset/Device Registration</strong></li><li><strong>Model Creation &amp; Connection</strong></li></ul><h4><strong>Step 1: MAS Data Preparation</strong></h4><p>The first step of this solution ingests the pre-processed data into MAS Monitor via the Watson IoT Platform, using a Python library called pmlib. In our scenario, the data is stored in the Cloud Object Storage in Watson Studio. However, in a real use case you could receive this data from third party DBs which could be connected directly to IoT Platform. Once, the sensors data is stored in the Data Lake (DB2Warehouse), this information is then passed on to MAS Manage in step 2 using the same pmlib library.</p><h4><strong>Step 2: MAS Asset/Device Registration</strong></h4><p>Step 2 is crucial to store information on failure history and installation dates etc. The main difference between MAS Monitor and MAS Manage is their function: MAS Monitor oversees the management of operational data (OT), instead MAS Manage will collect organization’s information (IT), making data available to business applications and users of those apps. While IT systems are the repositories and processors of data, their OT counterparts are responsible for generating the information processed by IT.</p><p>Once devices/assets are registered, organizations can leverage MAS’s features for tracking asset location, managing maintenance schedules, recording work history, monitoring asset health, and generating reports and analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*c0AZJCze6nZHKp-qcVKNBw.png" /><figcaption>MAS Monitor: Device Registration (Solution 1)</figcaption></figure><h4><strong>Step 3: Model Creation &amp; Connection</strong></h4><p>Finally, the third step, “Model Creation &amp; Connection,” focuses on creating an anomaly detection model and connected it to MAS.</p><ul><li>Solution 1: The final step leverages both Monitor and Manage data to create OOTB predict models (for this scenario we will only focus on Estimated time to failure, Failure probability and Anomaly detection) in Predict. The pipelines provided by pmlib have multiple parameters that can take values to suit your data and problem. These OOTB models are computed in Predict and subsequently deployed in Monitor. Thanks to the scheduler, the results can be displayed in both Health UI and Monitor UI.</li><li>Solution 2–3: In these solutions, we create and deploy a model or wrapper function in Watson Machine Learning. Once it is deployed, it is time to connect it to MAS. For this phase we need to prepare a wrapper function that through API calls score the model that we just deployed. This new wrapper function is then registered, deployed, and scheduled in Monitor.</li></ul><h3><strong>Final Solution</strong></h3><p>No matter where you are in your AI/ML modernization journey, using the accelerator, you will ultimately get these results in your MAS Health and Predict platform. By harnessing the power of the MAS, you will witness the manifestation of your models, offering invaluable insights into anomaly detection. These tools will empower reliability engineers to promptly identify when assets, such as air compressors, require intervention and repairs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*YE1yN9w-HC8-CgTqk2Ybig.png" /><figcaption>MAS Health &amp; Predict: Dashboard (Solution 1)</figcaption></figure><p>It’s important to note that while this example revolves around air compressors, the versatility of these notebooks extends to various other assets as well.</p><h3><strong>Conclusion</strong></h3><p>Regardless of where your organization is with respect to your MAS AI/ML modernization journey, the adoption and integration of AI-driven predictive maintenance is a game-changer, proactively addressing maintenance needs, minimizing downtime, and optimizing operational efficiency. However, the complexities associated with AI implementation can sometimes pose challenges.</p><p>That’s where our comprehensive <a href="https://marketplace.redhat.com/en-us/products/ibm-maximo-detection-and-prediction-for-air-compressors">MAS Accelerator </a>comes into play. Whether you are just starting out, looking to repurpose existing AI models, or seeking to optimize your current AI infrastructure, our solution provides the “on-ramps” you need to accelerate your time to market.</p><p>With <a href="https://www.ibm.com/products/cloud-pak-for-data">IBM Cloud Pak® for Data</a>, creating new AI models for predictive maintenance becomes an intuitive and efficient process. We simplify the model development journey, allowing you to customize AI algorithms to suit your specific use case.</p><p>Furthermore, our solution enables you to capitalize on the value of your existing AI investments. By repurposing and fine-tuning previous AI models, you can quickly apply them to predictive maintenance tasks without starting from scratch.</p><p>No matter where you find yourself on the path of APM modernization, our solution provides the tools, expertise, and support you need to accelerate your time to market. With our comprehensive approach, you can harness the transformative power of AI-driven predictive maintenance, unlock the full potential of your data, and drive operational excellence in today’s rapidly changing business landscape. Contact <a href="https://www.ibm.com/products/expertlabs">IBM Technology Expert Labs</a> today.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dd05b27b800" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/mas-ai-ml-modernization-accelerator-air-compressor-use-case-2dd05b27b800">MAS AI/ML Modernization Accelerator: Air Compressor Use Case</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[dynamite in R]]></title>
            <link>https://medium.com/ibm-data-ai/dynamite-in-r-a8251dfa0ab0?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8251dfa0ab0</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[panel-data-analysis]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Wed, 18 Oct 2023 15:39:17 GMT</pubDate>
            <atom:updated>2023-10-19T17:50:37.347Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Carolyn Saplicki, Data Scientist</em></p><p>I recently worked on a project that utilized panel data. Currently, many packages have been developed to handle panel data such as plm, fixest, and brms, among others. However, <a href="https://arxiv.org/pdf/2302.01607.pdf#:~:text=dynamite%20is%20an%20R%20package,multiple%20individuals%20measured%20in%20time."><em>dynamite</em></a> stands out by offering unique features that make it a unique for handling complex panel data.</p><p>As <em>dynamite</em> is relatively new, there were few examples of its use outside of documentation. In this blog post, I’ll dive into a basic example of <em>dynamite</em> using <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6964870/#:~:text=The%20PhysioNet%2FComputing%20in%20Cardiology%20Challenge%202019%20asked%20participants%20to,publicly%20and%20one%20remaining%20hidden.">PhysioNet Sepsis data</a>. Here, I’ll highlight <em>dynamite’s</em> easy to use formulas.</p><h3><strong>dynamite</strong></h3><p><em>dynamite</em> introduces the concept of Dynamic Multivariate Panel Models (DMPMs). These models, as implemented in the package, offer several advantages:</p><ul><li>Flexible Response Distributions: <em>dynamite</em> supports a variety of response variable distributions (Gaussian, Poisson, Bernoulli, and categorical distributions).</li><li>Smooth Time-Varying Effects: <em>dynamite</em> allow for the estimation of effects that vary over time, using Bayesian P-splines with random walk priors.</li><li>Multiple Simultaneous Measurements: <em>dynamite</em> can handle different number of measurements per individual, providing a solution for complex data.</li></ul><h3><strong>Sepsis Data</strong></h3><p>I began my analysis with sepsis data from <a href="https://www.kaggle.com/datasets/salikhussaini49/prediction-of-sepsis">Kaggle</a>, which consists of records for different patients over varying lengths of time (Hours). This data was similar to the project I had been working on as it is complex. Not only do patients vary in time-studied but also there was missing data.</p><p>To handle missing data, I implemented Last Observation Carried Forward, LOCF. This technique involves replacing missing values with the most recent observed value for each individual. After imputation, if any patient had missing values, they were excluded from the analysis.</p><p>One of the challenges in medical data analysis is dealing with varying lengths of patient stays. Often, padding is utilized to ensure that all patients have the same number of time points. However, <em>dynamite</em> simplifies this process, as it can handle data with varying time points. This is a <strong>significant</strong> advantage when working with real-world medical data where patient stays can differ significantly.</p><p>Lastly, I adjusted the dependent variable by shifting it up one time step for each patient in the dataset. This modification aligns the dependent variable’s values with the previous time step for all patients. This change eliminates the necessity for using lagged variables throughout the modeling process. To use lag variables, click <a href="https://docs.ropensci.org/dynamite/reference/lags.html">here</a>.</p><h3><strong>Define and Estimate Models</strong></h3><p>For this blog, I will focus on the White Blood Cell (WBC) variable to predict sepsis.</p><h4>Basic Model</h4><p>Here, I create a formula to model the binary sepsis status (SepsisLabel) and White Blood Cell count (WBC) within panel data. This formula utilizes a Bernoulli distribution to handle the binary outcome. Within the <em>dynamite’s</em> fit function, the model takes into consideration the time-based nature (Hours) of the data and the grouping of patients by their unique identifiers (Patient_ID). The model subsequently employs Bayesian inference via Markov Chain Monte Carlo (MCMC) sampling to estimate parameters.</p><pre>wbc_formula1 &lt;- obs(SepsisLabel ~ WBC, family = “bernoulli”)<br>print(wbc_formula1)</pre><pre>wbc_fit1 &lt;- dynamite(<br>  formula = wbc_formula1,<br>  data = test,<br>  time = &quot;Hour&quot;,<br>  group = &quot;Patient_ID&quot;,<br>  iter = 2000,<br>  chains = 4, cores = 4, refresh = 0<br>)</pre><h4>Time-Varying Model</h4><p>This formula looks very similar to the first; however, there are some key differences. Here, we’re introducing a varying effect for the predictor variable “WBC.” Additionally, the “-1” included in the model formula will create a time-varying intercept. This is especially helpful when the label variable is not consistent across all time frames.</p><p>To utilize WBC as a time-varying effect, we need a spline. The spline models how WBC changes over time, capturing non-linear patterns. In this case, the spline is designed with 10 degrees of freedom (df=10). The term noncentered = TRUE indicates the use of a non-centered parameterization for the spline. This approach is often preferred due to its ability to enhance the convergence and stability of the model.</p><pre>wbc_formula2 &lt;- obs(SepsisLabel ~ -1 + varying(~ WBC), family = &quot;bernoulli&quot;) + <br>                    splines(df=10, noncentered = TRUE)<br>print(wbc_formula2)</pre><pre>wbc_fit1 &lt;- dynamite(<br>  formula = wbc_formula2,<br>  data = test,<br>  time = &quot;Hour&quot;,<br>  group = &quot;Patient_ID&quot;,<br>  iter = 2000,<br>  chains = 4, cores = 4, refresh = 0<br>)</pre><h3>Results</h3><p>Note: For these results, I utilized fitted() with ndraws=1.</p><h4>Basic Model</h4><p>The ROC score is 0.63 for the training dataset and 0.63 for the validation dataset. Here, the model’s ability to discriminate between sepsis and non-sepsis cases is better than random chance but still has room for improvement. It indicates that the model has some predictive power, but it may not be capturing the full complexity of the relationship between the predictor variables and sepsis status.</p><p>Below, we can see the alpha and beta for this model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*98KPGb0WK9EkzKlX3hH38g.png" /></figure><h4>Varying Model</h4><p>In this model, the ROC score improved significantly to 0.76 for the training dataset and 0.76 for the validation dataset. This is notably better than the scores in the first model. It indicates that the model, with the inclusion of the varying parameter, is more effective at distinguishing sepsis from non-sepsis cases.</p><p>Below, we can see the alphas and deltas for this model. Here, we can see the varying WBC coefficient to capture how the effect of a predictor variable changes over time. Additionally, I added a time changing intercept. Here, we can see this value increases overtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*e08w8gQ8aTys4AATQ2iKzA.png" /></figure><h3><strong>Runtime Warnings</strong></h3><p>When introducing more variables to these models, I would face <a href="https://mc-stan.org/misc/warnings.html">runtime warnings</a> that often related to convergence issues. I found increasing iterations (iter) and warmup in the model fit helped to create more stable models. Additionally, there are many parameters that can be tuned in each of the formula building pieces. For instance, in splines with varying predictors, I added noncentered=TRUE. This creates a noncentered parameterization for the spline coefficients and is utilized when divergences are encountered.</p><h3>More Tools &amp; Resources</h3><h4>Tools</h4><p>There are many capabilities in this package such as group-specific random effects, latent factors, lagged effects, customized priors, summarized predictions, visualization tools, and support for various probability distributions. I suggest a read of the documentation.</p><h4>Resources</h4><ul><li><a href="https://ropensci.org/blog/2023/01/31/dynamite-r-package/">dynamite Prediction Example Code</a></li><li><a href="https://mc-stan.org/misc/warnings.html">Runtime warnings</a></li><li><a href="https://medium.com/@alex.pavlakis/making-predictions-from-stan-models-in-r-3e349dfac1ed">Predictions using Stan Models</a></li></ul><h3><strong>Conclusion</strong></h3><p><em>dynamite</em> is a game-changer for investigating panel data. Its ability to handle complex data, smoothly varying effects, diverse response distributions, and fully Bayesian estimation empowers researchers to unlock deeper insights from panel data. This R package is a valuable addition to the toolkit of anyone working with panel data in various fields, from social sciences to economics and beyond.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8251dfa0ab0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/dynamite-in-r-a8251dfa0ab0">dynamite in R</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shift the Focus: From Asset to Model Management]]></title>
            <link>https://medium.com/ibm-data-ai/shift-the-focus-from-asset-to-model-management-80b1b5c9425d?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/80b1b5c9425d</guid>
            <category><![CDATA[sustainability]]></category>
            <category><![CDATA[watson-openscale]]></category>
            <category><![CDATA[maximo-application-suite]]></category>
            <category><![CDATA[modelmanagement]]></category>
            <category><![CDATA[asset-management]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Fri, 16 Jun 2023 17:54:55 GMT</pubDate>
            <atom:updated>2023-07-17T13:42:58.611Z</atom:updated>
            <content:encoded><![CDATA[<p>By <a href="https://medium.com/u/ee180b77a57f">Carolyn Saplicki</a>, Senior Data Scientist, and <a href="https://medium.com/u/c40b81d1c699">Erika Agostinelli</a>, Senior Data Scientist</p><p>As industries begin to utilize AI in Sustainability practices, the focus, thus far, has been on Asset Performance Management (APM). In this process, statistical or machine learning (ML) models have been created to aid with predictive maintenance.</p><p>This means predictive maintenance depends on the health of your models. Therefore, to ensure the best performance of your assets, AI/ML models must have integrity and resiliency. In this blog, we will discuss how can we ensure our APM models are consistently functioning properly and who needs to ensure the credibility of our models. Additionally, we will highlight the importance of having a model management view in industry.</p><h3><strong>The Advancement of AI in Industry</strong></h3><p>AI can augment asset performance practices to increase utilization of assets and materials. In APM, AI models could include failure prediction, anomaly detection, end-of-life prediction and many others. Often, one asset may have multiple models. The benefits of using a model to aid with APM include transparency into operations, reduction in asset replacement cost and optimization of maintenance planning.</p><p>We would like to draw a comparison between the asset focus approach (manufacturing process) and model focus approach (ML lifecycle management process). Just as a manufacturing process is installed to ensure the quality and consistency of every manufactured good, a ML lifecycle management process must be appointed to ensure good quality and consistency of every deployed model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kc-ZNkk9iWDS6wU7_yW_gA.png" /><figcaption>Asset Focus &amp; Model Focus</figcaption></figure><p>We dissect each approach to find common elements such as process inputs, controlled variables, uncontrolled variables, and process outputs. This will highlight similarities, differences and the complementary nature of the two approaches.</p><h4>Table of contents:</h4><p><strong>Asset Focus — Manufacturing Control Process</strong></p><ul><li>Benefits of monitoring the Manufacturing Control Process</li><li>Example on Maximo Application Suite</li></ul><p><strong>Model Focus — ML Lifecycle Management Process</strong></p><ul><li>Benefits of monitoring the ML Lifecycle Process</li><li>Example on Watson OpenScale</li></ul><p><strong>Conclusions</strong></p><h3><strong>Asset Focus — Manufacturing Control Process</strong></h3><p>Problems in industry revolve around the manufacturing process. This process consists of using raw materials to create and output finished goods. For instance, in the automotive industry, raw materials such as steel and rubber and energy from fossil fuels are used to create a finished product, like a car tire. Below is an overview of the manufacturing process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*XJiocX3fGQiHtJPvi8LB4A.png" /></figure><p><strong>Process Inputs</strong>:</p><ul><li><strong>Materials</strong>: raw materials needed to create the manufacturing product</li><li><strong>Energy Sources</strong>: raw energy sources used to power the manufacturing process</li><li><strong>Human Intervention</strong>: human intervention needed to run the manufacturing process</li></ul><p><strong>Controlled variables</strong>:</p><ul><li><strong>Asset properties</strong>: asset properties that determine the functioning of the asset that are individually controlled for the specific process (machine settings)</li></ul><p><strong>Uncontrolled variables</strong>:</p><ul><li><strong>Environment</strong>: variables which we do not have any control, or would have limited control over, and may affect the manufacturing process production</li></ul><p><strong>Process Outputs</strong>:</p><ul><li><strong>Final good</strong>: final good that arises from the manufacturing process.</li><li><strong>Other output</strong>: I.E., Yield (proportion of good items), Waste (scrap items), Downtime (the period during which a machine or equipment is not functional)</li></ul><h4><strong>Benefits of monitoring the Manufacturing Control Process</strong></h4><p>With a manufacturing control process, operations managers and engineers can take immediate action on:</p><ol><li>reducing downtime and cost, making the APM more efficient</li><li>extension of the asset lifecycles through financial and performance analytics</li><li>optimization of maintenance work processes</li></ol><p>Together operations managers and engineers can support asset work orders using insights from anomaly detection, time to failure, and failure prediction models. Finally, this process can manage maintenance with work management for planned and unplanned activities.</p><h4><strong>Example</strong></h4><p><a href="https://www.ibm.com/products/maximo">IBM Maximo® Application Suite</a> (MAS) is a set of applications for asset monitoring, management, predictive maintenance, and reliability planning. Using IBM Maximo Predict, we use AI and <strong>process inputs</strong> (raw materials), <strong>controlled variables</strong> (IoT sensors data), <strong>uncontrolled variables</strong> (environmental data) and <strong>process outputs</strong> (final products) to predict downtime, degradation, and failures and to track imminent failures and maintenance schedules for oil wells.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hT3f5N4wwusPaYRAGYlIjw.png" /><figcaption>MAS Health and Predict</figcaption></figure><p>Based on our monitors, we can investigate assets and their respective predictive maintenance models. Through the manufacturing control process, operations engineers investigated:</p><p>a. current probability of different failure modes,</p><p>b. the number of days until a specific failure mode might occur, and</p><p>c. if detected anomalies signal a probable failure.</p><p>Specifically, <a href="https://www.ibm.com/products/maximo/predictive-maintenance">Maximo Predict</a> highlighted an anomaly threshold being violated. This could signal a probable failure. A work order was created, and an engineer investigated the asset to understand the anomaly. This monitor helped uncover a needed maintenance for this asset.</p><h3><strong>Model Focus — Machine Learning Lifecycle Management Process</strong></h3><p>The ML lifecycle management process can be compared to the manufacturing process. Just as a manufacturing process has inputs, variables and outputs, a model mirrors this dynamic. The model focus approach consists of using raw data as input to create predictions. For instance, in APM within the automotive industry, raw IoT sensors data is used to predict assembly line failures. Below is an overview of the machine learning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*nZtez-8-9o3GkYpJzWlHUQ.png" /></figure><h4><strong>Process Inputs</strong>:</h4><ul><li><strong>Payload Data</strong>: live sensor data to be scored utilizing the machine learning model (predictive model such as failure prediction, anomaly detection or end-of-life models).</li></ul><h4><strong>Controlled variables</strong>:</h4><ul><li><strong>Internal</strong>: variables internal to the model that are required for predictions such as hyperparameters. These can be estimated or learned from the data. These are usually created by Data Scientists.</li><li><strong>External</strong>: variables external to the model that are required for model understanding such as usage or monitor thresholds. These are usually determined by Subject Matter Experts (SMEs) and Data Scientists.</li></ul><h4><strong>Uncontrolled variables</strong>:</h4><ul><li><strong>Causal</strong>: variables part of the asset which is not functioning correctly (faulty sensor of the asset is recording unreliable data which may have an unwanted impact on the performance of a specific asset(s)).</li><li><strong>Non-Causal</strong>: variables not part of the asset which we do not have any control (limited control) over. For example, unexpected characteristics of the building (humidity) which is not captured by the model may have an unwanted impact on the performance of assets in different locations.</li></ul><p><strong>Process Outputs</strong>:</p><ul><li><strong>Predictions</strong>: final output of a model (anomaly score, time to failure prediction etc).</li><li><strong>Monitors</strong>: outputs from model monitors such as quality, drift, fairness and explainability providing insights into the overall model and final prediction.</li></ul><h4><strong>Benefits of monitoring the Machine Learning Process</strong></h4><p>With a machine learning lifecycle management process, machine learning engineers and data scientist can take immediate action on:</p><ol><li>ensuring the quality and performance of the predictive models for an efficient APM</li><li>maintaining resiliency of model predictions through root cause analysis and explainability</li><li>capturing if data drift or model drift is occurring</li></ol><p>Together, data scientists and operations can increase accuracy of predictions by identifying how AI is used and where it is lagging, enhancing collaboration across different teams. Finally, this process can mitigate potential performance issues with root cause analysis and model prediction explanations, supporting asset work orders.</p><h4><strong>Example</strong></h4><p>In our previous blog, <a href="https://medium.com/ibm-data-ai/ai-fairness-in-industry-13d44a3ab83e">AI Fairness in Industry</a>, we demonstrated the use of a fairness monitor in an industry related problem. We utilized model monitoring on our Prediction Maintenance Model by choosing a machine learning lifecycle management view. Specifically, we wanted to investigate if certain <strong>controlled variables</strong> (fairness thresholds) were being violated based on <strong>uncontrolled non-causal factors</strong> (regional humidity). We were able to quantify this bias using our <strong>process inputs</strong> (payload data; region) and <strong>process outputs</strong> (predictions and monitors).</p><p>To do this, we created a custom metric to uncover possible bias within our model. Using a fairness ratio, we were able to highlight whether the model is biased toward one region in certain asset conditions. In fact, based on historic data, the model learned that one region’s well was going to produce less faulty products than the other region on equal terms (i.e. despite having the same control variables). Below, you can see the fairness metric in <a href="https://www.ibm.com/docs/en/cloud-paks/cp-data/4.6.x?topic=services-watson-openscale">Watson OpenScale</a> over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qx5RQEi2PqBh55O65-zynA.png" /><figcaption>Watson OpenScale: Custom Monitor</figcaption></figure><p>Based on our monitor, we can initially see the metric is violating the lower threshold consistently until 12 PM. Through the machine learning process, Data Scientists, SMEs and Operations Engineers investigated the violations and saw if they could be explained by the model and/or mitigated through Machine Learning practices. After consistent violation, the source of the bias was found. The humidity of the building was causing unexpected results from our models. This monitor helped uncover a needed strategy change for this asset class.</p><h3><strong>Conclusion</strong></h3><p>It may seem like a machine learning lifecycle management process view with respect to your assets is not relevant at the industry level. However, model understanding is <strong><em>critical</em></strong> in the industry. We are living in an era where we rely on AI to make our assets more durable. It is important to care for those models as much as the assets and apply an approach which focuses on the ML lifecycle management.</p><p>The title of this blog “Shift the Focus: From Asset to Model Management” does not mean that assets (machines) are not at the heart of the APM. Instead, it highlights that a complementary strategy is needed to support the robustness of our AI/ML models. The combination of these two approaches will lead to more efficient APM decisions, revealing when strategy changes are necessary.</p><p>We recommend businesses to rethink their asset performance management strategy and include ML lifecycle management to optimize utilization of their assets and models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80b1b5c9425d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/shift-the-focus-from-asset-to-model-management-80b1b5c9425d">Shift the Focus: From Asset to Model Management</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Fairness in Industry]]></title>
            <link>https://medium.com/ibm-data-ai/ai-fairness-in-industry-13d44a3ab83e?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/13d44a3ab83e</guid>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[fairness]]></category>
            <category><![CDATA[industry]]></category>
            <category><![CDATA[sustainability]]></category>
            <category><![CDATA[watson-openscale]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Fri, 26 May 2023 13:06:07 GMT</pubDate>
            <atom:updated>2023-07-17T13:42:38.318Z</atom:updated>
            <content:encoded><![CDATA[<p>By <a href="https://medium.com/u/c40b81d1c699">Erika Agostinelli</a> (IBM Senior Data Scientist), <a href="https://medium.com/u/e9399da27be7">Stefan van der Stockt</a> (IBM Lead Data Scientist, Master Inventor) and <a href="https://medium.com/u/ee180b77a57f">Carolyn Saplicki</a> (IBM Senior Data Scientist)</p><p>Various industries are utilizing AI in Sustainability practices more to create models that aid with critical asset performance management (APM) initiatives. These models are applied to anything from condition-based monitoring, predictive maintenance, asset failure prediction, detecting anomalies in daily usage patterns, to predicting the end-of-life of assets. The business success of these initiatives is highly dependent on the veracity of the employed AI models.</p><p>AI models in industry track the operation of assets in the field. Models can be employed to perform tasks such as predicting when maintenance should be performed based on (variable) operating conditions. Models can track the conditions that are known to lead to asset degradation over time, and pre-emptively update asset end-of-life expectancies. Condition-based monitoring data could also be used by AI models to orchestrate and schedule the use of assets in a more optimal way to promote profits, extend asset life, and avoid unplanned down-time.</p><p>All AI models are only as good as the data and development lifecycle that yielded them. In a mature AI lifecycle, every deployed model is constantly monitored to ensure that reality still matches the assumptions (and data) that was used to train the model. Model monitors are employed to consistently check aspects such as input data drift, model quality (“<em>accuracy</em>”), model fairness, model explainability, and others. Model quality, drift, and explainability, while critical and involving sophisticated calculation by themselves, are relatively easy to relate to in terms of business goals for industry use cases. But what does <em>Fairness</em> mean<em> </em>in industrial use cases such as manufacturing a car or operating a field of oil wells?</p><h3>The Common Definition of AI Fairness</h3><p>In <a href="https://medium.com/ibm-data-ai/fairness-explained-definitions-and-metrics-9690f8e0a4ea">Fairness Explained: Definitions and Metrics</a>, fairness definitions and fairness metrics are presented in the context of a real-world example that predicts a criminal defendant’s likelihood of reoffending. In that scenario, we give a <em>fairness definition</em> based on <em>protected attributes</em> such as race, gender, or religion. We explore that there are numerous conflicting definitions of “fair” and the most appropriate mathematical formulation of a <em>fairness metric</em> is dependent on the use case itself. We utilize tools such as the <a href="https://arxiv.org/pdf/2102.08453.pdf">Fairness Compass</a> and <a href="https://aif360.mybluemix.net/">AI Fairness 360</a> to understand the existence of <em>bias</em> between <em>privileged (reference)</em> and <em>unprivileged (monitored) groups</em> and then quantify its effects on a model’s outcomes.</p><p>However, approaching the topic of fairness in Industry is not such a straightforward exercise:</p><ul><li>What is the definition of a <em>protected attribute</em> in any given industrial use case? Which groups are considered <em>privileged</em> and <em>unprivileged</em>? Why does the distinction even matter to the business?</li><li>What mathematical definition of “fairness” is appropriate to measure the disparity between the above groups?</li><li>What business objectives are supported by this <em>fairness definition</em> and supporting <em>fairness metric</em>? At what point (“threshold”) does any observed disparity between groups become a <em>bias</em> that needs to be addressed?</li><li>How should any such bias be <em>mitigated</em>? Does it involve retraining the model on new/different/additional data? Does it require physical changes to assets in the field? Does it require standard operating procedure or other process changes? Does it require a complete rethink of the AI method being used?</li></ul><p>These questions are not trivial and have far-reaching consequences. The answers are critical to reflect on when AI is used to avoid unplanned downtime or to decide when or when not to send technicians out to perform work on multi-million-dollar assets.</p><h3>The Definition of AI Fairness in Industry</h3><p>AI models learn from historical data and can expose which combinations of characteristics frequently lead to certain outcomes (such as failures). Subject-matter experts (SMEs) can inspect these relationships and patterns and can then make judgements on how well the model outcomes match the observed data.</p><p>A salient assumption in the above statement is that <strong><em>with all things being equal, all assets should be treated similarly by any AI model, regardless of an asset’s “non-causal demographics”. </em></strong>AI models typically use data such as sensor readings, weather data, service history, production volumes, etc. to reason about an asset’s condition. Such models rarely (if ever) use non-causal “demographic” data such as an asset’s randomly assigned group ID in the ERP system, proximity to by-passers on the factory floor, or other data items that seemingly do not actually aid in making predictions. Yet these data types might have measurable and profound effects on model outcomes that may not be detected by traditional data science techniques if they are not actively being looked for. Model monitoring could expose that model outcomes, directly or indirectly, may in fact driven by non-causal factors such as ERP groupings, location, or different supplier contracts.</p><p>A definition of <em>asset fairness</em> may be useful here. To get there we need to map fairness concepts to industrial use cases, for example:</p><ul><li><strong>Protected Attributes</strong>: A protected attribute should be defined that maps to business-sensitive “non-causal” data such as location or an arbitrary asset grouping. The criteria should be that any form of discrepancy between such groups should a) ideally not be present and b) if it DOES exist it should be deemed as negative to the business.</li><li><strong>Fairness Definition</strong>: With a defined protected attribute(s) we can now decide what “fair” means. This will inform us of what fairness metric to use (i.e. disparate impact, statistical parity difference, equality of odds, etc.).</li><li><strong>Thresholds</strong>: Once we have a definition and measure of fairness, we can define the critical thresholds above/below which any detected discrepancy becomes problematic.</li></ul><p>Defining the above concepts will allow us to detect situations in which the model is giving different ranges of positive/negative outcomes to different assets based on non-causal criteria (that is not even known to the model!). A data science team can now investigate why this is occurring and adapt the model accordingly.</p><h3>Why is it important to track potential bias in APM?</h3><p>Defining and monitoring the fairness of the ML models that you apply to your asset performance management (APM) strategy allows you to achieve greater insight into presence and magnitude of any detected bias. Ultimately, bias detection activities could uncover areas of improvement in the modelling approach that would otherwise not be detected. Strategy changes may also be needed to address any detected biases.</p><h3>How can we monitor AI Fairness?</h3><p>Utilizing SMEs’ knowledge and experience, you can create bias monitors to understand and quantify fairness. In this blog we used <a href="https://www.ibm.com/docs/en/cloud-paks/cp-data/4.6.x?topic=services-watson-openscale">Watson OpenScale</a>, a platform to oversee models lifecycle using quality, drift, fairness and explainability monitors. This enables data scientists to understand when and how a model is predicting failures in a biased way, allowing appropriate action to be taken. Learn more about APM model management here.</p><h3>AI Fairness Monitoring in Industry — An Example</h3><p>Using <a href="https://www.ibm.com/products/maximo">Maximo Application Suite</a> and <a href="https://www.ibm.com/products/cloud-pak-for-data">Cloud Pak for Data</a>, we created and deployed an Intelligent Maintenance model that predicts whether an asset will fail. This model utilizes dynamic data and static data. Dynamic data are sensor variables that come from the asset utilization, such as temperature and speed. Static data are stationary variables that come from the asset itself, such as age and location.</p><p>Using <a href="https://www.ibm.com/docs/en/cloud-paks/cp-data/4.6.x?topic=services-watson-openscale">Watson OpenScale</a>, we monitor the quality of the model using feedback data (ground truth data). We also monitor drift and explainability using payload data (scoring data). OpenScale offers out-of-the-box fairness metrics but for our use case, we decided to create a custom metric which allows to formulate more complex and tailored metrics. More information on custom monitors in Watson OpenScale can be found <a href="https://github.com/IBM/watson-openscale-samples/blob/main/Cloud%20Pak%20for%20Data/WML/notebooks/custom_metrics/Custom%20Metrics%20Provider%20for%20Cloud%20Pak%20for%20Data.ipynb">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*L_93xYBuer7sNdwcDrWwaQ.png" /><figcaption>Watson OpenScale</figcaption></figure><h3>Texas vs New York: Which assets are more likely to be predicted to fail?</h3><p>To uncover possible bias within our model, we created a custom fairness metric. Our metric essentially asks: <em>Do assets in Texas and New York have different predicted outcomes when the assets’ temperature is below a certain range? </em>This metric can provide us with information to highlight whether the model is biased toward one region in certain asset conditions.</p><p>In Watson OpenScale, our custom monitor continuously compares assets in different groups (regions where assets are located) using our fairness metric. Group 1 is Texas. Group 2 is New York. The predicted favorable label is “not-failing”. As Texas is a warmer climate than New York, SMEs are concerned general heat/humidity may impact the IoT sensors on our assets. Here, we are explicitly looking for an “non-causal”<em> </em>factor.</p><p>The metric we are monitoring is a ratio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hF1qSL2k2-JwKUMW0NHMHg.png" /></figure><ul><li>Ideally, we would aim for a value close to or equal to 1 to suggest equality between the different regions since the assets’ temperature should be the same in both locations.</li><li>A value &lt; 1 implies higher benefit for the assets in New York since more assets are predicted as not-failing.</li><li>A value &gt;1 implies a higher benefit for the assets in Texas since more assets are predicted as not-failing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*pLiD6QKJQdhgpmYEits3kQ.png" /><figcaption>Fairness Monitor</figcaption></figure><p>We utilized a bi-directional threshold. Our metric threshold is between 0.8 and 1.25 based on the 80% rule. We are concerned if our metric falls below 0.8 or above 1.25.​ Based on our monitor, we can initially see the metric is violating the lower threshold consistently until 5 PM. Here, we can begin to see some upward movement until the metric is not violating the thresholds.</p><p>When a threshold is violated, <em>Data Scientists </em>should first investigate if violations can be explained by the model and/or mitigated through Machine Learning practices. If a metric continues to violate a threshold, Data Scientists, SMEs, and Operations Engineers may need to collaborate to understand the source of the bias which could be connected to “non-causal” data.</p><h3>Conclusion: Fair Asset Performance Management</h3><p>AI in Sustainability practices aid with APM initiatives such as condition-based monitoring, predictive maintenance, asset failure prediction, detecting anomalies in daily usage patterns, and predicting the end-of-life of assets. These AI models increase profits, extend asset life, and avoid unplanned down-time. It is necessary that every deployed model is monitored to consistently check aspects such as drift, quality, fairness, explainability, and others.</p><p>At first glance, it may seem like AI fairness is not relevant with respect to APM in Industry related problems. However, fairness is <strong><em>critical</em></strong> in Industry. Fairness monitoring for APM AI models uncover and quantify how unexpected causal and non-causal factors can negatively affect your Machine Learning model and consequently your APM decisions. This fairness monitoring could uncover areas of improvement in the modelling approach that would otherwise not be detected. Furthermore, fairness metrics can uncover when strategy changes are needed to advance your APM approach.</p><p>We recommend that businesses rethink their ML-Ops strategy and include fairness-bias monitoring to ensure a <em>Fair </em>Asset Management Performance across their enterprise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13d44a3ab83e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/ai-fairness-in-industry-13d44a3ab83e">AI Fairness in Industry</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying and Monitoring Deep Learning Models on Cloud Pak for Data]]></title>
            <link>https://medium.com/ibm-data-ai/deploying-and-monitoring-deep-learning-models-on-cloud-pak-for-data-60c8c25ac522?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/60c8c25ac522</guid>
            <category><![CDATA[wmla]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[wml]]></category>
            <category><![CDATA[watson-openscale]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Tue, 21 Mar 2023 14:03:12 GMT</pubDate>
            <atom:updated>2023-07-17T13:42:22.166Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://medium.com/u/40b4d18b66a7"><em>Courtney Branson</em></a><em>, Advisory Data Scientist, and </em><a href="https://medium.com/u/ee180b77a57f"><em>Carolyn Saplicki</em></a><em>, Senior Data Scientist</em></p><p>Today, many businesses create AI models to aid in prediction. However, not all businesses have the tools or understanding of Machine Learning Operations (MLOps) to productionize an AI model. In fact, VentureBeat found that <a href="https://venturebeat.com/ai/why-do-87-of-data-science-projects-never-make-it-into-production/">87% of data science projects never make it in production</a>. Often, research teams do not generally consider the operationalization of models during creation. In this blog, we go over a use case of an AI assisted image recognition process that consisted of many deep learning models. The use case discussed here focuses on health care; however, this could apply to any deep learning image recognition model in industries such as bio-pharmaceutical, oil and gas, manufacturing, agricultural and more. We will highlight an architecture structure that allows for simple deployments and flexible orchestration in Cloud Pak for Data 4.5.2 (CPD) using Watson Studio, Watson Machine Learning (WML), Watson Machine Learning Accelerator (WMLA), and Watson OpenScale.</p><h3><strong>Project Description</strong></h3><p>Our client created an AI assisted workflow for a problem in health care. In this problem, AI models would aid the health care persona in deciding the diagnosis (class) of a certain condition. The steps are as follows:</p><h4><strong>Pre-Processing:</strong></h4><p>An image would be taken of the patient’s condition. This image would then be pre-processed into multiple different resolutions. Each resolution consisted of sections that make up the pre-processed image. The client utilized complex packages and created their own extensions for pre-processing. To pre-process images on CPD in Watson Studio, a custom image was created and employed so that we could use these packages.</p><h4><strong>Neural Network Models:</strong></h4><p>After an image is pre-processed, it will be used in a <a href="https://www.ibm.com/topics/neural-networks">Neural Network</a> model. Each Neural Network would loop over each section in the specific pre-processed image and output an array of prediction probabilities for each class, resulting in a matrix. For instance, if an image was pre-processed into 100 sections, the Neural Network corresponding to that resolution would output a matrix of 100 rows x <em>n</em> classes.</p><p>Each resolution level had multiple Neural Network models with a different number of classes for prediction. The models had varying numbers of classes, because they were built for different purposes, such as for additional research, testing, or diagnosis. Only one set of the models with the same number of classes would be used to generate a final prediction. We allowed for control of which set of models would be called by passing in a parameter that specified which set of models to use.</p><h4><strong>Aggregation Methods:</strong></h4><p>A single deep learning model produces a matrix of prediction probabilities of each class for each section of the original image. This matrix is aggregated for all sections based on any number of unique functions resulting in a final array of features. For instance, if the final matrix resulted in 100 sections x <em>n</em> classes, and we decided to use 3 aggregation functions, this would reduce the matrix to an array of 3<em>n</em> x 1.</p><h4><strong>Random Forest Model:</strong></h4><p>For a specific image, the final aggregated array from each Neural Network is concatenated. This is the input to the final <a href="https://www.ibm.com/topics/random-forest">Random Forest</a> model. The output is the final predicted class and associated prediction probabilities for each class.</p><p><em>Note: there is only one Random Forest model utilized for a final prediction.</em> Currently, the production version of this system only focuses on a specific number of classes.<em> </em>The Random Forest model has the same number of classes as one set of the Neural Network models. When a prediction shows low confidence, the Neural Network with more classes can be invoked to understand the predictions in more detail. This is useful in future work, specifically the creation of a final model with more classes.</p><h3><strong>Architecture Structure</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LlmX1IcH6Sr9VzcdFe1HFw.jpeg" /></figure><h3><strong>WMLA Model Deployments</strong></h3><p><a href="https://www.ibm.com/docs/en/wmla/3.0?topic=introduction-watson-machine-learning-accelerator-300">Watson Machine Learning Accelerator</a> (WMLA) is a deep learning service on Cloud Pak for Data. WMLA provides large model support that helps increase the amount of memory available for deep learning models (up to 16 GB or 32 GB per network layer), enabling more complex models and data inputs. WMLA also utilizes both CPUs and GPUs that are dynamically allocated.</p><p>We created a deployment in WMLA for each image resolution. Each deployment had multiple Neural Network models that supported a different model use such as research, testing, and/or diagnosis. Furthermore, each deployment also included the aggregation functions and the ability to save the matrix before aggregation. The deployment’s payload request specifies the image to be prediction upon, the path to the image, which model to be utilized and (optionally) the saving of the intermediate output.</p><h4><strong>Creating a </strong><a href="https://www.ibm.com/docs/en/wmla/2.3?topic=inference-create-service"><strong>WMLA Inference Service</strong></a><strong>:</strong></h4><p>WMLA uses a collection of deployment files in order to initialize and run an inference. For our deployment, these files were the <a href="https://www.ibm.com/docs/en/wmla/2.3?topic=service-create-kernel-file">kernel.py</a> which contains all the code we need to run within the deployment, a <a href="https://www.ibm.com/docs/en/wmla/2.3?topic=inference-edit-service">model.json</a> which contains all the configuration parameters, and a README.</p><h4><strong>Multiple Models:</strong></h4><p>Mutiple models are loaded in the on_kernel_start() method within our kernel.py. This method is only called once when the kernel is initialized. Loading the models in this function allows for quicker inference when the endpoint is called. Within our on_task_invoke() method we are then able to direct the inference request to the model specified in the payload request.</p><h4><strong>Storage Volume:</strong></h4><p>Both the original models and pre-processed images are pulled from an external storage volume. This storage volume can easily be connected to WMLA through a model.json with attributes as seen below:</p><pre>{<br> “name” : “example”,<br> “tag” : “example”,<br> “weight_path” : “N/A”,<br> “runtime” : “python”,<br> “kernel_path” : “kernel.py”,<br> “attributes” :<br> [<br> { “key”: “cpd-volumes-cpd::volume1”, “value”: “/mymount/path1”},<br> { “key”: “cpd-volumes-cpd::volume2”, “value”: “/mymount/path2”}<br> ],<br> “schema_version” : “1”<br>}</pre><p>A storage volume allows for controllable read/write access, persistent storage and sharing across projects. This means that the data we see in this volume can be mounted to JupyterLab environments as well as mounted to our WMLA deployment space. This, in addition to the volume APIs, made storage volumes easy to use across all phases of our MLOps pipeline.</p><p>As of WMLA 2.4, you are not able to connect to multiple storage volumes. However, plans to allow for this functionality are on the roadmap for future versions of WMLA.</p><h4><strong>Optional Save:</strong></h4><p>After discussion with the lead scientist, one important feature we added in the architecture structure is the ability to save the matrix produced by the Neural Network models. This matrix is probabilities of classes for each section within the pre-processed image. This will be useful in future work and understanding predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*WhyvDS1IcNCZSN0mSDnipQ.jpeg" /><figcaption>Optional Asset Save to Project Data Assets</figcaption></figure><p>This is saved to project data assets and is an optional input with respect to the deployments.</p><h3><strong>WML Deployment</strong></h3><p>Once the deep learning models were deployed within WMLA, we could focus on the WML deployment of the Random Forest model. This was a <a href="https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=functions-deployment-overview">standard WML deployment</a>. We were able to train the model within JupyterLab in Watson Studio and deploy it within the same notebook using the <a href="https://ibm-wml-api-pyclient.mybluemix.net/">WML Python SDK</a>. The deployment’s payload request specifies the final concatenation of the output of the deep learning models deployed in WMLA and provides a final prediction.</p><h3><strong>Watson OpenScale</strong></h3><p>Once the Random Forest model was successfully deployed to WML, we configured Watson OpenScale to monitor the deployment. We specifically monitored quality, drift and explainability monitors. An example of how to configure these monitors can be found <a href="https://medium.com/ibm-data-ai/practitioners-guide-to-trustworthy-ai-9ec9e94313e8">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*mFeeuTIR0gNHSdmtWtmpxw.png" /></figure><p>In this project, OpenScale does not directly monitor the deep learning models deployed within WMLA. However, the feature columns of the Random Forest model were the aggregated probabilities of each class from the deep learning models. This allowed the client to glean insights into what was happening within the complex deep learning models without setting up direct monitoring.</p><p>Our health care client appreciated the explainability aspect of Watson OpenScale, as their models are complex. The ability to explain predictions using an <a href="https://www.ibm.com/blogs/journey-to-ai/2020/04/comparing-ibm-watson-openscale-to-open-source-on-ai-explainability/">algorithm</a> largely based off <a href="https://lime-ml.readthedocs.io/en/latest/">LIME</a> allowed for them to consider future research areas. Furthermore, contrastive explanations helped them quantitatively understand which areas were the most impactful and by how much. This allowed them to better understand the nuance in the decision-making process of the models and see trends between diagnoses for their use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*jiJwwL3ToUJyAWNU2_tYlA.png" /><figcaption>LIME Analysis</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*PTd_W-gslYGdWHeK90rtSw.png" /><figcaption>What-If Analysis and Contrastive Explanations</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60c8c25ac522" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/deploying-and-monitoring-deep-learning-models-on-cloud-pak-for-data-60c8c25ac522">Deploying and Monitoring Deep Learning Models on Cloud Pak for Data</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fairness in Machine Learning: Pre-Processing Algorithms]]></title>
            <link>https://medium.com/ibm-data-ai/fairness-in-machine-learning-pre-processing-algorithms-a670c031fba8?source=rss-ee180b77a57f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a670c031fba8</guid>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[fairness]]></category>
            <category><![CDATA[aif360]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bias-mitigation]]></category>
            <dc:creator><![CDATA[Carolyn Saplicki]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 11:41:33 GMT</pubDate>
            <atom:updated>2023-07-17T13:41:57.314Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Co-authored by </em><a href="https://medium.com/u/ee180b77a57f"><em>Carolyn Saplicki</em></a><em>, Senior Data Scientist with Expert Labs, and </em><a href="https://medium.com/u/1ce94231dd05"><em>Mitali Bante</em></a><em>, Data Scientist with Expert Labs.</em></p><p>In our previous blog, <a href="https://medium.com/ibm-data-ai/fairness-explained-definitions-and-metrics-9690f8e0a4ea">Fairness Explained: Definitions and Metrics</a>, we discuss fairness definitions and fairness metrics through a real-world example. This sets the stage for how bias can be identified in machine learning. Specifically, we utilize tools such as the <a href="https://arxiv.org/pdf/2102.08453.pdf">Fairness Compass</a> and <a href="https://aif360.mybluemix.net/">AI Fairness 360</a> to understand the existence of bias and quantify its effects. The next crucial step is to take measures to mitigate bias. This blog focuses on pre-processing algorithms.</p><h3><strong>Bias Mitigation</strong></h3><p>There are several approaches to bias mitigation, which can be categorized into three categories: pre-processing, in-processing, and post-processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tvozGjrbNl81Z75U2DNORw.png" /></figure><p>Pre-processing algorithms involve modifying the dataset before training the model to remove or reduce the bias present in the data. Pre-processing algorithms are useful when the bias in the data is known or can be easily identified.</p><p>In-processing algorithms aim to reduce bias during the model training process. In-processing algorithms are useful when the bias in the data is not easily identifiable or when the dataset is too large to pre-process.</p><p>Post-processing algorithms involve modifying the output of the model after it has been trained. Post-processing algorithms are useful when the model is already trained, and the bias in the predictions is identified after deployment.</p><h3><strong>Pre-Processing Algorithms</strong></h3><p>Pre-processing algorithms are bias mitigation algorithms applied to training data, attempting to improve fairness metrics. These algorithms are the earliest mediation category and give the most flexibility to correct bias.</p><p>Pre-processing algorithms can only be utilized if a user is allowed to modify the training data before model building. This means pre-processing algorithms should not include fairness metrics that involve model predictions such as predictive parity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*9l0xdqnObOn5Baq-f6UAlg.jpeg" /></figure><h3><strong>AIF360 Methods</strong></h3><p>Below are the pre-processing algorithms found in <a href="https://aif360.readthedocs.io/en/stable/modules/algorithms.html#module-aif360.algorithms.preprocessing">AI Fairness 360</a> and some key take-aways.</p><h4><a href="https://aif360.readthedocs.io/en/stable/modules/generated/aif360.algorithms.preprocessing.DisparateImpactRemover.html#aif360.algorithms.preprocessing.DisparateImpactRemover">Disparate-Impact Remover</a></h4><p>Disparate-Impact Remover (DIR) increases group fairness by ensuring balanced representation of different protected groups. The DIR algorithm works by adjusting the features in the dataset so the distribution of these features is equal across all protected groups. Learn more about how <a href="https://arxiv.org/pdf/1412.3756.pdf">Disparate-Impact Remover</a> works and check out an <a href="https://github.com/Trusted-AI/AIF360/blob/master/examples/demo_disparate_impact_remover.ipynb">AIF360 code demo</a>.</p><ul><li>Primarily addresses group fairness (geared towards improving disparate impact)</li><li>Creates modified datasets in the same space as training data</li><li>Useful if pre-processing requires transparency</li></ul><h4><a href="https://aif360.readthedocs.io/en/stable/modules/generated/aif360.algorithms.preprocessing.LFR.html#aif360.algorithms.preprocessing.LFR">Learning Fair Representations</a></h4><p>Learning Fair Representations (LFR) finds a latent representation that encodes the data well but obfuscates information about protected attributes. The LFR algorithm works by identifying sensitive attributes and creating a new set of features by encoding the sensitive attributes, removing any association between the attributes and the output variable. Learn more about how <a href="http://proceedings.mlr.press/v28/zemel13.pdf">Learning Fair Representations</a> works and check out an <a href="https://github.com/Trusted-AI/AIF360/blob/master/examples/demo_lfr.ipynb">AIF360 code demo</a>.</p><ul><li>Addresses group and individual fairness</li><li>Creates modified datasets in a latent space (mapping each individual to a probability distribution you can make classifications on)</li></ul><h4><a href="https://aif360.readthedocs.io/en/stable/modules/generated/aif360.algorithms.preprocessing.OptimPreproc.html#aif360.algorithms.preprocessing.OptimPreproc">Optimized Pre-Processing</a></h4><p>Optimized Pre-Processing edits features and labels in the data with group fairness, individual distortion and data fidelity constraints and objectives through a probabilistic transformation. Learn more about how <a href="https://papers.nips.cc/paper/2017/file/9a49a25d845a483fae4be7e341368e36-Paper.pdf">Optimized Pre-Processing</a> works and check out an <a href="https://github.com/Trusted-AI/AIF360/blob/master/examples/demo_optim_data_preproc.ipynb">AIF360 code demo</a>.</p><ul><li>Primarily addresses group fairness while accounting for individual fairness</li><li>Creates modified datasets in the same space as training data</li><li>Useful if pre-processing requires transparency</li></ul><h4><a href="https://aif360.readthedocs.io/en/stable/modules/generated/aif360.algorithms.preprocessing.Reweighing.html#aif360.algorithms.preprocessing.Reweighing">Reweighing</a></h4><p>Reweighing generates weights for the training samples in each (group, label) combination differently, ensuring fairness before classification. Reweighing only changes weights applied to training samples (not any feature/label values). The reweighing algorithm first identifies sensitive attributes in the dataset and computes the weights for each instance. Higher weights are assigned to instances that are underrepresented and lower weights are assigned to instances that are overrepresented. Learn more about how <a href="https://files.core.ac.uk/pdf/2612/81728147.pdf">Reweighing</a> works and check out an <a href="https://github.com/Trusted-AI/AIF360/blob/master/examples/demo_reweighing_preproc.ipynb">AIF360 code demo</a>.</p><ul><li>Addresses group fairness (geared towards improving statistical parity)</li><li>Creates modified datasets in the same space as training data</li><li>Useful if unable to apply value changes</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0L99hcoMYTzC_vpeubP-Tw.png" /></figure><h3><strong>Best Practices</strong></h3><ol><li><strong>Remove bias as early as possible during the ML lifecycle!</strong></li><li><strong>All pre-processing algorithms should be tried on the training data.</strong> The algorithm(s) that remove the most bias should be utilized. Disparate-Impact Remover and Optimized Pre-Processing are useful if pre-processing requires transparency on the transformation.</li><li><strong>If algorithms are being compared to each other, ensure they are compared based on the same dataset and preprocessing.</strong> For algorithms that only work with numerical features, try one-hot encoding or target encoding for categorical features.</li><li>Furthermore, a single train/test split is insufficient. <strong>We recommend utilizing multiple randomized train/test splits or a K-fold validation.</strong> This will test stability of the algorithm.</li><li><strong>Compare algorithms outcomes based on multiple metrics</strong> (accuracy, fairness, ROC, etc).</li><li>There is a trade-off between fairness and accuracy. <strong>When applicable, use hyperparameter tuning to explore the possible solution space.</strong></li></ol><h3><strong>FAQs</strong></h3><p><strong><em>When to conclude a pre-processing bias mitigation algorithm is needed?</em></strong></p><p>Existence of bias in your data can be determined by using the <a href="https://medium.com/ibm-data-ai/fairness-explained-definitions-and-metrics-9690f8e0a4ea">fairness metrics</a> in AIF360.</p><p><strong><em>When to use which algorithm?</em></strong></p><p>Currently, <a href="https://aif360.mybluemix.net/resources#guidance">AI Fairness 360</a> suggests all pre-processing algorithms, “should be tested because the ultimate performance depends on dataset characteristics: there is no one best algorithm independent of dataset.” If you need transparency about the pre-processing transformation, Disparate-Impact remover and Optimized Pre-Processing are ideal.</p><p><strong><em>What if I have multiple protected attributes?</em></strong></p><p>Utilize multiple algorithms to investigate and mitigate bias for each protected attribute. Ensure you follow the same process each time you employ pre-processing algorithms. Optimized Pre-Processing and Reweighing can mitigate bias for multiple protected attributes at once.</p><p><strong><em>What about other bias-mitigation methods?</em></strong></p><p>Other bias mitigation methods can be utilized with pre-processing algorithms. Again, it is best practice to remove bias as early as possible during the ML lifecycle.</p><p><strong><em>How much bias removal matters?</em></strong></p><p>Any bias removal matters in machine learning. If a pre-processing algorithm can reduce the status quo of bias, it should be utilized. Often, multiple algorithms are used to reduce the presence of bias in a machine learning model.</p><p><strong><em>What if data is not accessible to perform pre-processing bias mitigation?</em></strong></p><p>As mentioned, it’s better to perform bias mitigation at the earliest stage in machine learning model development cycle, but if it cannot be performed during pre-processing, you can try in-processing or post-processing bias mitigation algorithms.</p><p>#IBM #MachineLearning #Fairness #PreProcessing #BiasMitigation</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a670c031fba8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/fairness-in-machine-learning-pre-processing-algorithms-a670c031fba8">Fairness in Machine Learning: Pre-Processing Algorithms</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>