<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Melissa Dunkeld on Medium]]></title>
        <description><![CDATA[Stories by Melissa Dunkeld on Medium]]></description>
        <link>https://medium.com/@melissadunkeld?source=rss-dda11447d179------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ppWLIXwJ0PEeUCrn</url>
            <title>Stories by Melissa Dunkeld on Medium</title>
            <link>https://medium.com/@melissadunkeld?source=rss-dda11447d179------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:01:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@melissadunkeld/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Beginner’s Guide to Vibe Coding: Build Faster With AI Without Skipping the Fundamentals]]></title>
            <link>https://medium.com/@melissadunkeld/a-beginners-guide-to-vibe-coding-build-faster-with-ai-without-skipping-the-fundamentals-320c9b3dd0db?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/320c9b3dd0db</guid>
            <category><![CDATA[vibe-coding]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Tue, 12 May 2026 18:48:45 GMT</pubDate>
            <atom:updated>2026-05-12T18:48:45.066Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QJQI0R8cHtbnvYoOiNLY7w.png" /></figure><p>A few years ago, learning to code meant staring at a blank editor and manually writing every line from scratch. Today, AI tools have changed that workflow completely.</p><p>Instead of starting with syntax, boilerplate, and documentation, many developers now begin with a simple sentence:</p><blockquote>“Build me a login page with React and Tailwind.”</blockquote><p>The AI generates the first draft, you test it, refine it, and continue iterating until the software works the way you want.</p><p>This workflow is commonly called <strong>vibe coding</strong>.</p><p>And despite the hype around it, vibe coding is <strong>not magic,</strong> nor is it a replacement for learning software development fundamentals. It’s simply a new way of interacting with code using AI as a <strong>collaborative assistant</strong>.</p><p>For beginners, this can make programming feel more approachable, more creative, and significantly less intimidating.</p><p>In this guide, you’ll learn:</p><ul><li>What vibe coding actually means</li><li>How the workflow works in practice</li><li>The best AI tools for coding</li><li>Common mistakes beginners make</li><li>Debugging and testing strategies</li><li>When AI-assisted coding works well — and when it doesn’t</li><li>Ethical and security risks to avoid</li></ul><h3>So What Is Vibe Coding?</h3><p>Vibe coding is the process of building software using natural language prompts with AI assistance.</p><p>Instead of manually writing every line of code yourself, you describe what you want the software to do, and the AI generates an initial implementation.</p><p>You then:</p><ol><li>Review the code</li><li>Run it</li><li>Test it</li><li>Refine it through follow-up prompts</li></ol><p>The process becomes an iterative loop between you and the AI.</p><p>A typical vibe coding workflow looks like this:</p><ol><li>Describe the goal</li><li>AI generates code</li><li>Run and test the code</li><li>Identify issues or improvements</li><li>Refine the prompt</li><li>Repeat until the result works</li></ol><p>This approach is sometimes called <strong>prompt-driven development</strong> because prompts become the starting point of development instead of raw syntax.</p><p>But there’s an important reality beginners need to understand:</p><p>AI can generate code, but it cannot replace understanding.</p><p>You still need to:</p><ul><li>Read the generated code</li><li>Debug failures</li><li>Verify outputs</li><li>Understand basic logic</li><li>Identify security risks</li><li>Test edge cases</li></ul><p>Think of AI as a <strong>coding partner</strong> — not an autopilot.</p><h3>Tools Commonly Used for Vibe Coding</h3><p>Different AI tools support different coding workflows. Here are the major categories beginners should know.</p><h3>1. Chat-Based AI Tools</h3><p>Examples include:</p><ul><li><a href="https://chatgpt.com/?utm_source=chatgpt.com">ChatGPT</a></li><li><a href="https://claude.ai/?utm_source=chatgpt.com">Claude</a></li></ul><p>These tools work conversationally. You describe a problem, paste code, ask questions, or request revisions.</p><p>They’re excellent for:</p><ul><li>Learning concepts</li><li>Explaining errors</li><li>Prototyping ideas</li><li>Writing isolated components</li><li>Understanding unfamiliar frameworks</li></ul><p>Because they rely heavily on prompts, clarity matters a lot.</p><h3>2. IDE-Integrated AI Assistants</h3><p>Examples include:</p><ul><li><a href="https://github.com/features/copilot?utm_source=chatgpt.com">GitHub Copilot</a></li><li><a href="https://cursor.com/?utm_source=chatgpt.com">Cursor</a></li></ul><p>These tools work directly inside your editor and provide:</p><ul><li>Inline code completion</li><li>Auto-suggestions</li><li>Refactors</li><li>Boilerplate generation</li></ul><p>They’re useful for improving coding speed while staying inside your development environment.</p><h3>3. Agent-Based Coding Tools</h3><p>Examples include:</p><ul><li><a href="https://replit.com/ai?utm_source=chatgpt.com">Replit Agent</a></li><li><a href="https://www.anthropic.com/claude-code?utm_source=chatgpt.com">Claude Code</a></li></ul><p>These tools attempt to manage larger tasks across multiple files.</p><p>Some can:</p><ul><li>Generate full applications</li><li>Run tests</li><li>Modify existing projects</li><li>Understand project structure</li><li>Maintain context across sessions</li></ul><p><strong>They can be powerful, but beginners should still review everything carefully because AI agents can make large-scale mistakes quickly.</strong></p><h3>4. Local LLM Setups</h3><p>Examples include:</p><ul><li><a href="https://ollama.com/?utm_source=chatgpt.com">Ollama</a></li><li><a href="https://lmstudio.ai/?utm_source=chatgpt.com">LM Studio</a></li></ul><p>These allow you to run AI models locally on your own hardware.</p><p>Benefits include:</p><ul><li>Better privacy</li><li>Offline usage</li><li>More control over data</li></ul><p>The tradeoff is additional setup complexity and hardware requirements.</p><h3>The Core Vibe Coding Workflow</h3><p>Vibe coding works best when treated as a continuous loop rather than a single request.</p><p>Here’s the process beginners should follow.</p><h3>Step 1: Describe the Goal Clearly</h3><p>Focus on <em>what</em> you want the software to do.</p><p>Bad prompt:</p><blockquote>“Build a website.”</blockquote><p>Better prompt:</p><blockquote>“Build a responsive React landing page with a pricing section and dark mode toggle.”</blockquote><p>The more context you provide, the better the AI performs.</p><p>Include:</p><ul><li>Programming language</li><li>Frameworks</li><li>Constraints</li><li>Desired functionality</li><li>Input/output examples</li></ul><h3>Step 2: Generate the First Draft</h3><p>Let the AI create an initial version.</p><p>Do not assume this version is production-ready.</p><p>Expect:</p><ul><li>Missing edge cases</li><li>Poor structure</li><li>Incorrect assumptions</li><li>Security gaps</li><li>Outdated APIs</li></ul><p><strong>Your job is to validate the output.</strong></p><h3>Step 3: Run the Code Immediately</h3><p>This is where many beginners fail.</p><p>Never continue generating more code before verifying the current version actually works.</p><p>Run it in:</p><ul><li>Your browser</li><li>Terminal</li><li>Local environment</li><li>Sandbox</li></ul><p>Fix runtime issues early before complexity grows.</p><h3>Step 4: Refine the Prompt</h3><p>Once you identify issues, improve your instructions.</p><p>For example:</p><ul><li>“Add validation for empty form fields.”</li><li>“Use Tailwind instead of plain CSS.”</li><li>“Refactor this into reusable components.”</li></ul><p>Small prompt improvements can dramatically improve outputs.</p><h3>Step 5: Debug and Iterate</h3><p>This is where learning actually happens.</p><p>When errors appear:</p><ul><li>Copy the exact error message</li><li>Paste it back into the AI</li><li>Ask for an explanation first</li><li>Then ask for a fix</li></ul><p>Understanding <strong><em>why</em> </strong>something failed matters far more than blindly applying patches.</p><h3>How to Write Better Coding Prompts</h3><p>You don’t need advanced prompt engineering skills, but a few habits can improve your results significantly.</p><h3>Be Specific About Your Stack</h3><p>Instead of:</p><blockquote>“Build a dashboard.”</blockquote><p>Say:</p><blockquote>“Build a dashboard using Next.js, TypeScript, and Tailwind CSS.”</blockquote><p>Specificity prevents the AI from making incorrect assumptions.</p><h3>Provide Constraints</h3><p>Mention:</p><ul><li>Performance limits</li><li>Allowed libraries</li><li>Coding styles</li><li>Browser support</li><li>Version requirements</li></ul><p>Constraints guide the AI toward practical solutions.</p><h3>Include Sample Inputs and Outputs</h3><p>AI performs much better when examples are included.</p><p>Example:</p><p>“Input: CSV file with names and emails<br>Output: JSON array grouped by company.”</p><p>Examples reduce ambiguity.</p><h3>Ask for Explanations</h3><p>Always ask:</p><blockquote>“Explain how this works.”</blockquote><p>This improves:</p><ul><li>Debugging ability</li><li>Conceptual understanding</li><li>Long-term learning</li></ul><h3>Break Large Tasks Into Smaller Pieces</h3><p>Avoid:</p><p>“Build an entire SaaS platform.”</p><p>Instead:</p><ol><li>Build authentication</li><li>Build dashboard layout</li><li>Add database integration</li><li>Add billing</li></ol><p>Smaller prompts produce cleaner results.</p><h3>Why Testing Matters in Vibe Coding</h3><p>AI-generated code can <strong><em>look</em> </strong>perfect while being completely broken underneath.</p><p>That’s why testing is non-negotiable.</p><p>Always:</p><ul><li>Run the code yourself</li><li>Verify outputs manually</li><li>Test unexpected inputs</li><li>Check edge cases</li><li>Read console errors carefully</li></ul><p>Useful debugging tools include:</p><ul><li>Console logs</li><li>Print statements</li><li>Stack traces</li><li>Browser developer tools</li></ul><p>One of the biggest beginner mistakes is assuming:</p><blockquote>“The AI wrote it, so it must work.”</blockquote><p>It often doesn’t.</p><p>Treat AI suggestions as hypotheses, not truths.</p><h3>Common Beginner Mistakes in Vibe Coding</h3><h3>1. Trusting AI Without Review</h3><p>Never paste code you don’t understand into production.</p><p>AI can:</p><ul><li>Delete data accidentally</li><li>Introduce vulnerabilities</li><li>Misconfigure systems</li><li>Add unnecessary complexity</li></ul><p>Always review generated code.</p><h3>2. Building Too Much Too Fast</h3><p>Beginners often ask AI to generate huge systems immediately.</p><p>This usually creates:</p><ul><li>Massive debugging problems</li><li>Inconsistent architecture</li><li>Conflicting dependencies</li></ul><p>Build incrementally instead.</p><p><strong>3. Ignoring Edge Cases</strong></p><p>AI tends to optimise for “happy path” scenarios.</p><p>You still need to handle:</p><ul><li>Empty inputs</li><li>Invalid data</li><li>Offline states</li><li>Failed requests</li><li>Permission issues</li></ul><p>Always test weird user behavior.</p><p><strong>4. Adding Too Many Dependencies</strong></p><p>AI may suggest multiple libraries for simple problems.</p><p>Every dependency increases:</p><ul><li>Maintenance burden</li><li>Security risk</li><li>Build complexity</li></ul><p>Use standard language features whenever possible.</p><h3>5. Letting Projects Become Disorganised</h3><p>As projects grow, AI context becomes weaker.</p><p>Without structure:</p><ul><li>AI gives conflicting code</li><li>Files become inconsistent</li><li>Refactors break functionality</li></ul><p>Keep your project organised early.</p><h3>Where Vibe Coding Works Best</h3><p>Vibe coding shines in situations where speed and experimentation matter more than perfect architecture.</p><h3>Prototyping</h3><p>You can validate ideas quickly before investing heavily in engineering.</p><h3>Learning New Frameworks</h3><p>AI can:</p><ul><li>Generate examples</li><li>Explain concepts</li><li>Compare approaches</li><li>Teach syntax</li></ul><p>This dramatically lowers learning friction.</p><h3>Automating Small Tasks</h3><p>Great for:</p><ul><li>Data cleanup scripts</li><li>Workflow automation</li><li>One-off utilities</li><li>Internal tooling</li></ul><h3>Boilerplate Generation</h3><p>AI is excellent at repetitive setup tasks like:</p><ul><li>Configuration files</li><li>CRUD scaffolding</li><li>Component templates</li><li>API routes</li></ul><p><strong>Writing Documentation</strong></p><p>Once your code works, AI can help generate:</p><ul><li>README files</li><li>Comments</li><li>API docs</li><li>Setup instructions</li></ul><h3>Where Vibe Coding Starts to Break</h3><p>AI-assisted coding becomes less reliable as complexity increases.</p><h3>Large Codebases</h3><p>AI models have context limits.</p><p>In large projects:</p><ul><li>Important dependencies get missed</li><li>Refactors become inconsistent</li><li>Hidden bugs appear</li></ul><h3>Complex Stateful Systems</h3><p>Applications with complicated long-term state management often confuse AI.</p><p>Examples include:</p><ul><li>Multiplayer systems</li><li>Real-time synchronisation</li><li>Financial transaction flows</li></ul><h3>Security-Critical Applications</h3><p>Never rely solely on AI for:</p><ul><li>Payment systems</li><li>Authentication infrastructure</li><li>Sensitive databases</li></ul><p>AI often prioritises convenience over security.</p><h3><strong>Performance-Critical Systems</strong></h3><p>AI tends to favor readability over optimisation.</p><p>For high-performance systems, manual engineering still matters heavily.</p><h3>Team-Based Development</h3><p>Without coding standards and reviews, vibe coding can create inconsistent architectures across teams.</p><h3>Vibe Coding vs Traditional Coding</h3><p>The biggest misconception online is that vibe coding replaces traditional development.</p><p>It doesn’t.</p><p>They are complementary workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Nr4qaytKdg0cKRP2g6wGw.png" /></figure><p>A practical workflow often looks like this:</p><ol><li>Use AI to prototype quickly</li><li>Validate the concept</li><li>Refactor manually</li><li>Harden the architecture</li><li>Optimise and secure the system</li></ol><h3>Is Vibe Coding a Real Career Skill?</h3><p>Yes — but not in the way social media often portrays it.</p><p>Companies still care deeply about:</p><ul><li>Problem solving</li><li>Debugging</li><li>System design</li><li>Data structures</li><li>Software fundamentals</li></ul><p>However, AI literacy is becoming increasingly valuable.</p><p>Developers who know:</p><ul><li>When to use AI</li><li>How to validate outputs</li><li>How to debug generated code</li><li>How to integrate AI into workflows</li></ul><p>…will likely become more productive than those who ignore these tools completely.</p><p>The key is balance.</p><p>AI should accelerate your learning — not replace it.</p><h3>Ethical and Security Risks Beginners Should Know</h3><h3>API Key Leaks</h3><p>Never paste secrets directly into prompts or source files.</p><p>Use:</p><ul><li>.env files</li><li>Environment variables</li><li>Secret managers</li></ul><h3>Licensing Issues</h3><p>AI can occasionally reproduce licensed code patterns.</p><p>If building commercial software, review generated code carefully.</p><h3>Hallucinated Libraries</h3><p>AI sometimes invents:</p><ul><li>Fake APIs</li><li>Nonexistent packages</li><li>Incorrect functions</li></ul><p>Always verify documentation manually.</p><h3>Data Privacy Concerns</h3><p>Be cautious about pasting:</p><ul><li>Company data</li><li>Customer information</li><li>Proprietary code</li><li>Sensitive documents</li></ul><p>Public AI tools may store prompts for training purposes.</p><h3><strong>Code Review Responsibility</strong></h3><p>AI-generated code should be treated like third-party contributions:</p><ul><li>Review it</li><li>Test it</li><li>Validate it</li><li>Secure it</li></ul><p>The responsibility still belongs to the developer.</p><h3>Final Thoughts</h3><p>Vibe coding is one of the most accessible ways beginners can start building software today.</p><ul><li>It lowers friction.</li><li>It speeds up experimentation.</li><li>It makes coding feel more creative and interactive.</li></ul><p>But it works best when paired with real understanding.</p><p>The developers who benefit most from AI are not the ones who blindly paste code — they’re the ones who:</p><ul><li>Ask better questions</li><li>Test carefully</li><li>Debug patiently</li><li>Understand the systems they build</li></ul><p>AI can help you move faster.</p><p>But fundamentals are still what keep your software standing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=320c9b3dd0db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Creativity May Be the Most Underrated Superpower in Data Strategy]]></title>
            <link>https://medium.com/@melissadunkeld/why-creativity-may-be-the-most-underrated-superpower-in-data-strategy-c31e034db536?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/c31e034db536</guid>
            <category><![CDATA[data-strategy]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Wed, 03 Dec 2025 18:07:59 GMT</pubDate>
            <atom:updated>2025-12-03T18:07:59.691Z</atom:updated>
            <content:encoded><![CDATA[<p>When people think about data, they picture dashboards, governance frameworks, SQL queries, and decision-making.<br> What they <em>don’t</em> often imagine is creativity.</p><p>Yet creativity may be the most powerful — and underused — accelerator in any data initiative.</p><p>A few years ago, the author of <em>Humanizing Data Strategy</em> released a rap song about data governance. Yes, a rap song! It sparked conversations across the broader data community for one simple reason: <strong>no one expected creativity in a field stereotyped as rigid and analytical.</strong></p><p>And that’s the point.</p><p>Creativity surprises. It engages. It reframes.<br> And if there’s one thing every data team needs today, it’s the power to reframe problems and invent better solutions.</p><h3>Creativity and Data: Not So Different After All</h3><p>At first glance, music and data may seem worlds apart. But look a little deeper:</p><ul><li>Music has a defined scale of notes — data has defined sets of values.</li><li>Composers turn sounds into art — analysts turn data points into insights.</li><li>Musicians express emotion — data storytellers express meaning.</li><li>Bands collaborate — data and business teams collaborate.</li><li>People have different musical tastes — stakeholders have different data needs.</li></ul><p>The overlap is clear: both disciplines are about translating structure into something meaningful.</p><p>Creativity isn’t just for “artistic” professions. With the right lens, it becomes a force multiplier for data strategy.</p><h3>Not All Creativity Looks the Same</h3><p>The book breaks creativity into two simple dimensions:</p><h3>1. Spontaneous vs. Solution-Oriented</h3><ul><li><em>Spontaneous creativity</em> is the spark of an idea — the unexpected analogy, the novel storyline, the surprising connection.</li><li><em>Solution-oriented creativity</em> is more practical — optimising a process, reframing a data problem, or finding a better way to deliver value.</li></ul><h3>2. Incremental vs. Disruptive</h3><ul><li><em>Incremental creativity</em> makes small improvements to what exists.</li><li><em>Disruptive creativity</em> introduces something significantly new — a shift in platforms, methods, or capabilities.</li></ul><p>Together, these create a matrix with four modes:</p><ul><li><strong>Experimentation</strong> (solution-oriented + incremental)</li><li><strong>Continuous Improvement</strong> (spontaneous + incremental)</li><li><strong>Change Management</strong> (solution-oriented + disruptive)</li><li><strong>Guided Innovation</strong> (spontaneous + disruptive)</li></ul><p>Understanding these modes helps teams harness creativity intentionally rather than accidentally.</p><h3>Creating an Environment Where Creativity Thrives</h3><p>Creativity sounds exciting — but being creative can feel risky.</p><p>It means putting forward something new.<br> It means uncertainty.<br> It means the chance of failure.</p><p>And failure <em>hurts</em> — unless the environment makes it safe.</p><p>The book highlights an important truth: <strong>creative people don’t fear failure because they’ve survived it.</strong> They’ve shared bold ideas, seen them flop, and realised the world didn’t end. In fact, failure builds resilience.</p><p>To nurture creativity in data teams, leaders need to cultivate psychological safety:</p><ul><li>Encouraging ideas — even unfinished ones</li><li>Letting teams experiment without punishment</li><li>Framing failures as learning, not liabilities</li><li>Modelling curiosity and openness</li></ul><p>Without this foundation, creativity suffocates. With it, innovation becomes natural.</p><h3>Creativity As a Muscle</h3><p>Creativity isn’t a mysterious talent — it’s a skill.<br> And like any skill, it strengthens with use.</p><p>The book frames it across three practices:</p><h3>Mindset</h3><p>Believing creativity belongs in data, and recognising that not every idea must succeed.</p><h3>Practice</h3><p>Giving teams real opportunities to experiment, collaborate, and build creative confidence.</p><h3>Habit</h3><p>Embedding creativity into how the team works every day — through brainstorming, iteration, retrospectives, and exploration.</p><p>The more creativity is exercised, the more effortlessly it appears.</p><h3>Reflection Fuels Creativity</h3><p>We often wait for a crisis or a major initiative to start innovating. But creativity thrives in everyday moments.</p><p>Reflection — on frustrations, inefficiencies, awkward processes, or confusing data experiences — is a surprisingly rich source of ideas. Minor annoyances often spark significant improvements.</p><p>Some of the best creativity comes from asking:<br> <strong>“What could be better here?”</strong></p><p>And then acting on it.</p><h3>A Call to Re-Humanise Data Work</h3><p>Data strategy doesn’t succeed on dashboards or governance policies alone.<br> It succeeds because of the quality of the human thinking behind it.</p><p>The book makes a clear argument:</p><blockquote>To build better data strategies, we must tap into originality, empathy, experimentation, and unconventional thinking.</blockquote><p>Not everything needs to be optimised. Not everything needs to be automated.<br> But everything benefits from creativity.</p><p>When organisations embrace creativity — from the most minor SQL tweak to the boldest digital transformation — their data strategies stop being mechanical and become meaningful.</p><p>And meaningful is where the real impact happens.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c31e034db536" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Humanising Data Strategy]]></title>
            <link>https://medium.com/@melissadunkeld/humanising-data-strategy-3f52af1fcbfc?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f52af1fcbfc</guid>
            <category><![CDATA[data-strategy]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Wed, 03 Dec 2025 17:39:28 GMT</pubDate>
            <atom:updated>2025-12-03T17:39:28.984Z</atom:updated>
            <content:encoded><![CDATA[<p>When people talk about “data strategy,” the conversation often gets technical fast — platforms, pipelines, analytics, governance. But after digging into countless definitions, one thing becomes clear: there’s <em>no universal agreement</em> on what a data strategy truly is.</p><p>What most definitions <em>do</em> share is a focus on long-term planning, people, processes, technology, and the management of information and data. Yet even with these themes, organisations still struggle to make their data strategies work.</p><p>So here’s a simple definition I use:</p><p><strong>A data strategy is a long-term plan that defines the people, processes, and technologies needed to create, process, and use data in ways that intentionally drive value — meaningfully, securely, and transparently.</strong></p><p>Straightforward enough. But in practice, many data strategies still fail. In my experience, it usually comes down to five recurring issues:</p><ol><li><strong>A lack of data literacy.</strong><br> Many employees don’t feel equipped or empowered to adopt data-driven work in their daily routines.</li><li><strong>Data work reinforces silos rather than breaking them down.</strong><br> Worse, it can split teams into “I love data” vs. “I hate data” factions.</li><li><strong>Nobody knows what data <em>actually</em> means for the business.</strong><br> When data is not understood as an asset — and people don’t know what to do with it — they disengage.</li><li><strong>Data feels like a chore.</strong><br> It’s treated as tedious, monotonous, and something done out of obligation rather than value.</li><li><strong>Data becomes political.</strong><br> A few significant data issues escalated to leadership, making people feel like data work is a minefield they’d rather avoid.</li></ol><p>To solve these problems, we need to address the <em>human</em> side of data strategy. That’s where the <strong>5Cs Framework</strong> comes in.</p><h3>The 5Cs of Humanising Data Strategy</h3><p>The “people side” of data strategy can be broken down into five dimensions. Together, they create a foundation not only for better data work but for better collaboration, decision-making, and innovation across the business.</p><h3>1. Competence</h3><p>Empowering people with the right business, data, and technology knowledge — while supporting their motivation to learn, grow, and connect — builds a healthier, more capable data culture. Competence isn’t just training; it’s confidence.</p><h3>2. Collaboration</h3><p>Data flourishes when teams work together. That means transparency, shared goals, accountability, and a culture that encourages collective problem-solving. Collaboration turns data into a cross-functional asset rather than a departmental burden.</p><h3>3. Communication</h3><p>The best data work fails if nobody understands it. Effective communication uses audience-specific storytelling, highlights real business value, and provides both proactive and reactive feedback loops. It turns insights into action.</p><h3>4. Creativity</h3><p>Data doesn’t eliminate creativity — it requires it. When people are rewarded for experimenting, improving processes, and thinking differently, data work becomes a space for innovation instead of a rigid routine.</p><h3>5. Conscience</h3><p>Ethics, critical thinking, and responsible decision-making must remain at the core of data efforts. Conscience keeps data secure, compliant, and aligned with organisational values — not just technical requirements.</p><h3>But What About Technology?</h3><p>If the 5Cs focus so heavily on people, where does technology fit in?</p><p>The answer: <strong>technology is a tool — nothing more.</strong><br> Vendors may promise automation, intelligence, and efficiency, but tools exist to help humans achieve their goals, not replace critical thinking.</p><p>Even with automation advancing rapidly, we still rely on human judgment for complex decisions, nuanced interpretation, and ethical considerations. This applies across both business processes and technical data workflows.</p><p><strong>Tools can and should support:</strong></p><ul><li>user experience</li><li>collaboration</li><li>automation</li><li>repetitive or time-consuming tasks</li><li>data discovery and change management</li></ul><p>…but they work best when paired with the human elements outlined in the 5Cs.</p><h3>Bringing It All Together</h3><p>The 5Cs — Competence, Collaboration, Communication, Creativity, and Conscience — form the backbone of a human-centred data strategy. They address the typical failures we see across organisations and give leaders a practical, memorable way to elevate their data culture.</p><p>In future posts, we can dive deeper into each C, share examples, and explore how people, processes, and technology come together to build a truly successful data strategy.</p><p>For now, remember this:<br> <strong>Data strategy succeeds not because of tools, but because of people.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f52af1fcbfc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Web Scraping with Python: A Guide for Data Scientists]]></title>
            <link>https://medium.com/@melissadunkeld/automating-web-scraping-with-python-a-guide-for-data-scientists-007d7b47cd01?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/007d7b47cd01</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Mon, 18 Nov 2024 20:26:17 GMT</pubDate>
            <atom:updated>2024-11-18T20:26:17.409Z</atom:updated>
            <content:encoded><![CDATA[<p>Web scraping is an important skill within data science and analytics for extracting information from web pages when APIs are not available or have been poorly provided.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MctC9ONnLV0TAgm4.jpeg" /></figure><p>This article provides a step-by-step guide to developing a Python-based web scraper that can scrape data from pages within an XML sitemap and write the results into a CSV file. The examples include main Python libraries such as BeautifulSoup for parsing HTML, making HTTP calls with requests, and xml.etree.ElementTree for XML processing.</p><h3>Key Concepts Covered:</h3><ol><li><strong>Loading and Parsing XML Files</strong></li><li><strong>Extracting Data from Web Pages</strong></li><li><strong>Handling HTML Tables</strong></li><li><strong>Storing Data in CSV Files</strong></li></ol><p>Let’s break down the approach and code step by step</p><h3>Loading and Parsing the XML Sitemap</h3><p>For this project I used a website to provide XML sitemaps, making it easier to find and access all the URLs for scraping. First up I start by loading the sitemap XML file and parsing it using xml.etree.ElementTree.</p><p>Before you do this open your XML file to find the namespace to use.</p><pre>import xml.etree.ElementTree as ET<br><br># Load and parse the XML file<br>tree = ET.parse(&#39;filename.xml&#39;)<br>root = tree.getroot()<br><br># Namespace used in the XML<br>namespace = {&#39;ns&#39;: &#39;http://www.sitemaps.org/schemas/sitemap/0.9&#39;}</pre><p>The namespace dictionary is used to handle XML namespaces, which are common in well-structured sitemaps. By providing the correct namespace, you can accurately find elements.</p><h3>Scraping the Data</h3><p>Once we have the list of URLs from the sitemap, we use the requests library to fetch the HTML content from each page. The slot game data we’re interested in is located within a table, so BeautifulSoup helps us parse this HTML and extract the relevant information.</p><pre>import requests<br>from bs4 import BeautifulSoup<br><br># Loop through each &lt;url&gt; in the sitemap XML<br>for url_tag in root.findall(&#39;ns:url&#39;, namespace):<br>    url_loc = url_tag.find(&#39;ns:loc&#39;, namespace)<br>    if url_loc is not None:<br>        url = url_loc.text<br>        print(f&quot;Scraping data from: {url}&quot;)<br><br>        # Fetch the page content<br>        try:<br>            response = requests.get(url)<br>            response.raise_for_status()  # This raises an error for bad responses<br>        except requests.exceptions.RequestException as e:<br>            print(f&quot;Failed to fetch {url}: {e}&quot;)<br>            continue</pre><p>I then loop through the URLs found in the sitemap and make HTTP GET requests to fetch each page’s content. If the request fails, I have added some exception handling to ensure the program doesn’t crash, allowing it to move on to the next URL in the file.</p><h3>Parsing and Extracting The Data</h3><p>Now I have the HTML content of a page, which I then use BeautifulSoup to navigate the HTML and extract the game information. In this example, I’m interested in properties like “Provider,” “Release Date,” “RTP,” and several other attributes.</p><pre># Extract data for each game<br>html_content = response.text<br>soup = BeautifulSoup(html_content, &#39;html.parser&#39;)<br><br># Dictionary to store the data<br>attributes = {key: None for key in [&quot;Name&quot;] + prop_left_values}<br><br># Extract the name and do a little bit of data cleaning <br>name_tag = soup.find(&#39;h3&#39;, {&#39;name&#39;: &#39;attributes&#39;, &#39;id&#39;: &#39;attributes&#39;})<br>if name_tag:<br>    name = name_tag.text.strip().replace(&quot;Attributes&quot;, &quot;&quot;).strip()<br>    attributes[&quot;Name&quot;] = name<br>else:<br>    print(f&quot;Game name not found for {url}, skipping...&quot;)<br>    continue<br><br># Extract other attributes from table rows<br>rows = soup.find_all(&#39;tr&#39;)<br>for row in rows:<br>    prop_left = row.find(&#39;th&#39;, class_=&#39;propLeft&#39;)<br>    prop_right = row.find(&#39;td&#39;, class_=&#39;propRight&#39;)<br>    if prop_left and prop_right:<br>        prop_left_text = prop_left.text.strip()<br>        if prop_left_text in prop_left_values:<br>            prop_right_text = prop_right.text.strip()<br>            slot_attributes[prop_left_text] = prop_right_text</pre><p>This block of code does the heavy lifting of extracting the data I want. First, it searches for the attribute I want, then loops through each row in the table to grab the left (attribute name) and right (attribute value) columns. The prop_left_values list acts as a filter, ensuring I’m only capturing those attributes I care about.</p><h3>Storing Data in CSV</h3><p>Once the data is extracted, we save it to a CSV file for further analysis. The csv.DictWriter makes it easy to write a dictionary of attributes as a row in the CSV file.</p><p>Here, we create a directory (game_data) If it doesn&#39;t exist then specify the CSV file path. The data is written row-by-row, allowing me to accumulate and store a large number of records efficiently.</p><h3>Conclusion</h3><p>This script is a powerful starting point for web scraping projects. By automating the extraction of data from XML sitemaps and HTML pages, you can collect large datasets for analysis. The approach can be easily adapted for different types of web scraping tasks, making it a versatile tool for data scientists.</p><p>With this data in CSV format, you can now perform further analysis using Python libraries like pandas or even load it into machine learning models for training.</p><h3>Key Takeaways:</h3><ul><li><strong>Web scraping</strong>: Automate data collection from web pages using requests and BeautifulSoup.</li><li><strong>XML parsing</strong>: Easily extract URLs from sitemaps using xml.etree.ElementTree.</li><li><strong>Error handling</strong>: Implement robust error handling to ensure your scraper continues running in case of failures.</li><li><strong>CSV storage</strong>: Store scraped data efficiently for further analysis.</li></ul><p>Whether you’re scraping data for machine learning projects or just exploring the web for insights, this approach is a valuable tool for your data science toolkit.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=007d7b47cd01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of Exploratory Data Analysis (EDA) in Product Data Science]]></title>
            <link>https://medium.com/@melissadunkeld/the-importance-of-exploratory-data-analysis-eda-in-product-data-science-fe5308965630?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe5308965630</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Sun, 29 Sep 2024 17:12:11 GMT</pubDate>
            <atom:updated>2024-09-29T17:12:11.213Z</atom:updated>
            <content:encoded><![CDATA[<p>In product data science, the journey from raw data to actionable insights is intricate and fascinating. At the heart of this journey lies a crucial step: Exploratory Data Analysis (EDA). EDA is the process of analysing datasets to summarise their main characteristics, often employing visual methods. It is an essential practice that enables product data scientists to understand the data at hand, uncover underlying patterns, and inform subsequent analysis. It’s akin to a detective’s investigation — understanding the evidence before drawing conclusions. Let’s delve into why EDA is indispensable in product data science.</p><h4>Uncovering Insights and Patterns</h4><p>EDA enables data scientists to get a preliminary understanding of the data’s structure, anomalies, and patterns. By examining data distributions, relationships, and variances, EDA helps uncover hidden insights that might not be immediately apparent. For example, visualising user behaviour on an app can reveal unexpected usage patterns, guiding product improvements and feature development.</p><h4>Identifying Data Quality Issues</h4><p>No data set is perfect. Missing values, outliers, and inconsistencies are common. EDA acts as a quality control measure, helping to identify and rectify these issues early in the analysis process. For product teams, ensuring data quality is critical as it affects the reliability of insights and the subsequent decisions made from those insights.</p><h4>Informing Hypotheses and Guiding Further Analysis</h4><p>EDA is the groundwork upon which hypotheses are built. By understanding the data’s underlying trends and anomalies, data scientists can formulate more accurate and testable hypotheses. In product development, this means focusing on the most promising areas for improvement or innovation. For instance, if EDA reveals that a particular feature is underutilised, further analysis can determine why and how to enhance it.</p><h4>Enhancing Model Selection and Accuracy</h4><p>In predictive modelling, EDA is crucial in feature selection and engineering. By understanding the relationships between variables, data scientists can select the most relevant features and create new ones that better capture the underlying patterns. This, in turn, improves the accuracy and performance of predictive models, which is essential for product recommendations, personalisation, and other data-driven features.</p><h4>Communicating Findings Effectively</h4><p>Visualisations are a crucial component of EDA, transforming raw data into understandable insights. For product managers and stakeholders who may not have a technical background, these visual representations make complex data more accessible. Effective communication of EDA findings ensures that all team members are aligned and can make informed decisions based on the data.</p><h4>Practical Examples of EDA in Product Data Science</h4><ol><li><strong>User Engagement Analysis</strong>: In a scenario where a product team wants to understand user engagement, EDA can reveal insights into usage patterns. By plotting session durations and frequency of visits, the team can identify key user segments and tailor their strategies accordingly.</li><li><strong>A/B Testing</strong>: When running A/B tests to compare different product features, EDA is crucial in examining the test results. Visualising the performance metrics of the control and treatment groups helps in assessing the impact of the changes and in making data-driven decisions.</li><li><strong>Churn Prediction</strong>: For predicting customer churn, EDA helps in identifying which features (e.g., frequency of use, customer support interactions) are most indicative of churn. This understanding can then be used to build more effective predictive models.</li></ol><h4>Example of some Basic EDA being Performed on the Telco Churn Dataset</h4><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br># Load the data<br>file_path = &#39;Telco_Churn.csv&#39;<br>data = pd.read_csv(file_path)<br><br># General Information<br>print(&quot;General Information&quot;)<br>print(data.info())<br><br># Summary Statistics<br>print(&quot;\nSummary Statistics&quot;)<br>print(data.describe())<br><br># Checking for missing values<br>print(&quot;\nMissing Values&quot;)<br>print(data.isnull().sum())<br><br># Checking for unique values in each column<br>print(&quot;\nUnique Values in Each Column&quot;)<br>print(data.nunique())<br><br># Distribution of Categorical Features<br>categorical_features = data.select_dtypes(include=[&#39;object&#39;]).columns<br>for feature in categorical_features:<br>    plt.figure(figsize=(10, 5))<br>    sns.countplot(data[feature])<br>    plt.title(f&#39;Distribution of {feature}&#39;)<br>    plt.xticks(rotation=45)<br>    plt.show()<br><br># Distribution of Numerical Features<br>numerical_features = data.select_dtypes(include=[np.number]).columns<br>data[numerical_features].hist(bins=15, figsize=(15, 10))<br>plt.suptitle(&#39;Distribution of Numerical Features&#39;)<br>plt.show()<br><br># Correlation Matrix<br>plt.figure(figsize=(12, 8))<br>correlation_matrix = data.corr()<br>sns.heatmap(correlation_matrix, annot=True, cmap=&#39;coolwarm&#39;)<br>plt.title(&#39;Correlation Matrix&#39;)<br>plt.show()<br><br># Pairplot of Numerical Features<br>sns.pairplot(data[numerical_features])<br>plt.suptitle(&#39;Pairplot of Numerical Features&#39;)<br>plt.show()<br><br># Churn Distribution<br>plt.figure(figsize=(6, 4))<br>sns.countplot(data[&#39;Churn&#39;])<br>plt.title(&#39;Churn Distribution&#39;)<br>plt.show()<br><br># Tenure vs Monthly Charges by Churn<br>plt.figure(figsize=(12, 6))<br>sns.scatterplot(x=&#39;tenure&#39;, y=&#39;MonthlyCharges&#39;, hue=&#39;Churn&#39;, data=data)<br>plt.title(&#39;Tenure vs Monthly Charges by Churn&#39;)<br>plt.show()<br><br># Contract Type vs Churn<br>plt.figure(figsize=(10, 5))<br>sns.countplot(x=&#39;Contract&#39;, hue=&#39;Churn&#39;, data=data)<br>plt.title(&#39;Contract Type vs Churn&#39;)<br>plt.show()<br><br># Monthly Charges Distribution by Churn<br>plt.figure(figsize=(10, 5))<br>sns.boxplot(x=&#39;Churn&#39;, y=&#39;MonthlyCharges&#39;, data=data)<br>plt.title(&#39;Monthly Charges Distribution by Churn&#39;)<br>plt.show()<br><br># Total Charges Distribution by Churn<br>plt.figure(figsize=(10, 5))<br>sns.boxplot(x=&#39;Churn&#39;, y=&#39;TotalCharges&#39;, data=data)<br>plt.title(&#39;Total Charges Distribution by Churn&#39;)<br>plt.show()<br><br># Senior Citizen vs Churn<br>plt.figure(figsize=(6, 4))<br>sns.countplot(x=&#39;SeniorCitizen&#39;, hue=&#39;Churn&#39;, data=data)<br>plt.title(&#39;Senior Citizen vs Churn&#39;)<br>plt.show()<br><br># Gender vs Churn<br>plt.figure(figsize=(6, 4))<br>sns.countplot(x=&#39;gender&#39;, hue=&#39;Churn&#39;, data=data)<br>plt.title(&#39;Gender vs Churn&#39;)<br>plt.show()</pre><h4>Conclusion</h4><p>Exploratory Data Analysis is not just a preliminary step in data analysis; it is a critical process that sets the stage for all subsequent analyses and decisions. For product data science teams, EDA provides the insights and understanding necessary to improve product features, enhance user experiences, and drive business growth. Ignoring EDA is akin to navigating without a map — possible but fraught with missed opportunities and potential pitfalls. By investing time and effort into EDA, product teams can ensure that their decisions are data-driven, accurate, and ultimately more effective.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe5308965630" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elevating Decisions, Driving Growth: Why B2B Product Teams Need Data Scientists.]]></title>
            <link>https://medium.com/@melissadunkeld/elevating-decisions-driving-growth-why-b2b-product-teams-need-data-scientists-6d8c2f01ecb0?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d8c2f01ecb0</guid>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[product-data-science]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Thu, 27 Jun 2024 07:51:21 GMT</pubDate>
            <atom:updated>2024-06-27T09:47:17.914Z</atom:updated>
            <content:encoded><![CDATA[<p>B2B companies are increasingly waking up to realise the growing role data plays in building products and strategies. It is no longer a trend to integrate data science into B2B product teams. It’s a necessity. This article delves deeper to showcase the main reasons B2B product teams need data scientists and how these data scientists could transform businesses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Bs9DqfpxRg-U713" /></figure><h3>Understanding the Role of Data Scientists</h3><p>What does a data scientist do before we dive into the details? These professionals harness statistical methods, algorithms, and machine-learning techniques to derive insights from data. Their work includes:</p><ul><li><strong>Data Collection and Cleanin</strong>g: this includes retrieving data from diverse sources and establishing quality data.</li><li><strong>Data Analysis</strong>: The process of interpreting data in terms of discovering hidden patterns, correlations, and trends.</li><li><strong>Predictive Modeling</strong>: Using current data to predict future trends and behaviour.</li><li><strong>Data Visualisation</strong>: Representing data in ways that will allow stakeholders to make sense of it and take action on that understanding.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*28Uk2lrhar01SgHj" /></figure><h3>Enhancing Decision-Making with Data-Driven Insights</h3><p>Decisions in the world of B2B are usually complex and hold high stakes for the company. Data scientists empower product teams to make informed decisions based on empirical evidence rather than intuition. Here’s how:</p><ol><li><strong>Market and customer insights</strong>: The data scientist is involved in analysing customer/platform data, which explains needs, preferences, and behaviours. This insight can be used to tailor products to meet specific market demands.</li><li><strong>Product Development</strong>: Data scientists guide the feature development process by understanding which features users find valuable most. This allows them to allocate resources efficiently toward building impactful features.</li><li><strong>Performance Metrics</strong>: A data scientist defines some key performance indicators in this section, and to measure the product’s success, he will use them. This is like an infinite feedback loop to be used to further refine and keep improving the products with time.</li></ol><h3>Driving Innovation and Competitive Advantage</h3><p>Driving Innovation and Competitive Advantage Innovation is a must in the B2B sector that is always looking to stay ahead, fuelled by high competition. Data scientists contribute by :</p><ol><li><strong>Opportunity Identification</strong>: This involves discovering new market opportunities and niches through data analysis. This makes companies innovate proactively rather than reactively.</li><li><strong>Personalised Experiences</strong>: data helps build a user experience that ensures customer satisfaction and loyalty in the B2B landscape.</li><li><strong>Improved operational efficiency</strong>: A data scientist optimises internal processes by analysing operational data to identify bottlenecks and suggest corrections. This usually reduces costs and improves productivity.</li></ol><h3>Mitigating Risks</h3><p>In this regard, risk management is one of the key B2B activities, and a number of how data scientists help in managing risk include:</p><ol><li><strong>Fraud Detection:</strong> Data scientists can detect unusual patterns that may indicate fraudulent activities using machine learning algorithms, thus protecting the company from potential losses.</li><li><strong>Market Risk Analysis</strong>: Predictions about market shifts are made by data scientists through market trends and past data to identify and manage market shifts.</li><li><strong>Reliability of Products</strong>: Analysis of product usage data can help identify problem areas on time so that the problems do not become crises in the reliability of products and, therefore, do not lead to customer dissatisfaction.</li></ol><h3>Fostering a Data-Driven Culture</h3><p>Integrating data scientists into product teams helps cultivate a data-driven culture within the organisation. This cultural shift is pivotal for:</p><p><strong>Encourage Collaborative Decision-Making: </strong>Encouraging teams to base their decisions on data drives collaboration and ensures that all decisions are attuned to the business strategy.</p><p><strong>Continuous learning:</strong> inspires a data-driven culture and encourages constant learning and improvement, where all teams continuously analyse data to learn from success and failure.</p><p><strong>Transparency and Accountability</strong>: Data transparency ensures everyone is on the same page, and measurable outcomes maintain accountability.</p><h4>Empowering Data Scientists to Maximise Their Impact</h4><p>To fully maximise the potential of data scientists, B2B companies need to empower these professionals with the right tools, resources, and environment. Ways to empower data scientists include:</p><p><strong>Access to Quality Data:</strong> The data accessible by data scientists must be rich and of high quality. Companies need to ensure good data infrastructure in terms of lakes, warehouses, etc., which will enable easy and sorted access to the data, with reduced time dedicated to cleaning and preparing it so that data scientists can focus on analysis and insights.</p><p><strong>Advanced Analytical Tools:</strong> state-of-the-art analytical tools and software must be provided to data scientists. These include Python, SQL, and other specialised machine-learning platforms, which allow data scientists to perform advanced analysis and be adept at developing predictive models. It can also lead to improved collaboration and knowledge sharing with these tools through collaborative platforms.</p><p><strong>Continuous Learning and Development: </strong>Data science is an evolving field; new techniques and technologies arise frequently. Therefore, companies should continue to invest in opportunities for training, workshops, and access to conferences in this particular industry. Development funds associated with research and employee training are essential. Encouraging data scientists to stay updated will enable them to apply state-of-the-art methods in solving business challenges.</p><p><strong>Collaboration:</strong> Data scientists are best placed in an environment where they are working with not only product managers and engineers but also other stakeholders. A culture of collaboration and open communication can make data scientists internalise business objectives and connect their analyses to the goals set at a more strategic level. This might happen through regular cross-functional meetings or collaborative projects.</p><p><strong>Autonomy and Power in Decision-making:</strong> Data scientists require freedom to some extent in their work so that they have a chance to look for novel solutions and move on their initiative. Empowered to give recommendations on ways decisions can be made with information available, which will go ahead to influence strategic business moves, is the value such a professional brings to the organisation. This autonomy raises the aspect of ownership and accountability concerning business outcomes.</p><p><strong>Clear metrics and goals:</strong> Agreed, clear, measurable goals for every data science effort track the progress of different projects and are meant to show value. KPIs must be set up for data-driven projects to ensure that variegated efforts, together with their corresponding values, stay aligned with business objectives and that there’s a framework for evaluating such success. Review these metrics regularly with stakeholders to keep everyone in the loop and engaged.</p><p><strong>Organisational Support:</strong> Data science cannot be integrated with the product team if the leadership does not support the idea. This support may be financial in terms of monetary investment, but more so, it is buying into a data-driven culture that operates in the organisation. They should push and sell the value of data science to become an influential essential tool within an organisation’s business realms.</p><p>Now, it will be from maximizing the result of analytics capabilities that B2B firms can accomplish by empowering their data scientists with the correct set of tools, environment, and support to enhance the effectiveness of data scientists. Business outcomes can only be improved and competitively sustained by B2B firms in the market with the aid of the above.</p><h3>Real-World Success Stories</h3><p>Several B2B businesses have already proven the value of embedding data scientists into product teams, delivering dramatic results. For instance, Salesforce: Using data science, Salesforce has powered up its CRM product with predictive analytics, and hence, businesses are empowered to discover and use information regarding customer needs to adapt and fine-tune sales processes.</p><p><strong>Slack:</strong> Data scientists at Slack analyze data about user engagement to keep refining and improving the platform, making it relevant for changing business needs.</p><p>HubSpot harnesses the power of data science to drive more effective campaigns and higher ROI for businesses</p><h3>Conclusion</h3><p>To sum up, data scientists in B2B product teams are beneficial and required. These people provide an analytical backbone for making decisions, driving innovation with mitigation of risks, and nurturing a data-driven culture. In essence, at this increased point of reliance upon and careful use of organizational data, the data scientist’s position becomes ever more inextricably linked to success within B2B product teams. Investment in data science is an investment in the future of your business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*102lzL6sLPYKjscr" /><figcaption>Dalle Generated Image so Not Perfect</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d8c2f01ecb0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Power of Explainable AI — Building Trust and Transparency with LIME and SHAP]]></title>
            <link>https://medium.com/@melissadunkeld/the-power-of-explainable-ai-building-trust-and-transparency-with-lime-and-shap-295948dee4ec?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/295948dee4ec</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Tue, 11 Jun 2024 12:02:37 GMT</pubDate>
            <atom:updated>2024-06-11T12:02:37.346Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Power of Explainable AI — Building Trust and Transparency with LIME and SHAP</h3><p>With every industry leveraging artificial intelligence (AI) and machine learning (ML) models, many businesses are on their journey with AI/ML, often used to make informed and/or automated decisions. As the technology and models become more widespread, understanding and retracing the decision-making processes of these AI solutions becomes increasingly important. This challenge can be addressed using explainable AI (XAI).</p><p>In this article, I’ll explore explainable AI, highlight its importance, and illustrate its objectives and benefits.</p><p>I’ll then provide an overview of how to implement two popular surrogate models, LIME and SHAP, using Python, which can help interpret machine learning models.</p><h3>So, What is Explainable AI?</h3><p>Explainable AI refers to a set of processes and methods that aim to provide clear and human-understandable explanations for the decisions generated by AI and ML models.</p><p>By integrating an explainability layer into these models, data scientists can create more trustworthy and transparent systems to assist a wide range of stakeholders, including themselves (data scientists) as developers, regulators, and end-users. With organisations like the ICO in the UK bringing out guidance for AI and fairness in relation to UK-GDPR, it’s becoming more critical to implement explainable AI in your workflows.</p><h3>Building Trust Through Explainable AI</h3><p>Here are some principles of explainable AI that contribute to building trust:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*8-du-B3liXO_MGWK.png" /><figcaption>Principles of Explainable AI</figcaption></figure><ul><li><strong>Transparency:</strong> Ensuring stakeholders understand the models’ decision-making process.</li><li><strong>Fairness:</strong> Ensuring that the models’ decisions are fair for everyone, including people in protected groups (race, religion, gender, disability, ethnicity).</li><li><strong>Trust:</strong> Assessing the confidence level of human users using the AI system.</li><li><strong>Robustness:</strong> Being resilient to changes in input data or model parameters, maintaining consistent and reliable performance even when faced with uncertainty or unexpected situations.</li><li><strong>Privacy:</strong> Guaranteeing the protection of sensitive user information.</li><li><strong>Interpretability:</strong> Providing human-understandable explanations for their predictions and outcomes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PvFan-3nN6QQmtRd" /><figcaption>Microsoft’s Responsible AI Principles</figcaption></figure><h3>Elements of Explainable AI</h3><p>Implementing explainable AI offers several benefits. It provides a clear understanding of the rationale behind AI-driven decisions for decision-makers and other stakeholders, enabling better-informed choices. It also helps identify potential biases or errors in the models, leading to more accurate and fair outcomes.</p><h3>Explainable AI Examples</h3><p>There are two broad categories of model explainability: model-specific methods and model-agnostic methods. Both techniques offer valuable insights into the inner workings of ML models while ensuring effectiveness and accountability.</p><h3>Model-Agnostic Methods</h3><p>These methods can be applied to any machine learning model, regardless of its structure or type. They focus on analysing the features’ input-output pairs. This section will introduce and discuss LIME and SHAP, two widely used surrogate models.</p><h4><strong>SHAP (SHapley Additive exPlanations)</strong></h4><p>SHAP aims to explain the prediction of an instance/observation by computing the contribution of each feature to the prediction. The code below walks through how to implement Shap using the Diabetes dataset.</p><pre># import the relevant libraries needed <br>import shap<br>from sklearn.ensemble import RandomForestRegressor<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import mean_squared_error, r2_score<br>from sklearn.datasets import load_diabetes<br>import pandas as pd<br>import matplotlib.pyplot as plt<br><br># Load the diabetes dataset from sklearn <br>diabetes = load_diabetes()<br>X, y = diabetes.data, diabetes.target<br>feature_names = diabetes.feature_names<br><br># Create a Pandas dataframe <br>df = pd.DataFrame(X, columns=feature_names)<br>df[&#39;target&#39;] = y<br><br># Split the dataset into training and test sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># Train a Random Forest Model <br>model = RandomForestRegressor(n_estimators=50, random_state=42)<br>model.fit(X_train, y_train)<br><br># Evaluate the model <br>y_pred = model.predict(X_test)<br>mse = mean_squared_error(y_test, y_pred)<br>r2 = r2_score(y_test, y_pred)<br>print(f&#39;Mean Squared Error: {mse}&#39;)<br>print(f&#39;R^2 Score: {r2}&#39;)<br><br># Explain the predictions using SHAP<br>explainer = shap.TreeExplainer(model)<br>shap_values = explainer.shap_values(X_test, check_additivity=False)<br><br># Initialize JavaScript visualisation library<br>shap.initjs()<br><br># Show the SHAP values for a test subject - Shows how each feature contributes to the prediction<br>index_to_explain = 7 # can change this <br>print(f&quot;Explaining prediction for index {index_to_explain}&quot;)<br>shap.force_plot(explainer.expected_value, shap_values[index_to_explain], features=X_test[index_to_explain], feature_names=feature_names)<br><br># Feature importance across all samples in the test set<br>shap.summary_plot(shap_values, X_test, feature_names=feature_names)<br><br># Dependence plot for a specific feature (&#39;bmi&#39;) highlighting interactions with other features + shap Values<br>shap.dependence_plot(&#39;bmi&#39;, shap_values, X_test, feature_names=feature_names)<br><br># Feature importance plot ranking features by their average impact on the model output<br>shap.summary_plot(shap_values, X_test, plot_type=&#39;bar&#39;, feature_names=feature_names)<br><br></pre><h4>LIME (Local Interpretable Model-agnostic Explanations)</h4><p>LIME focuses on explaining the model’s prediction for individual instances. The code below walks through how to implement LIME using the Diabetes dataset.</p><pre>import pandas as pd<br>import numpy as np<br>from sklearn.model_selection import train_test_split<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import accuracy_score<br>import lime<br>import lime.lime_tabular<br>import matplotlib.pyplot as plt<br><br># Load the Diabetes dataset from sklearn<br>from sklearn.datasets import load_diabetes<br><br>data = load_diabetes()<br>X = pd.DataFrame(data.data, columns=data.feature_names)<br>y = pd.Series(data.target)<br><br># Split the dataset into training and testing sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># Train a Random Forest Classifier<br>clf = RandomForestClassifier(n_estimators=50, random_state=42)<br>clf.fit(X_train, y_train)<br><br># Predict on the test set and evaluate<br>y_pred = clf.predict(X_test)<br>print(f&quot;Accuracy: {accuracy_score(y_test, y_pred)}&quot;)<br><br># Create a LIME explainer<br>explainer = lime.lime_tabular.LimeTabularExplainer(<br>    training_data=np.array(X_train),<br>    feature_names=X_train.columns,<br>    class_names=[&#39;Target&#39;],<br>    mode=&#39;regression&#39;<br>)<br><br># Choose an instance to explain<br>i = 7 # Example instance index<br>exp = explainer.explain_instance(X_test.iloc[i], clf.predict, num_features=5)<br><br># Display the explanation<br>exp.show_in_notebook(show_table=True, show_all=False)<br><br># Plot the explanation<br>fig = exp.as_pyplot_figure()<br>plt.show()</pre><h3>Model-Specific Methods</h3><p>LIME and SHAP can only be applied to specific models such as linear regression, decision trees, and neural networks. Techniques like DeepLIFT, Grad-CAM, or Integrated Gradients can explain deep-learning models.</p><p>However, a decision tree classifier can be visualised using inbuilt functionality from sci-kit-learn.</p><pre>import pandas as pd<br>import numpy as np<br>from sklearn.model_selection import train_test_split<br>from sklearn.tree import DecisionTreeClassifier, plot_tree<br>from sklearn.datasets import load_diabetes<br>import matplotlib.pyplot as plt<br><br># Load the Diabetes dataset from sklearn<br>data = load_diabetes()<br>X = pd.DataFrame(data.data, columns=data.feature_names)<br>y = pd.Series(data.target)<br><br># Split the dataset into training and testing sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># Train the Decision Tree model<br>dt_clf = DecisionTreeClassifier(max_depth=3, min_samples_leaf=2)<br>dt_clf.fit(X_train, y_train)<br><br># Plot the decision tree<br>fig = plt.figure(figsize=(25, 20))<br>_ = plot_tree(dt_clf, feature_names=X_train.columns, filled=True)<br>plt.show()</pre><h3>Challenges of Explainable AI</h3><p>As AI technology advances, understanding and interpreting algorithms to discern how they produce outcomes becomes more challenging. This is especially true for deep learning models, which are complex and often act like black boxes. Making these models understandable can reduce their accuracy and performance. Creating explanations that non-experts can understand is also difficult but necessary for trust. Ethical and privacy issues arise since detailed explanations could reveal sensitive information or proprietary algorithms.</p><h3>Conclusion</h3><p>This article provides an overview of explainable AI, highlighting principles that contribute to building trust and providing stakeholders with the skills to construct trustworthy models. I also covered model-agnostic and model-specific methods, focusing on LIME and SHAP. Furthermore, I discussed the challenges and limitations of explainable AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=295948dee4ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is AI?]]></title>
            <link>https://medium.com/@melissadunkeld/what-is-ai-5a22f1eb281e?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a22f1eb281e</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Thu, 06 Jun 2024 21:04:07 GMT</pubDate>
            <atom:updated>2024-06-06T21:04:07.754Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is AI? A Quick-Start Guide For Beginners</h3><p>Artificial intelligence (AI) has been a buzzword for a while now, but with the advent of tools like ChatGPT becoming part of everyday life, we’re entering a new era.</p><p><strong>But what exactly is AI? How does it work? And why does it matter?</strong></p><p>In this guide, we’ll explore the basics of AI, dispel common misconceptions, and provide resources for further learning.</p><h4>What is AI?</h4><p>Artificial Intelligence is a branch of computer science focused on creating systems that can perform tasks requiring human intelligence. These tasks include problem-solving, speech recognition, decision-making, and more. AI can be rule-based, operating under predefined conditions, or use machine learning to adapt and learn from data, making it versatile and capable of handling unforeseen scenarios.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*MzuksqQM6LIKzR7L.png" /></figure><h4>AI in Relation to Data Science and Other Key Concepts</h4><p>AI is an interdisciplinary field that intersects with data science, machine learning, and more. Understanding these connections is crucial to grasping how AI functions and its potential applications.</p><h4>Common Misconceptions</h4><p>It’s important to clarify what AI is not:</p><ol><li><strong>AI is not synonymous with robots</strong>: AI encompasses a broad range of technologies, not just robotics.</li><li><strong>AI will not surpass human intelligence soon</strong>: The concept of Artificial General Intelligence (AGI) remains theoretical and is far from being realized.</li><li><strong>AI does not “understand” content like humans</strong>: AI processes data based on patterns and lacks human-like comprehension.</li><li><strong>AI is not inherently unbiased</strong>: AI can inherit biases from its training data and designers.</li><li><strong>AI will not replace all human jobs</strong>: While AI can automate specific tasks, it cannot replicate human-specific skills like creativity and emotional intelligence.</li></ol><h4>AI Glossary:</h4><p>Here are some key AI terms and their meanings:</p><ul><li><strong>Algorithm</strong>: A set of instructions a computer follows to perform a task.</li><li><strong>Artificial General Intelligence (AGI)</strong>: A theoretical form of AI that can understand and learn across different domains.</li><li><strong>Deep Learning</strong>: A type of machine learning that mimics the human brain to learn from data.</li><li><strong>Machine Learning</strong>: A method that enables computers to learn from data and make decisions.</li><li><strong>Natural Language Processing (NLP)</strong>: AI that focuses on the interaction between computers and human language.</li><li><strong>Neural Network</strong>: A computing model inspired by the human brain, used in pattern recognition tasks.</li></ul><h4>Types of Artificial Intelligence</h4><p>AI can be categorized by its capabilities and functionalities:</p><p><strong>By Capabilities:</strong></p><ul><li><strong>Narrow AI</strong>: Designed for specific tasks, like reactive machines and limited memory machines.</li><li><strong>Artificial General Intelligence (AGI)</strong>: A theoretical AI that could understand and learn across various domains.</li><li><strong>Artificial Super Intelligence</strong>: An advanced form of AI that surpasses human intelligence, currently a concept more than a reality.</li></ul><p><strong>By Functionalities:</strong></p><ul><li><strong>Reactive Machines</strong>: Basic AI designed for specific tasks without the ability to learn from past experiences.</li><li><strong>Limited Memory</strong>: AI that can use past data to inform future decisions.</li><li><strong>Theory of Mind</strong>: AI that could potentially understand human emotions and thoughts.</li><li><strong>Self-Awareness</strong>: AI that would be aware of its existence and make decisions based on self-interest.</li></ul><h4>How Machine Learning is a Subset of AI</h4><p>Machine learning is a crucial subset of AI, focusing specifically on the development of algorithms that enable computers to learn from data. While AI encompasses the broader goal of creating systems that mimic human intelligence, machine learning hones in on the ability of these systems to learn and improve from experience.</p><p>Machine learning involves training algorithms on large datasets, allowing them to identify patterns and make predictions. For instance, a machine learning algorithm can be trained to recognize spam emails by analyzing thousands of examples and learning the characteristics of spam versus legitimate messages. Over time, the algorithm improves its accuracy without additional explicit programming.</p><p>Machine learning powers many of the AI applications we see today, from recommendation systems on Netflix to voice recognition in virtual assistants like Siri. It’s through machine learning that AI systems gain their “intelligence” and adaptability, making them more effective and efficient in performing tasks.</p><h4>Applications and Examples of AI</h4><p>AI is integrated into many aspects of our lives:</p><ul><li><strong>Everyday Technology</strong>: From Google Maps to virtual assistants like Siri and Alexa, AI powers many daily tools.</li><li><strong>Gaming and Entertainment</strong>: AI enhances video game experiences and provides content recommendations on platforms like Spotify and Netflix.</li></ul><h4>How Does AI Work?</h4><p>To understand AI, it helps to break down the process:</p><ol><li><strong>Data Collection</strong>: Gathering raw data, such as images, text, or human behaviour.</li><li><strong>Data Preparation</strong>: Cleaning and formatting data for AI systems.</li><li><strong>Choosing an Algorithm</strong>: Selecting a suitable algorithm based on the task.</li><li><strong>Training the Model</strong>: Feeding data into the algorithm to train the AI model.</li><li><strong>Testing the Model</strong>: Evaluating the model’s performance and making adjustments if necessary.</li><li><strong>Deployment</strong>: Implementing the trained model in real-world applications.</li><li><strong>Ongoing Learning</strong>: Allowing AI systems to learn and adapt over time for better performance.</li></ol><h4>Conclusion</h4><p>With this overview, you should have a clearer understanding of what AI is, how it works, and its potential applications. Whether you’re a beginner or looking to expand your knowledge, AI offers a wealth of opportunities for exploration and innovation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a22f1eb281e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tracking Changes in Your Machine Learning Pipeline with Model Drift Detection and Fine-Tuning]]></title>
            <link>https://medium.com/@melissadunkeld/tracking-changes-in-your-machine-learning-pipeline-with-model-drift-detection-and-fine-tuning-0cc27f41b3a7?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/0cc27f41b3a7</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machie-learning]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Wed, 29 May 2024 16:58:48 GMT</pubDate>
            <atom:updated>2024-05-29T16:59:21.904Z</atom:updated>
            <content:encoded><![CDATA[<p>Your role as a data scientist or machine learning engineer is crucial within machine learning or AI. Your expertise monitors model performance over time and adjusts models as new data becomes available.</p><p>This article demonstrates how to train a simple neural network for binary classification, detect data drift using NannyML, log experiments with MLFlow, and fine-tune the model on new data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-1ovHvcJJlfQzL0n" /></figure><p>We will cover:</p><ol><li>Building and training a simple neural network.</li><li>Detecting data drift.</li><li>Logging the model and metrics using MLFlow.</li><li>Fine-tuning the model with new data.</li></ol><h3>1. Building and Training a Simple Neural Network</h3><p>We start by generating synthetic data for demonstration purposes. The data is split into training and testing sets, and PyTorch DataLoader objects are created for efficient batch processing.</p><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>from torch.utils.data import DataLoader, TensorDataset<br>from sklearn.datasets import make_classification<br>from sklearn.model_selection import train_test_split<br>import numpy as np<br><br># Generate some synthetic data<br>X, y = make_classification(n_samples=1000, n_features=10, n_classes=2, random_state=42)<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># Convert to PyTorch tensors and create data loaders<br>train_data = TensorDataset(torch.tensor(X_train, dtype=torch.float32), torch.tensor(y_train, dtype=torch.long))<br>test_data = TensorDataset(torch.tensor(X_test, dtype=torch.float32), torch.tensor(y_test, dtype=torch.long))<br><br>train_loader = DataLoader(train_data, batch_size=64, shuffle=True)<br>test_loader = DataLoader(test_data, batch_size=64, shuffle=False)</pre><p>Next, we define a simple neural network with one hidden layer and train it for 10 epochs.</p><pre># Define a simple neural network<br>class SimpleNN(nn.Module):<br>    def __init__(self):<br>        super(SimpleNN, self).__init__()<br>        self.fc1 = nn.Linear(10, 64)<br>        self.relu = nn.ReLU()<br>        self.fc2 = nn.Linear(64, 2)  # Output layer for 2 classes<br><br>    def forward(self, x):<br>        x = self.relu(self.fc1(x))<br>        x = self.fc2(x)<br>        return x<br><br># Initialize model, loss function, and optimizer<br>model = SimpleNN()<br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.Adam(model.parameters(), lr=0.001)<br><br># Training loop<br>num_epochs = 10<br>for epoch in range(num_epochs):<br>    for inputs, labels in train_loader:<br>        optimizer.zero_grad()<br>        outputs = model(inputs)<br>        loss = criterion(outputs, labels)<br>        loss.backward()<br>        optimizer.step()<br>    print(f&#39;Epoch {epoch+1}/{num_epochs}, Loss: {loss.item()}&#39;)</pre><h3>2. Detecting Data Drift</h3><p><strong>Data drift</strong> refers to the change in data distribution over time, which can significantly impact the performance of machine learning models. It occurs when the statistical properties of the target variable, independent variables, or the relationship between them change. This can happen for various reasons, such as changes in user behaviour, external factors, or data collection processes.</p><ul><li><strong>Covariate Drift</strong>: Changes in the distribution of input features.</li><li><strong>Concept Drift</strong>: Changes in the relationship between input features and the target variable.</li></ul><p>Detecting data drift is crucial for maintaining model performance and making timely adjustments. We use NannyML, a library designed for post-deployment monitoring of machine learning models, to detect and visualise data drift.</p><pre>import nannyml as nml<br>import pandas as pd<br><br># Prepare data for drift calculation<br>reference_data = pd.DataFrame(X_train, columns=[f&#39;feature_{i}&#39; for i in range(X_train.shape[1])])<br>target_data = pd.DataFrame(X_test, columns=[f&#39;feature_{i}&#39; for i in range(X_test.shape[1])])<br><br># Add a timestamp column for drift calculation<br>reference_data[&#39;timestamp&#39;] = pd.to_datetime(&#39;2020-01-01&#39;)<br>target_data[&#39;timestamp&#39;] = pd.to_datetime(&#39;2020-02-01&#39;)<br><br># Calculate drift using NannyML<br>drift_calculator = nml.UnivariateDriftCalculator(column_names=reference_data.columns[:-1], timestamp_column_name=&#39;timestamp&#39;)<br>drift_calculator.fit(reference_data)<br>drift_results = drift_calculator.calculate(target_data)<br><br># Visualize drift results<br>drift_figures = drift_results.plot(kind=&#39;drift&#39;)<br>drift_figures.show()</pre><p>In this example, we calculate univariate drift for each feature between the training and testing datasets. The results can help identify which features have undergone significant changes and may require further investigation or model adjustment.</p><h3>3. Logging with MLFlow</h3><p>We use MLFlow to log the model and training metrics, ensuring reproducibility and easier tracking of experiments.</p><pre>import mlflow<br>import mlflow.pytorch<br><br># Set MLFlow tracking URI and experiment name<br>mlflow.set_tracking_uri(&#39;http://localhost:5000&#39;)   <br>mlflow.set_experiment(&#39;Simple_NN&#39;)<br><br># Start an MLFlow run and log the model<br>with mlflow.start_run(run_name=&#39;basic_nn&#39;):<br>    mlflow.pytorch.log_model(model, &#39;model&#39;)<br>    mlflow.log_param(&#39;epochs&#39;, num_epochs)<br>    mlflow.log_metric(&#39;final_loss&#39;, loss.item())</pre><h3>4. Fine-Tuning the Model</h3><p>When significant data drift is detected, retraining or fine-tuning the model is often necessary to maintain its performance. We generate new synthetic data, adjust the learning rate, and fine-tune the model for a few more epochs.</p><pre># Generate or load new data for fine-tuning<br>X_new, y_new = make_classification(n_samples=500, n_features=10, n_classes=2, random_state=42)<br>new_data = TensorDataset(torch.tensor(X_new, dtype=torch.float32), torch.tensor(y_new, dtype=torch.long))<br>new_data_loader = DataLoader(new_data, batch_size=64, shuffle=True)<br><br># Adjust the optimizer for fine-tuning<br>fine_tune_optimizer = optim.Adam(model.parameters(), lr=0.0001)<br><br># Fine-tuning loop<br>fine_tune_epochs = 5<br>for epoch in range(fine_tune_epochs):<br>    for inputs, labels in new_data_loader:<br>        fine_tune_optimizer.zero_grad()<br>        outputs = model(inputs)<br>        loss = criterion(outputs, labels)<br>        loss.backward()<br>        fine_tune_optimizer.step()<br>    print(f&#39;Fine-tuning Epoch {epoch+1}/{fine_tune_epochs}, Loss: {loss.item()}&#39;)</pre><p>Using MLflow, We then evaluate the fine-tuned model and log the results.</p><pre># Evaluate the fine-tuned model<br>correct = 0<br>total = 0<br>with torch.no_grad():<br>    for inputs, labels in test_loader:<br>        outputs = model(inputs)<br>        _, predicted = torch.max(outputs.data, 1)<br>        total += labels.size(0)<br>        correct += (predicted == labels).sum().item()<br><br>print(f&#39;Accuracy of the fine-tuned model on the test data: {100 * correct / total}%&#39;)<br><br># Log the fine-tuned model in MLFlow<br>with mlflow.start_run(run_name=&#39;basic_nn_fine_tuned&#39;):<br>    mlflow.pytorch.log_model(model, &#39;model_fine_tuned&#39;)<br>    mlflow.log_param(&#39;fine_tune_epochs&#39;, fine_tune_epochs)<br>    mlflow.log_metric(&#39;fine_tune_final_loss&#39;, loss.item())<br>    mlflow.log_metric(&#39;fine_tune_accuracy&#39;, 100 * correct / total)</pre><h4>Conclusion</h4><p>This article demonstrated how to build, train, and fine-tune a simple neural network for binary classification. We also showed how to detect data drift using NannyML and track experiments with MLFlow. These techniques are essential for maintaining robust and reliable machine-learning models in production environments.</p><p>By incorporating the techniques of data drift detection and model fine-tuning into your workflow, you can ensure that your models remain accurate and performant over time, adapting to new and changing data. This proactive approach helps mitigate the risks associated with model degradation and ensures the continuous delivery of high-quality predictions, making it a valuable addition to your machine-learning practices.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0cc27f41b3a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harnessing the Power of MLflow to Implement Online Learning for Real-Time Model Updating]]></title>
            <link>https://medium.com/@melissadunkeld/harnessing-the-power-of-mlflow-to-implement-online-learning-for-real-time-model-updating-4c03ce37bb5e?source=rss-dda11447d179------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c03ce37bb5e</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Melissa Dunkeld]]></dc:creator>
            <pubDate>Wed, 29 May 2024 16:40:19 GMT</pubDate>
            <atom:updated>2024-05-29T16:40:19.429Z</atom:updated>
            <content:encoded><![CDATA[<p>The ability to update models in real-time as new data emerges is not just advantageous—it’s essential. This strategy, known as online learning, empowers models to adapt continuously, ensuring they stay in sync with the latest data trends and maintain their accuracy and effectiveness.</p><p>In this article, we’ll explore how to implement an online learning pipeline using the River library and log the model updates with MLflow for tracking and reproducibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*etZ7hyATTTj_gUxH" /></figure><h3>Initialising MLflow</h3><p>We’ll start by setting up MLflow to track our experiments. MLflow provides a powerful interface for logging and tracking experiments, models, and other relevant metadata.</p><pre>import mlflow<br><br># Set the MLflow tracking URI to point to your tracking server<br>mlflow.set_tracking_uri(&#39;http://localhost:5000&#39;)  <br><br># Set the experiment name<br>experiment_name = &#39;ML_Experiment&#39;<br>mlflow.set_experiment(experiment_name)</pre><h3>Defining the Online Learning Pipeline</h3><p>We’ll define a simple pipeline using River, consisting of a standard scaler and a logistic regression model. River is a library designed for online learning, where models can be trained incrementally on mini-batches of data.</p><pre>from river import compose, preprocessing, linear_model<br><br># Define a pipeline with a standard scaler and logistic regression model<br>model = compose.Pipeline(<br>    preprocessing.StandardScaler(),<br>    linear_model.LogisticRegression()<br>)</pre><h3>Logging the Model to MLflow</h3><p>We need a function to serialise and log our model to MLflow periodically. This ensures we can track the model’s evolution over time and revert to previous versions if necessary.</p><pre>import pickle<br>import os<br><br>def log_model_to_mlflow(model, run_id):<br>    # Serialize the model using pickle<br>    serialized_model = pickle.dumps(model)<br>    <br>    # Save the serialized model to a file<br>    model_filename = &quot;online_model.pkl&quot;<br>    with open(model_filename, &quot;wb&quot;) as f:<br>        f.write(serialized_model)<br>    <br>    # Log the model file to MLflow<br>    mlflow.log_artifact(model_filename, artifact_path=&#39;model&#39;)<br>    <br>    # Clean up the model file to save space<br>    os.remove(model_filename)</pre><h3>Retrieving the Latest Model from MLflow</h3><p>We’ll periodically load the latest version from MLflow to ensure our model stays up-to-date. This is particularly useful in distributed systems where multiple instances of the model might be running and updating concurrently.</p><pre>def load_latest_model():<br>    client = mlflow.tracking.MlflowClient()<br>    experiment_id = client.get_experiment_by_name(experiment_name).experiment_id<br>    run_infos = client.search_runs([experiment_id])<br><br>    if run_infos:<br>        # Sort the runs by start time and get the latest one<br>        latest_run_info = max(run_infos, key=lambda run_info: run_info.info.start_time)<br>        latest_run = client.get_run(latest_run_info.info.run_id)<br><br>        try:<br>            # Download the latest model artifact<br>            local_path = client.download_artifacts(latest_run.info.run_id, &quot;model/online_model.pkl&quot;)<br>            with open(local_path, &quot;rb&quot;) as f:<br>                return pickle.load(f)<br>        except Exception as e:<br>            print(f&quot;Failed to download model artifact: {e}&quot;)<br>            return None<br>    else:<br>        return None</pre><h3>Simulating a Data Stream</h3><p>For demonstration purposes, we’ll use a synthetic data stream. This would be replaced with your actual data source in a real-world scenario. The synth.Agrawal dataset is commonly used for evaluating classification algorithms in a streaming context.</p><pre>from river.datasets import synth<br>from river.utils import VectorDict<br><br># Generate a synthetic data stream for the blog <br>data_stream = synth.Agrawal(classification_function=0, seed=42).take(1000)</pre><h3>Online Learning Loop</h3><p>Finally, we set up the online learning loop, where we continuously update the model with new data points and periodically log the updated model to MLflow. This loop ensures that the model remains current with the latest data trends.</p><pre># Initialize an MLFlow run outside the loop<br>with mlflow.start_run() as run:<br>    run_id = run.info.run_id<br><br>    # Online learning loop<br>    for i, (x, y) in enumerate(data_stream):<br>        # Convert x to a regular dict if it&#39;s a VectorDict<br>        if isinstance(x, VectorDict):<br>            x = dict(x)<br>        <br>        # Periodically load the latest model from MLFlow<br>        if i % 100 == 0:<br>            latest_model = load_latest_model()<br>            if latest_model:<br>                model = latest_model<br><br>        # Inference (test-then-train): Predict the label for the current instance<br>        y_pred = model.predict_one(x)<br><br>        # Update the model with the new data point<br>        model.learn_one(x, y)<br><br>        # Log the updated model to MLFlow periodically<br>        if i % 100 == 99:<br>            log_model_to_mlflow(model, run_id)<br><br>    # Log the final model state at the end of the loop<br>    log_model_to_mlflow(model, run_id)<br><br>print(&quot;Online learning and model updating completed.&quot;)</pre><h3>Conclusion</h3><p>This setup demonstrates a robust approach to implementing online learning with real-time model updates using River and MLflow. By continuously updating the model and logging the changes, we ensure our machine-learning models remain accurate and effective over time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c03ce37bb5e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>