<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sarowar Jahan Saurav on Medium]]></title>
        <description><![CDATA[Stories by Sarowar Jahan Saurav on Medium]]></description>
        <link>https://medium.com/@sarowar.saurav10?source=rss-991b7159b87b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qUntcyGXLlOsZOC4JdPFhQ.png</url>
            <title>Stories by Sarowar Jahan Saurav on Medium</title>
            <link>https://medium.com/@sarowar.saurav10?source=rss-991b7159b87b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:37:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sarowar.saurav10/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mastering MCP: The Game-Changing Model Context Protocol for Data Science and a Deep Dive into…]]></title>
            <link>https://medium.com/@sarowar.saurav10/mastering-mcp-the-game-changing-model-context-protocol-for-data-science-and-a-deep-dive-into-b61fcb11f2b2?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b61fcb11f2b2</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-ai]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Sat, 21 Jun 2025 07:01:59 GMT</pubDate>
            <atom:updated>2025-06-21T07:01:59.106Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mastering MCP: The Game-Changing Model Context Protocol for Data Science and a Deep Dive into Coding Examples!</h3><p>Hey there, data science enthusiasts! It’s me, diving into the electrifying world of AI innovation with a topic that’s setting the tech sphere ablaze: <strong>Model Context Protocol (MCP)</strong>! Buckle up, because this open-standard protocol is the USB-C of AI integrations, and it’s revolutionizing how we supercharge large language models (LLMs) with real-world data. In this blog, I’ll break down what MCP is, why it’s a total game-changer for data science, and drop a sizzling Python implementation to get you pumped. Let’s ignite this journey!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*8exxSMwI-IfucDJof3rwPQ.jpeg" /></figure><h3>What’s MCP? The Hottest Protocol in AI Town!</h3><p>Picture this: your AI model is a genius, but it’s stuck in a silo, cut off from the juicy, real-time data it craves. Enter <strong>Model Context Protocol (MCP)</strong>, launched by Anthropic in November 2024, a futuristic standard that’s like a universal adapter for AI agents. MCP creates a seamless, standardized bridge between LLMs and external tools, databases, and APIs, obliterating the messy, custom integrations of the past.</p><p>MCP’s architecture is pure brilliance:</p><ul><li><strong>MCP Hosts</strong>: The AI apps (like Claude Desktop or your custom data science dashboard) that need data to dazzle.</li><li><strong>MCP Clients</strong>: The middlemen that connect hosts to servers, ensuring secure, smooth communication.</li><li><strong>MCP Servers</strong>: Lightweight programs exposing tools, resources, or prompts, from GitHub to PostgreSQL to your local Jupyter notebooks.</li></ul><p>Think of MCP as the ultimate API orchestrator, slashing the dreaded “M×N problem” (where every AI app needs custom code for every tool) into a sleek “M+N” solution. It’s open-source, model-agnostic, and screaming with potential to make your AI workflows scalable, secure, and downright futuristic</p><h3>Why MCP is a Data Science Superstar</h3><p>Data science thrives on context — real-time, structured, and actionable. MCP is the secret sauce that empowers LLMs to tap into dynamic data sources, making your models smarter, faster, and more impactful. Here’s why MCP is the MVP for data scientists:</p><ol><li><strong>Real-Time Data Access</strong>: MCP lets your LLM query live databases or APIs, so your models aren’t stuck with stale training data. Imagine analyzing stock market trends or customer sentiment on the fly!</li><li><strong>Seamless Tool Integration</strong>: Connect your Python scripts, Jupyter notebooks, or visualization tools like Plotly to your AI, automating end-to-end workflows.</li><li><strong>Scalable Automation</strong>: MCP’s standardized interface means you can plug in new tools without rewriting code, perfect for scaling data pipelines.</li><li><strong>Context-Rich Insights</strong>: By feeding LLMs with structured data from knowledge bases or vector stores, MCP turbocharges retrieval-augmented generation (RAG) for deeper insights.</li><li><strong>Collaboration Nirvana</strong>: Share MCP servers across teams, enabling multi-agent systems to work harmoniously on complex data tasks.</li></ol><h3>Use Cases That’ll Blow Your Data Science Mind</h3><p>MCP is a Swiss Army knife for data science, unlocking use cases that make your heart race. Here are some jaw-dropping applications:</p><ul><li><strong>Automated Data Exploration</strong>: Let your LLM query a PostgreSQL database via an MCP server to uncover hidden patterns, generating SQL queries and visualizations autonomously.</li><li><strong>Dynamic RAG Pipelines</strong>: Supercharge RAG by connecting your LLM to a vector database (like Pinecone) through an MCP server, retrieving relevant documents for razor-sharp predictions.</li><li><strong>Real-Time Sentiment Analysis</strong>: Integrate MCP with social media APIs to fetch live posts, letting your AI analyze customer sentiment and generate actionable reports.</li><li><strong>Code-to-Insight Workflows</strong>: Use MCP to link GitHub repos with your LLM, enabling it to analyze code, suggest optimizations, and document findings in Notion.</li><li><strong>Multi-Agent Data Orchestration</strong>: Deploy multiple AI agents, each using MCP to access different tools (e.g., one for data cleaning, another for modeling), collaborating in a shared workspace.</li></ul><h3>Let’s Get Coding: Building an MCP Server for Data Science</h3><p>Enough hype — let’s dive into the code! Below, I’m sharing a Python implementation of a simple MCP server that lets an LLM perform basic statistical analysis on a dataset. This server exposes a tool to calculate mean, median, and standard deviation, perfect for quick data insights.</p><h3>Step 1: Setting Up the Environment</h3><p>We’ll use the fastmcp Python library to create our MCP server. First, ensure you have Python 3.10+ and install uv (a lightning-fast package manager):</p><pre>curl -LsSf https://astral.sh/uv/install.sh | sh<br>uv venv<br>source .venv/bin/activate<br>uv pip install fastmcp pandas</pre><h3>Step 2: Creating the MCP Server</h3><p>Here’s the code for our MCP server, stats_server.py. It defines a tool called calculate_stats that takes a list of numbers and returns statistical measures.</p><pre>from fastmcp import FastMCP<br>import pandas as pd<br>from typing import List, Dict</pre><pre># Initialize the MCP server<br>mcp = FastMCP(&quot;StatsServer&quot;)</pre><pre># Define a tool for statistical analysis<br>@mcp.tool()<br>def calculate_stats(data: List[float]) -&gt; Dict[str, float]:<br>    &quot;&quot;&quot;<br>    Calculate mean, median, and standard deviation of a dataset.<br>    Args:<br>        data: A list of numerical values<br>    Returns:<br>        A dictionary with mean, median, and std_dev<br>    &quot;&quot;&quot;<br>    df = pd.DataFrame(data, columns=[&#39;values&#39;])<br>    return {<br>        &#39;mean&#39;: float(df[&#39;values&#39;].mean()),<br>        &#39;median&#39;: float(df[&#39;values&#39;].median()),<br>        &#39;std_dev&#39;: float(df[&#39;values&#39;].std())<br>    }</pre><pre># Start the server<br>if __name__ == &quot;__main__&quot;:<br>    mcp.run()</pre><h3>Step 3: Running and Testing the Server</h3><p>Save the code as stats_server.py and run it:</p><pre>python stats_server.py</pre><p>This starts the MCP server, ready to connect with an MCP client (like Claude Desktop). To test it, you’d typically use the MCP CLI’s Inspector (requires Node.js), but for simplicity, assume an AI client sends a request like: “Calculate stats for [1, 2, 3, 4, 5]”. The server responds with:</p><pre>{<br>  &quot;mean&quot;: 3.0,<br>  &quot;median&quot;: 3.0,<br>  &quot;std_dev&quot;: 1.4142135623730951<br>}</pre><h3>Step 4: Connecting to an AI Client</h3><p>In a real-world setup, you’d connect this server to an MCP client integrated with an LLM (e.g., Claude or a custom Streamlit app). The LLM could ask, “What’s the average of my sales data?” and the client would invoke calculate_stats on the server, delivering instant insights.</p><h3>Why This Code Rocks</h3><ul><li><strong>Simplicity</strong>: The fastmcp library abstracts away protocol complexities, letting you focus on building tools.</li><li><strong>Flexibility</strong>: Swap out calculate_stats for more advanced functions, like regression analysis or anomaly detection.</li><li><strong>Scalability</strong>: Deploy this server on a cloud instance for team-wide access, amplifying collaboration.</li></ul><h3>Example 1:</h3><pre>import fastmcp<br>import pandas as pd<br>from typing import List, Dict, Union<br>from textblob import TextBlob  # For sentiment analysis (mocking LLM-like behavior)<br>import json<br><br># 🚀 Initialize the MCP server - the heart of our AI-powered data science magic!<br>mcp = fastmcp.FastMCP(&quot;SentimentAnalysisServer&quot;)<br><br># 🎉 Define a tool for sentiment analysis - let the LLM tap into this bad boy!<br>@mcp.tool()<br>def analyze_sentiment(texts: List[str]) -&gt; List[Dict[str, Union[str, float]]]:<br>    &quot;&quot;&quot;<br>    Analyze sentiment of a list of texts, returning polarity and subjectivity.<br>    Args:<br>        texts: List of text strings to analyze<br>    Returns:<br>        List of dictionaries with text, sentiment (positive/negative/neutral), polarity, and subjectivity<br>    &quot;&quot;&quot;<br>    results = []<br>    for text in texts:<br>        # Use TextBlob as a stand-in for LLM sentiment analysis (in production, swap with real LLM API)<br>        blob = TextBlob(text)<br>        polarity = blob.sentiment.polarity  # Range: [-1, 1]<br>        subjectivity = blob.sentiment.subjectivity  # Range: [0, 1]<br>        # Classify sentiment based on polarity<br>        sentiment = &quot;positive&quot; if polarity &gt; 0 else &quot;negative&quot; if polarity &lt; 0 else &quot;neutral&quot;<br>        results.append({<br>            &quot;text&quot;: text,<br>            &quot;sentiment&quot;: sentiment,<br>            &quot;polarity&quot;: polarity,<br>            &quot;subjectivity&quot;: subjectivity<br>        })<br>    return results<br><br># 🔌 Mock LLM client to simulate interaction with the MCP server<br>class MockLLMClient:<br>    def __init__(self, mcp_server):<br>        self.mcp_server = mcp_server<br><br>    def process_query(self, user_query: str, texts: List[str]) -&gt; Dict:<br>        &quot;&quot;&quot;<br>        Simulate an LLM processing a user query by calling the MCP server&#39;s sentiment tool.<br>        Args:<br>            user_query: The user&#39;s natural language query<br>            texts: List of texts to analyze<br>        Returns:<br>            Dictionary with the query and MCP tool results<br>        &quot;&quot;&quot;<br>        # Simulate LLM parsing the query (in reality, LLM would parse and decide to call MCP)<br>        if &quot;sentiment&quot; in user_query.lower():<br>            # Call the MCP server&#39;s sentiment analysis tool<br>            result = self.mcp_server.call_tool(&quot;analyze_sentiment&quot;, {&quot;texts&quot;: texts})<br>            return {<br>                &quot;query&quot;: user_query,<br>                &quot;results&quot;: result<br>            }<br>        return {&quot;query&quot;: user_query, &quot;results&quot;: &quot;No sentiment analysis requested&quot;}<br><br># 🌟 Main function to tie it all together<br>def main():<br>    # Sample data - imagine these are customer reviews from a database<br>    sample_texts = [<br>        &quot;This product is absolutely amazing, I love it!&quot;,<br>        &quot;Terrible experience, would not recommend.&quot;,<br>        &quot;It&#39;s okay, nothing special but gets the job done.&quot;<br>    ]<br><br>    # Create a mock LLM client<br>    llm_client = MockLLMClient(mcp)<br><br>    # Simulate a user query to the LLM<br>    user_query = &quot;Analyze the sentiment of these customer reviews&quot;<br>    print(f&quot;🔥 User Query: {user_query}&quot;)<br>    response = llm_client.process_query(user_query, sample_texts)<br>    <br>    # Pretty-print the results<br>    print(&quot;\n🎉 MCP + LLM Results:&quot;)<br>    print(json.dumps(response, indent=2))<br><br>    # Start the MCP server (in production, run this in a separate process)<br>    print(&quot;\n🚀 Starting MCP Server...&quot;)<br>    mcp.run()<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h3>How It Works</h3><ol><li><strong>MCP Server</strong>: The script uses fastmcp to create an MCP server named SentimentAnalysisServer. It exposes a tool, analyze_sentiment, which takes a list of texts and returns sentiment analysis results (polarity, subjectivity, and sentiment label).</li><li><strong>Sentiment Analysis</strong>: I used TextBlob as a stand-in for an LLM’s sentiment analysis capability to keep the example self-contained. In a real-world scenario, you’d replace this with a call to an LLM API (e.g., Claude or Grok via MCP client).</li><li><strong>Mock LLM Client</strong>: The MockLLMClient simulates an LLM interacting with the MCP server, parsing a user query and invoking the analyze_sentiment tool if relevant.</li><li><strong>Output</strong>: The script processes sample customer reviews, calls the MCP tool, and prints the results in JSON format.</li></ol><h3>Setup Instructions</h3><ol><li>Install dependencies</li></ol><pre>curl -LsSf https://astral.sh/uv/install.sh | sh<br>uv venv<br>source .venv/bin/activate<br>uv pip install fastmcp textblob</pre><p>2. Run the script</p><pre>python mcp_llm_sentiment.py</pre><h3>Sample Output</h3><pre>🔥 User Query: Analyze the sentiment of these customer reviews<br><br>🎉 MCP + LLM Results:<br>{<br>  &quot;query&quot;: &quot;Analyze the sentiment of these customer reviews&quot;,<br>  &quot;results&quot;: [<br>    {<br>      &quot;text&quot;: &quot;This product is absolutely amazing, I love it!&quot;,<br>      &quot;sentiment&quot;: &quot;positive&quot;,<br>      &quot;polarity&quot;: 0.6125,<br>      &quot;subjectivity&quot;: 0.75<br>    },<br>    {<br>      &quot;text&quot;: &quot;Terrible experience, would not recommend.&quot;,<br>      &quot;sentiment&quot;: &quot;negative&quot;,<br>      &quot;polarity&quot;: -0.6,<br>      &quot;subjectivity&quot;: 0.7<br>    },<br>    {<br>      &quot;text&quot;: &quot;It&#39;s okay, nothing special but gets the job done.&quot;,<br>      &quot;sentiment&quot;: &quot;positive&quot;,<br>      &quot;polarity&quot;: 0.25,<br>      &quot;subjectivity&quot;: 0.5<br>    }<br>  ]<br>}<br><br>🚀 Starting MCP Server...</pre><h3>Why This Adds Value</h3><ul><li><strong>Buzz-Worthy Integration</strong>: Combines MCP’s seamless tool access with LLM intelligence, perfect for data science tasks like sentiment analysis.</li><li><strong>Scalable</strong>: Swap TextBlob for a real LLM API or add more tools (e.g., data cleaning, visualization) to the MCP server.</li><li><strong>Real-World Ready</strong>: Mimics a production setup where an LLM queries live data via MCP, ideal for dashboards, RAG, or automated reporting.</li></ul><h3>The Future of MCP in Data Science</h3><p>MCP is just getting started, and its open-source ecosystem is exploding with over 1,000 community-built servers by February 2025! From integrating with Snowflake for big data analytics to powering real-time dashboards with Streamlit, MCP is poised to redefine how data scientists wield AI. Its model-agnostic design means you’re never locked into one LLM provider, giving you the freedom to innovate.</p><h3>Final Thoughts: Join the MCP Revolution!</h3><p>MCP is the spark that’s igniting a new era of context-aware, intelligent AI systems. For data scientists, it’s a golden ticket to automate, scale, and amplify your workflows with unparalleled ease. Whether you’re exploring datasets, building RAG pipelines, or orchestrating multi-agent systems, MCP is your trusty sidekick.</p><p><strong>References</strong>:</p><ul><li>Anthropic’s MCP Documentation</li><li>DataCamp’s MCP Guide</li><li>Towards Data Science MCP Tutorial</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b61fcb11f2b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Next-Gen E-commerce Product Recommendations Model with Deep Interest Evolution Network…]]></title>
            <link>https://medium.com/@sarowar.saurav10/build-your-next-gen-e-commerce-product-recommendations-model-with-deep-interest-evolution-network-79d575295c03?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/79d575295c03</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Wed, 02 Apr 2025 08:21:08 GMT</pubDate>
            <atom:updated>2025-04-02T08:21:08.975Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build Your Next-Gen E-commerce Product Recommendations Model with Deep Interest Evolution Network (DIEN)</h3><h3><strong>Introduction</strong></h3><p><strong>In a world where e-commerce giants like Alibaba and Amazon make billions of personalized recommendations daily, staying at the cutting edge of recommendation algorithms is crucial. The Deep Interest Evolution Network (DIEN) is one such advanced architecture pioneered by Alibaba. It addresses a critical challenge in recommendation: effectively modeling the dynamic and evolving nature of user interests.</strong></p><p>In traditional deep learning recommenders, we might capture user interactions via embeddings or RNNs (Recurrent Neural Networks). However, user preferences don’t stay static—someone searching for a “wireless keyboard” might later switch interests to “gaming headphones” or “ergo chairs.” DIEN was introduced to handle these shifting patterns more accurately, driving a more engaging and profitable recommendation system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3x8a5Ou2ajNZA8ulXPU0xg.png" /></figure><h4>In this blog post, we’ll:</h4><ol><li><strong><em>Summarize DIEN and its key components.</em></strong></li></ol><p><strong><em>2. Walk through a simplified example using PyTorch and a publicly available dataset from Kaggle.</em></strong></p><p><strong><em>3. Show how to evaluate the recommendations.</em></strong></p><h3><strong>1. What is Deep Interest Evolution Network (DIEN)?</strong></h3><p><strong>Deep Interest Evolution Network (DIEN) is an enhancement of Alibaba’s earlier architecture, Deep Interest Network (DIN). While DIN focuses on attention-based user interest extraction, DIEN extends this further by introducing a GRU-based interest evolution mechanism to capture how user interests change over time.</strong></p><h4><strong>Key Components of DIEN:</strong></h4><ol><li>Local Activation Unit (LAU): Learns to “activate” the most relevant user behavior subsequence for a given target item.</li><li>Interest Extractor: Uses GRUs to model the user’s historical behavior sequence.</li><li>Interest Evolving Layer: Applies an auxiliary loss and another GRU to track how interest transitions over time, allowing the model to learn fine-grained evolution.</li></ol><h4><strong>Why It Matters:</strong></h4><ol><li>Dynamic Behavior: Users often explore multiple product categories or shift focus. DIEN’s multi-stage GRUs adapt to these changes.</li><li>Attention Mechanism: It explicitly learns which past behaviors are most relevant for the current product recommendation.</li><li>Improved Performance: Alibaba reported significant improvements in both offline metrics and online CTR gains using DIEN.</li></ol><h3><strong>2. Dataset: Retailrocket Recommender Dataset (Kaggle)</strong></h3><p>For illustration, we’ll use the <strong>Retailrocket Recommender Dataset</strong> from Kaggle. This dataset contains user behavior logs including page views, product categories, and purchases, which we can transform into user-behavior sequences.</p><ul><li><strong>Kaggle Link</strong>: Retailrocket Recommender System Dataset — <a href="https://www.kaggle.com/retailrocket/ecommerce-dataset">https://<strong>www.kaggle.com</strong>/retailrocket/ecommerce-dataset</a></li></ul><h4><strong>Dataset Overview</strong>:</h4><ul><li><strong>Events</strong> file: includes (timestamp, visitorid, event, itemid, transactionid)</li><li><strong>Category Tree</strong> file: includes category hierarchy.</li><li><strong>Item Properties</strong> file: includes item attributes.</li></ul><p>Although DIEN is typically used with more complex user signals (like clicks, add-to-carts, purchases, etc.), we can adapt the event logs from this dataset to build a sequence for each user.</p><h3>3. Data Preparation Steps</h3><p><strong>Download the dataset</strong> from Kaggle and place it in your working directory. You’ll have CSV files like events.csv, item_properties_part1.csv, etc.</p><p><strong>Filter events</strong> to keep a subset of the data for demonstration (e.g., last 30 days of logs or a smaller random sample if you’re short on memory).</p><p><strong>Generate user sequence data</strong>:</p><ul><li>Sort events by visitorid and timestamp.</li><li>Group by visitorid. For each user, create a chronological list of (item_id, event_type) pairs or simply item_id for now.</li><li>Make sure to keep a small portion of the last item(s) for each user as the “label” or future event to predict.</li></ul><p><strong>Encode items</strong>:</p><ul><li>Map each unique item ID to an integer index (e.g., 0 to N-1).</li><li>You may also build embeddings for categories or other item properties, if desired.</li></ul><p><strong>Train-Test Split</strong>:</p><ul><li><strong>Train</strong>: Use the sequence except for the final interaction of each user.</li><li><strong>Test</strong>: The final interaction (or final few) for each user. You can then evaluate whether your model can predict the correct next item(s).</li></ul><h3>4. Simplified DIEN-Like Architecture in PyTorch</h3><p>Below is an example of how you might implement a scaled-down DIEN-like model in PyTorch. We’ll focus on the <strong>interest extractor</strong> using a GRU (or two GRUs) and an <strong>attention mechanism</strong>. We won’t replicate all the complexities (like auxiliary losses), but this should demonstrate the core idea.</p><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>from torch.utils.data import Dataset, DataLoader<br><br># -----------------------------<br># 1. Dataset &amp; Utility Classes<br># -----------------------------<br>class RetailRocketDataset(Dataset):<br>    def __init__(self, user_sequences, item2idx, max_seq_len=20):<br>        &quot;&quot;&quot;<br>        user_sequences: list of (user_id, [item_id_sequence], next_item_id)<br>        item2idx: dictionary mapping item_id -&gt; index<br>        max_seq_len: how many events we keep for each user (truncate or pad)<br>        &quot;&quot;&quot;<br>        self.samples = []<br>        self.max_seq_len = max_seq_len<br>        self.item2idx = item2idx<br>        <br>        for user_id, seq, label in user_sequences:<br>            self.samples.append((user_id, seq, label))<br>    <br>    def __len__(self):<br>        return len(self.samples)<br>    <br>    def __getitem__(self, idx):<br>        user_id, seq, label = self.samples[idx]<br>        # Convert item IDs to indices<br>        seq_idx = [self.item2idx.get(item, 0) for item in seq[-self.max_seq_len:]]<br>        <br>        # Pad sequence if needed<br>        if len(seq_idx) &lt; self.max_seq_len:<br>            seq_idx = [0]*(self.max_seq_len - len(seq_idx)) + seq_idx<br>        <br>        target_idx = self.item2idx.get(label, 0)<br>        <br>        return torch.tensor(seq_idx, dtype=torch.long), torch.tensor(target_idx, dtype=torch.long)<br><br># -------------<br># 2. DIEN Model<br># -------------<br>class AttentionUnit(nn.Module):<br>    &quot;&quot;&quot;<br>    A simplified attention mechanism for extracting relevant user behavior <br>    with respect to a target item.<br>    &quot;&quot;&quot;<br>    def __init__(self, embedding_dim):<br>        super(AttentionUnit, self).__init__()<br>        self.linear_item = nn.Linear(embedding_dim, embedding_dim)<br>        self.linear_seq = nn.Linear(embedding_dim, embedding_dim)<br>        self.softmax = nn.Softmax(dim=1)<br><br>    def forward(self, seq_embeddings, target_embedding):<br>        # seq_embeddings: (batch_size, seq_len, embedding_dim)<br>        # target_embedding: (batch_size, embedding_dim)<br>        <br>        # Expand target to match sequence length<br>        target_expanded = self.linear_item(target_embedding).unsqueeze(1)  # (batch_size, 1, embedding_dim)<br>        seq_transformed = self.linear_seq(seq_embeddings)                  # (batch_size, seq_len, embedding_dim)<br>        <br>        # Compute attention scores<br>        attn_scores = torch.bmm(seq_transformed, target_expanded.transpose(1,2)).squeeze(-1)  <br>        # (batch_size, seq_len)<br>        <br>        attn_weights = self.softmax(attn_scores)  # (batch_size, seq_len)<br>        <br>        # Weighted sum<br>        attn_output = torch.bmm(attn_weights.unsqueeze(1), seq_embeddings).squeeze(1)  <br>        # (batch_size, embedding_dim)<br>        <br>        return attn_output, attn_weights<br><br>class DIEN(nn.Module):<br>    def __init__(self, num_items, embedding_dim=32, hidden_size=32):<br>        super(DIEN, self).__init__()<br>        self.item_embedding = nn.Embedding(num_items, embedding_dim)<br>        <br>        # GRU for interest extraction<br>        self.gru = nn.GRU(input_size=embedding_dim, hidden_size=hidden_size, batch_first=True)<br>        <br>        # Attention to focus on relevant part of user sequence given the candidate item<br>        self.attention = AttentionUnit(hidden_size)<br>        <br>        # Final prediction layer<br>        self.output = nn.Linear(hidden_size, num_items)<br><br>    def forward(self, seq_items, candidate_items):<br>        &quot;&quot;&quot;<br>        seq_items: (batch_size, seq_len) - user behavior sequence<br>        candidate_items: (batch_size,)   - the &quot;label&quot; or next item<br>        &quot;&quot;&quot;<br>        # Embed user sequence<br>        seq_emb = self.item_embedding(seq_items)  # (batch_size, seq_len, embedding_dim)<br>        <br>        # Embed candidate item<br>        cand_emb = self.item_embedding(candidate_items)  # (batch_size, embedding_dim)<br>        <br>        # GRU-based interest extraction<br>        gru_out, _ = self.gru(seq_emb)  # (batch_size, seq_len, hidden_size)<br>        <br>        # Attention over GRU output<br>        attn_output, attn_weights = self.attention(gru_out, cand_emb)<br>        <br>        # We’ll use the attention output to predict the next item<br>        logits = self.output(attn_output)  # (batch_size, num_items)<br>        <br>        return logits, attn_weights<br><br># -------------------<br># 3. Training Script<br># -------------------<br>def train_model(model, train_loader, num_epochs=5, lr=1e-3, device=&#39;cpu&#39;):<br>    criterion = nn.CrossEntropyLoss()<br>    optimizer = optim.Adam(model.parameters(), lr=lr)<br>    <br>    model.to(device)<br>    <br>    for epoch in range(num_epochs):<br>        model.train()<br>        total_loss = 0<br>        for seq_batch, label_batch in train_loader:<br>            seq_batch = seq_batch.to(device)<br>            label_batch = label_batch.to(device)<br>            <br>            optimizer.zero_grad()<br>            logits, _ = model(seq_batch, label_batch)<br>            <br>            loss = criterion(logits, label_batch)<br>            loss.backward()<br>            optimizer.step()<br>            <br>            total_loss += loss.item()<br>        <br>        print(f&quot;Epoch [{epoch+1}/{num_epochs}], Loss: {total_loss/len(train_loader):.4f}&quot;)<br><br># Example usage (pseudo-code / demonstration):<br>if __name__ == &quot;__main__&quot;:<br>    # Suppose we have user_sequences like:<br>    # [(user_id, [item_ids...], next_item_id), ...]<br>    # We skip the actual data loading / preprocessing steps for brevity.<br>    fake_user_sequences = [<br>        (1, [10, 11, 12], 13),<br>        (2, [2, 3, 7], 3),<br>        (3, [8, 8, 10, 11], 12),<br>        # ... more<br>    ]<br>    <br>    # Build item2idx (For demonstration, we assume items range from 0..15)<br>    item2idx = {i: i for i in range(16)}  # trivial example<br>    <br>    dataset = RetailRocketDataset(fake_user_sequences, item2idx, max_seq_len=4)<br>    dataloader = DataLoader(dataset, batch_size=2, shuffle=True)<br>    <br>    model = DIEN(num_items=16, embedding_dim=8, hidden_size=8)<br>    train_model(model, dataloader, num_epochs=3)</pre><h3>Explanation of Key Parts</h3><ol><li><strong>RetailRocketDataset</strong>: Custom Dataset that transforms user sequences into fixed-length arrays for training.</li><li><strong>AttentionUnit</strong>: A simplified attention mechanism to highlight the relevant parts of the GRU output.</li><li><strong>CrossEntropyLoss</strong>: Used for multi-class classification (predicting one item out of num_items).</li><li><strong>DIEN Model</strong>:</li></ol><ul><li><strong>GRU</strong> to extract latent interest from the sequence.</li><li><strong>Attention</strong> to adapt these interests to the current target item context.</li></ul><h3>5. Evaluation</h3><p>After training, you can generate <strong>top-K</strong> recommendations by:</p><ol><li>For each user’s sequence, pass it through the model for <strong>all candidate items</strong> (or a subset).</li><li>Rank items by the predicted score logits[:, item_id].</li><li>Compare the recommended items with the actual next item or purchase event.</li></ol><p>Common metrics include:</p><ul><li><strong>Precision@K</strong></li><li><strong>Recall@K</strong></li><li><strong>Normalized Discounted Cumulative Gain (NDCG)@K</strong></li></ul><h4><strong>For instance:</strong></h4><pre>def evaluate_topk(model, test_dataset, k=10):<br>    model.eval()<br>    correct = 0<br>    total = 0<br>    <br>    for seq_batch, label_batch in test_dataset:<br>        seq_batch = seq_batch.unsqueeze(0)  # batch_size=1 for demonstration<br>        label_batch = label_batch.item()<br>        <br>        with torch.no_grad():<br>            # Compute logits for all items (in practice, we might do batch inference)<br>            logits, _ = model(seq_batch, torch.tensor([label_batch]))<br>            # logits shape: (1, num_items)<br>            <br>            topk_items = torch.topk(logits, k=k, dim=1).indices.squeeze(0).tolist()<br>            if label_batch in topk_items:<br>                correct += 1<br>            total += 1<br><br>    return correct / total<br><br># Example usage (with a real test set) might look like:<br># test_accuracy = evaluate_topk(model, test_dataset)<br># print(f&quot;Top-{k} accuracy: {test_accuracy:.4f}&quot;)</pre><h3>6. Future Improvements</h3><ul><li><strong>Auxiliary Loss</strong>: DIEN includes an auxiliary loss for each GRU step to better learn interest evolution.</li><li><strong>Multiple GRUs</strong>: One is for interest extraction, and the other is for interest evolution.</li><li><strong>Item &amp; Context Embeddings</strong>: Incorporate category embeddings, brand embeddings, or user context (location, time of day).</li><li><strong>Scaling</strong>: For large-scale data, distribute training across multiple GPUs or use frameworks like PySpark for data preprocessing.</li></ul><h3>Conclusion</h3><p><strong>Deep Interest Evolution Network (DIEN)</strong> offers a powerful mechanism to handle changing user behavior, pushing beyond conventional sequence models by explicitly modeling how user interest evolves. Although this post provides a simplified PyTorch code snippet, the underlying concepts — attention mechanisms for relevance and GRU-based time-series modeling — are essential for any advanced e-commerce recommendation system.</p><p>By applying DIEN (or similar sequence-based neural networks) on a real dataset such as the <strong>Retailrocket Recommender Dataset</strong>, you can:</p><ul><li>Gain deeper insights into user behavior patterns.</li><li>Potentially boost CTR and conversions through more personalized, context-aware recommendations.</li><li>Stay competitive in a market where personalization can be the deciding factor in user retention and revenue growth.</li></ul><p>If you found this tutorial helpful, feel free to leave a comment below or share your experiences on how you’re using advanced deep-learning models to tackle recommendation problems!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79d575295c03" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[12 Python Libraries in 2025 for AI Engineers or Data Scientists]]></title>
            <link>https://medium.com/@sarowar.saurav10/12-python-libraries-in-2025-for-ai-engineers-or-data-scientists-aeb023fecd09?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeb023fecd09</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Sun, 09 Feb 2025 09:30:40 GMT</pubDate>
            <atom:updated>2025-02-09T09:30:40.816Z</atom:updated>
            <content:encoded><![CDATA[<p>In the rapidly evolving world of Artificial Intelligence (AI) and Data Science, mastering new libraries and tools is essential to stay ahead. Python has been at the forefront of this evolution, thanks to its simplicity and a rich ecosystem of libraries that facilitate everything from data preprocessing and machine learning to model deployment.</p><p>In 2025, new and powerful Python libraries are making their mark. Whether you’re a seasoned AI engineer or a budding data scientist, it’s crucial to familiarize yourself with these modern tools to optimize your workflows and take advantage of the latest advancements.</p><p>In this blog, we’ll explore 12 of the most powerful and newly added Python libraries for AI and Data Science. We’ll also show how to use them with a common dataset — the <strong>Titanic dataset</strong> — which is widely used for classification tasks. The Titanic dataset contains data about passengers aboard the ill-fated Titanic, including features like age, class, fare, and whether they survived.</p><p>Let’s dive into these libraries with code examples that you can run using this data source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*j0aMZfS664A6Npn-zeNWLw.png" /></figure><h3>Titanic Dataset Setup</h3><p>You can download the Titanic dataset from Kaggle (<a href="https://www.kaggle.com/c/titanic/data">https://www.kaggle.com/c/titanic/data</a>). The dataset includes the following columns:</p><ul><li><strong>PassengerId</strong>: Unique ID for each passenger</li><li><strong>Pclass</strong>: Passenger class (1st, 2nd, or 3rd)</li><li><strong>Name</strong>: Name of the passenger</li><li><strong>Sex</strong>: Gender of the passenger</li><li><strong>Age</strong>: Age of the passenger</li><li><strong>SibSp</strong>: Number of siblings or spouses aboard the Titanic</li><li><strong>Parch</strong>: Number of parents or children aboard the Titanic</li><li><strong>Ticket</strong>: Ticket number</li><li><strong>Fare</strong>: Passenger fare</li><li><strong>Cabin</strong>: Cabin number</li><li><strong>Embarked</strong>: Port of embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)</li><li><strong>Survived</strong>: Whether the passenger survived (0 = No, 1 = Yes)</li></ul><p>Save this dataset as train.csv and use it in the following examples.</p><h3>1. TensorFlow 2.x</h3><p>TensorFlow is one of the most popular libraries for deep learning. With the release of TensorFlow 2.x, the library became more user-friendly with the inclusion of Keras for high-level APIs. Here’s how you can build a neural network for predicting survival on the Titanic.</p><pre>import tensorflow as tf<br>from tensorflow.keras.models import Sequential<br>from tensorflow.keras.layers import Dense<br>from sklearn.model_selection import train_test_split<br>import pandas as pd</pre><pre># Load Titanic dataset<br>data = pd.read_csv(&quot;train.csv&quot;)</pre><pre># Preprocessing<br>data = data.fillna(0)<br>X = data[[&#39;Pclass&#39;, &#39;Age&#39;, &#39;SibSp&#39;, &#39;Fare&#39;]].values<br>y = data[&#39;Survived&#39;].values<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><pre># Model definition<br>model = Sequential([<br>    Dense(64, activation=&#39;relu&#39;, input_dim=X_train.shape[1]),<br>    Dense(32, activation=&#39;relu&#39;),<br>    Dense(1, activation=&#39;sigmoid&#39;)<br>])</pre><pre>model.compile(optimizer=&#39;adam&#39;, loss=&#39;binary_crossentropy&#39;, metrics=[&#39;accuracy&#39;])<br>model.fit(X_train, y_train, epochs=10, batch_size=32)</pre><pre># Evaluate model<br>accuracy = model.evaluate(X_test, y_test)<br>print(&quot;Model accuracy:&quot;, accuracy[1])</pre><h3>2. PyCaret</h3><p>PyCaret is an open-source, low-code machine learning library that simplifies the process of training, evaluating, and deploying models. It’s perfect for rapid experimentation.</p><pre>from pycaret.classification import *<br>import pandas as pd</pre><pre># Load Titanic dataset<br>data = pd.read_csv(&quot;train.csv&quot;)</pre><pre># Setup PyCaret environment<br>clf = setup(data, target=&#39;Survived&#39;, session_id=123)</pre><pre># Create and compare models<br>best_model = compare_models()</pre><h3>3. Optuna</h3><p>Optuna is an optimization framework for hyperparameter tuning that automates the search for the best parameters for your model, making your models more efficient.</p><pre>import optuna<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.model_selection import train_test_split<br>import pandas as pd</pre><pre># Load Titanic dataset<br>data = pd.read_csv(&quot;train.csv&quot;)<br>X = data[[&#39;Pclass&#39;, &#39;Age&#39;, &#39;SibSp&#39;, &#39;Fare&#39;]].values<br>y = data[&#39;Survived&#39;].values<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><pre># Define objective function<br>def objective(trial):<br>    n_estimators = trial.suggest_int(&#39;n_estimators&#39;, 50, 200)<br>    max_depth = trial.suggest_int(&#39;max_depth&#39;, 3, 10)<br>    model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth)<br>    model.fit(X_train, y_train)<br>    return model.score(X_test, y_test)</pre><pre># Start optimization<br>study = optuna.create_study(direction=&#39;maximize&#39;)<br>study.optimize(objective, n_trials=100)</pre><pre># Best parameters<br>print(f&quot;Best trial: {study.best_trial.params}&quot;)</pre><h3>4. Streamlit</h3><p>Streamlit makes it easy to build interactive web applications. It’s a great tool for showcasing machine learning models and data visualizations.</p><pre>import streamlit as st<br>import pandas as pd</pre><pre># Load Titanic dataset<br>data = pd.read_csv(&quot;train.csv&quot;)</pre><pre># Display dataset<br>st.write(&quot;Titanic Dataset&quot;)<br>st.write(data.head())</pre><pre># Input age and predict survival<br>age = st.number_input(&quot;Enter Age&quot;, min_value=0, max_value=100)<br>if st.button(&quot;Predict&quot;):<br>    st.write(f&quot;Prediction for Age {age}: {model.predict([[age]])}&quot;)</pre><h3>5. Dask</h3><p>Dask is a parallel computing library that integrates with pandas and is great for handling large datasets efficiently.</p><pre>import dask.dataframe as dd</pre><pre># Load Titanic dataset<br>df = dd.read_csv(&quot;train.csv&quot;)</pre><pre># Perform operations on the dataframe<br>df_filtered = df[df[&#39;Age&#39;] &gt; 20]<br>print(df_filtered.compute())</pre><h3>6. Fastai</h3><p>Fastai is a high-level deep learning library built on top of PyTorch. It simplifies training models with minimal code, allowing you to quickly build state-of-the-art models.</p><pre>from fastai.tabular.all import *<br>import pandas as pd</pre><pre># Load Titanic dataset<br>data = pd.read_csv(&quot;train.csv&quot;)</pre><pre># Set up data for Fastai<br>dls = TabularDataLoaders.from_df(data, path=&#39;.&#39;, procs=[Categorify, FillMissing, Normalize], <br>                                 y_names=&quot;Survived&quot;, cat_names=[&quot;Pclass&quot;, &quot;Sex&quot;, &quot;Embarked&quot;], <br>                                 cont_names=[&quot;Age&quot;, &quot;SibSp&quot;, &quot;Fare&quot;])</pre><pre># Create and train model<br>learn = tabular_learner(dls, metrics=accuracy)<br>learn.fit_one_cycle(5)</pre><h3>7. Ray</h3><p>Ray is a distributed computing framework that’s ideal for parallelizing hyperparameter optimization, reinforcement learning, and other compute-heavy tasks.</p><pre>import ray<br>import pandas as pd<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.model_selection import train_test_split</pre><pre># Load Titanic dataset<br>data = pd.read_csv(&quot;train.csv&quot;)<br>X = data[[&#39;Pclass&#39;, &#39;Age&#39;, &#39;SibSp&#39;, &#39;Fare&#39;]].values<br>y = data[&#39;Survived&#39;].values<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><pre># Initialize Ray<br>ray.init()</pre><pre>@ray.remote<br>def train_model(n_estimators):<br>    model = RandomForestClassifier(n_estimators=n_estimators)<br>    model.fit(X_train, y_train)<br>    return model.score(X_test, y_test)</pre><pre># Train models in parallel<br>futures = [train_model.remote(i) for i in [50, 100, 150]]<br>results = ray.get(futures)<br>print(results)</pre><h3>8. MLflow</h3><p>MLflow is a powerful tool for managing the entire machine learning lifecycle, from experiment tracking to deployment.</p><pre>import mlflow<br>import pandas as pd<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.model_selection import train_test_split</pre><pre># Load Titanic dataset<br>data = pd.read_csv(&quot;train.csv&quot;)<br>X = data[[&#39;Pclass&#39;, &#39;Age&#39;, &#39;SibSp&#39;, &#39;Fare&#39;]].values<br>y = data[&#39;Survived&#39;].values<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><pre># Track experiment<br>with mlflow.start_run():<br>    model = LogisticRegression()<br>    model.fit(X_train, y_train)<br>    mlflow.log_metric(&#39;accuracy&#39;, model.score(X_test, y_test))</pre><h3>9. Hugging Face Transformers</h3><p>The Hugging Face Transformers library provides pre-trained state-of-the-art models for NLP tasks. It’s perfect for tasks like text classification and sentiment analysis.</p><pre>from transformers import pipeline</pre><pre># Sentiment analysis on Titanic dataset&#39;s &#39;Name&#39; column<br>classifier = pipeline(&#39;sentiment-analysis&#39;)</pre><pre># Analyze text<br>result = classifier(&quot;I love the Titanic dataset!&quot;)<br>print(result)</pre><h3>10. JAX</h3><p>JAX is designed for high-performance numerical computing and automatic differentiation. It’s especially useful in machine learning and deep learning tasks.</p><pre>import jax<br>import jax.numpy as jnp<br>import pandas as pd</pre><pre># Load Titanic dataset<br>data = pd.read_csv(&quot;train.csv&quot;)</pre><pre># Example function<br>def add_numbers(x, y):<br>    return x + y</pre><pre># JAX version of add_numbers<br>jax_add = jax.jit(add_numbers)<br>print(jax_add(1, 2))</pre><h3>11. Polars</h3><p>Polars is a fast DataFrame library designed to handle larger-than-memory datasets with a syntax similar to pandas. It’s ideal for big data tasks.</p><pre>import polars as pl</pre><pre># Load Titanic dataset<br>df = pl.read_csv(&quot;train.csv&quot;)</pre><pre># Perform operations<br>df_filtered = df.filter(pl.col(&quot;Age&quot;) &gt; 30)<br>print(df_filtered)</pre><h3>12. Shap</h3><p>Shap is a powerful library for interpreting machine learning models and understanding the impact of features on predictions.</p><pre>import shap<br>import pandas as pd<br>import xgboost as xgb<br>from sklearn.model_selection import train_test_split</pre><pre># Load Titanic dataset<br>data = pd.read_csv(&quot;train.csv&quot;)<br>X = data[[&#39;Pclass&#39;, &#39;Age&#39;, &#39;SibSp&#39;, &#39;Fare&#39;]].values<br>y = data[&#39;Survived&#39;].values<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><pre># Train model<br>model = xgb.XGBClassifier()<br>model.fit(X_train, y_train)</pre><pre># SHAP values<br>explainer = shap.TreeExplainer(model)<br>shap_values = explainer.shap_values(X_test)</pre><pre># Plot SHAP values<br>shap.summary_plot(shap_values, X_test)</pre><h3>Conclusion</h3><p>With the rapid advancements in AI and data science tools, it’s crucial for every AI engineer and data scientist to be proficient in the latest libraries. The libraries listed in this blog, paired with the Titanic dataset, will help you stay on top of trends and give you the flexibility to build powerful models. By mastering these tools, you’ll be better equipped to handle data processing, model training, optimization, and interpretation.</p><p>Start exploring these libraries, experiment with the Titanic dataset, and see how they can help you improve your machine learning workflows. Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeb023fecd09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[20 Advanced Statistical Approaches Every Data Scientist Should Know ‍]]></title>
            <link>https://medium.com/@sarowar.saurav10/20-advanced-statistical-approaches-every-data-scientist-should-know-ccc70ae4df28?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccc70ae4df28</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Thu, 06 Feb 2025 08:02:00 GMT</pubDate>
            <atom:updated>2025-02-15T19:59:15.481Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>20 Important Statistical Approaches Every Data Scientist Knows🐱‍🚀</strong></h3><p>Data science is a multidisciplinary field that combines mathematics, statistics, computer science, and domain expertise to extract insights from data. While machine learning algorithms often steal the spotlight, a solid foundation in <strong>statistical methods</strong> can be just as pivotal. In this writing, we explore <strong>20 advanced statistical approaches</strong> that every data scientist should be familiar with. Each approach includes a brief explanation, an example use case, and a snippet of Python code to illustrate its usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CX-BDIgw8kk42l5m4pU11g.png" /></figure><h3>1. Bayesian Inference</h3><p><strong>What It Is</strong><br> Bayesian inference uses Bayes’ theorem to update the probability of a hypothesis as more evidence or information becomes available. Unlike frequentist methods, Bayesian approaches allow you to incorporate prior beliefs and update those beliefs with observed data.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Spam Filtering</strong>: Prior belief on how likely an email is spam can be combined with new evidence (words in the email) to update the spam probability.</li></ul><p><strong>Code Snippet</strong><br> Below is a simple example using <a href="https://docs.pymc.io/">PyMC3</a> or <a href="https://www.pymc.io/">PyMC</a> to perform Bayesian parameter estimation for a Bernoulli process (e.g., coin flips).</p><pre>!pip install pymc3  # or pymc</pre><pre>import pymc3 as pm<br>import numpy as np</pre><pre># Suppose we observed 20 coin flips with 12 heads and 8 tails<br>observed_heads = 12<br>observed_tails = 8</pre><pre>with pm.Model() as model:<br>    # Prior for the bias of the coin (theta)<br>    theta = pm.Beta(&#39;theta&#39;, alpha=1, beta=1)<br>    <br>    # Likelihood<br>    y = pm.Binomial(&#39;y&#39;, n=observed_heads + observed_tails, p=theta, observed=observed_heads)<br>    <br>    # Posterior sampling<br>    trace = pm.sample(2000, tune=1000, cores=1, chains=2)</pre><pre>pm.summary(trace)</pre><h3>2. Maximum Likelihood Estimation (MLE)</h3><p><strong>What It Is</strong><br> MLE finds the parameter values that maximize the likelihood of the observed data under a certain statistical model.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Fitting Distributions</strong>: Estimate parameters (mean, variance) of a normal distribution that best fit your data.</li></ul><p><strong>Code Snippet</strong><br> Using scipy.stats to fit a normal distribution:</p><pre>import numpy as np<br>from scipy.stats import norm</pre><pre># Generate synthetic data<br>data = np.random.normal(loc=5, scale=2, size=1000)</pre><pre># Estimate MLE for mean and standard deviation<br>mu_hat, std_hat = norm.fit(data)</pre><pre>print(f&quot;Estimated Mean (mu): {mu_hat:.2f}&quot;)<br>print(f&quot;Estimated Std (sigma): {std_hat:.2f}&quot;)</pre><h3>3. Hypothesis Testing (t-test)</h3><p><strong>What It Is</strong><br> Hypothesis testing involves formulating a null hypothesis (no difference/effect) and an alternative hypothesis. A t-test specifically checks if the means of two groups are significantly different.</p><p><strong>Example Use Case</strong></p><ul><li><strong>A/B Testing</strong>: Testing if a new website layout (Group B) leads to a significantly different average session time compared to the old layout (Group A).</li></ul><p><strong>Code Snippet</strong><br> Using scipy.stats.ttest_ind for a two-sample t-test:</p><pre>import numpy as np<br>from scipy.stats import ttest_ind</pre><pre># Synthetic data<br>group_A = np.random.normal(5, 1, 50)<br>group_B = np.random.normal(5.5, 1.2, 50)</pre><pre>stat, pvalue = ttest_ind(group_A, group_B)</pre><pre>print(f&quot;T-statistic: {stat:.2f}, p-value: {pvalue:.4f}&quot;)<br>if pvalue &lt; 0.05:<br>    print(&quot;Reject the null hypothesis (Significant difference).&quot;)<br>else:<br>    print(&quot;Fail to reject the null hypothesis (No significant difference).&quot;)</pre><h3>4. Analysis of Variance (ANOVA)</h3><p><strong>What It Is</strong><br> ANOVA tests whether there is a statistically significant difference between the means of three or more groups.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Marketing Experiment</strong>: Evaluate the effectiveness of three different advertising strategies by measuring sales lift.</li></ul><p><strong>Code Snippet</strong><br> Using scipy.stats.f_oneway:</p><pre>import numpy as np<br>from scipy.stats import f_oneway</pre><pre># Synthetic data<br>group1 = np.random.normal(10, 2, 30)<br>group2 = np.random.normal(12, 2, 30)<br>group3 = np.random.normal(14, 2, 30)</pre><pre>f_stat, p_val = f_oneway(group1, group2, group3)<br>print(f&quot;F-statistic: {f_stat:.2f}, p-value: {p_val:.4f}&quot;)</pre><h3>5. Principal Component Analysis (PCA)</h3><p><strong>What It Is</strong><br> PCA reduces the dimensionality of data by projecting it onto new, orthogonal axes (principal components) that capture the most variance.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Image Compression</strong>: Reduce high-dimensional pixel data into fewer features for faster processing.</li></ul><p><strong>Code Snippet</strong><br> Using sklearn.decomposition.PCA:</p><pre>import numpy as np<br>from sklearn.decomposition import PCA</pre><pre># Synthetic data: 100 samples with 10 features<br>X = np.random.rand(100, 10)</pre><pre>pca = PCA(n_components=2)<br>X_reduced = pca.fit_transform(X)</pre><pre>print(&quot;Explained variance ratios:&quot;, pca.explained_variance_ratio_)<br>print(&quot;Reduced shape:&quot;, X_reduced.shape)</pre><h3>6. Factor Analysis</h3><p><strong>What It Is</strong><br> Factor Analysis models the observed variables as linear combinations of latent (unobserved) factors, often used for dimensionality reduction or to uncover hidden structure.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Psychometrics</strong>: Identifying underlying personality traits from questionnaire data.</li></ul><p><strong>Code Snippet</strong><br> Using factor_analyzer library:</p><pre>!pip install factor_analyzer</pre><pre>import numpy as np<br>from factor_analyzer import FactorAnalyzer</pre><pre># Synthetic data (100 samples, 6 variables)<br>X = np.random.rand(100, 6)</pre><pre>fa = FactorAnalyzer(n_factors=2, rotation=&#39;varimax&#39;)<br>fa.fit(X)</pre><pre>print(&quot;Loadings:\n&quot;, fa.loadings_)</pre><h3>7. Cluster Analysis (K-means)</h3><p><strong>What It Is</strong><br> Clustering partitions data into homogeneous groups (clusters) based on similarity. K-means is a popular centroid-based clustering technique.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Customer Segmentation</strong>: Group customers by purchasing patterns.</li></ul><p><strong>Code Snippet</strong><br> Using sklearn.cluster.KMeans:</p><pre>import numpy as np<br>from sklearn.cluster import KMeans</pre><pre># Synthetic data: 200 samples, 2D<br>X = np.random.rand(200, 2)</pre><pre>kmeans = KMeans(n_clusters=3, random_state=42)<br>kmeans.fit(X)</pre><pre>print(&quot;Cluster centers:&quot;, kmeans.cluster_centers_)<br>print(&quot;Cluster labels:&quot;, kmeans.labels_[:10])</pre><h3>8. Bootstrapping</h3><p><strong>What It Is</strong><br> Bootstrapping involves repeatedly sampling with replacement from your dataset to estimate the distribution (and uncertainty) of a statistic (e.g., mean, median).</p><p><strong>Example Use Case</strong></p><ul><li><strong>Confidence Intervals</strong>: Estimate a 95% confidence interval for the mean of a small dataset.</li></ul><p><strong>Code Snippet</strong><br> Illustrating a simple bootstrap for the mean:</p><pre>import numpy as np</pre><pre>np.random.seed(42)<br>data = np.random.normal(50, 5, size=100)  # Original sample</pre><pre>def bootstrap_mean_ci(data, n_bootstraps=1000, ci=95):<br>    means = []<br>    n = len(data)<br>    for _ in range(n_bootstraps):<br>        sample = np.random.choice(data, size=n, replace=True)<br>        means.append(np.mean(sample))<br>    lower = np.percentile(means, (100-ci)/2)<br>    upper = np.percentile(means, 100 - (100-ci)/2)<br>    return np.mean(means), (lower, upper)</pre><pre>mean_estimate, (lower_ci, upper_ci) = bootstrap_mean_ci(data)<br>print(f&quot;Bootstrap Mean: {mean_estimate:.2f}&quot;)<br>print(f&quot;{95}% CI: [{lower_ci:.2f}, {upper_ci:.2f}]&quot;)</pre><h3>9. Time Series Analysis (ARIMA)</h3><p><strong>What It Is</strong><br> ARIMA (AutoRegressive Integrated Moving Average) is a popular model for forecasting univariate time series data by capturing autocorrelation in the data.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Sales Forecasting</strong>: Predict future sales based on past performance.</li></ul><p><strong>Code Snippet</strong><br> Using statsmodels.tsa.arima.model:</p><pre>!pip install statsmodels</pre><pre>import numpy as np<br>import pandas as pd<br>from statsmodels.tsa.arima.model import ARIMA</pre><pre># Synthetic time series data<br>np.random.seed(42)<br>data = np.random.normal(100, 5, 50)<br>time_series = pd.Series(data)</pre><pre># Fit ARIMA model (p=1, d=1, q=1)<br>model = ARIMA(time_series, order=(1,1,1))<br>model_fit = model.fit()</pre><pre># Forecast next 5 points<br>forecast = model_fit.forecast(steps=5)<br>print(&quot;Forecast:&quot;, forecast.values)</pre><h3>10. Survival Analysis</h3><p><strong>What It Is</strong><br> Survival Analysis deals with time-to-event data, often focusing on the probability of an event (e.g., churn) occurring after a certain time.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Customer Churn</strong>: Estimate how long a customer will remain active before canceling a subscription.</li></ul><p><strong>Code Snippet</strong><br> Using lifelines:</p><pre>!pip install lifelines</pre><pre>import numpy as np<br>import pandas as pd<br>from lifelines import KaplanMeierFitter</pre><pre># Synthetic survival times and censorship<br>np.random.seed(42)<br>durations = np.random.exponential(scale=10, size=100)<br>event_observed = np.random.binomial(1, 0.8, size=100)</pre><pre>kmf = KaplanMeierFitter()<br>kmf.fit(durations, event_observed=event_observed, label=&#39;Test Group&#39;)</pre><pre>kmf.plot_survival_function()</pre><h3>11. Multivariate Regression (Multiple Linear Regression)</h3><p><strong>What It Is</strong><br> Multiple linear regression models the relationship between a dependent variable and multiple independent variables.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Pricing Models</strong>: Predict house price based on square footage, number of rooms, and location.</li></ul><p><strong>Code Snippet</strong><br> Using sklearn.linear_model.LinearRegression:</p><pre>import numpy as np<br>from sklearn.linear_model import LinearRegression</pre><pre># Synthetic data: price = 100 + 2*rooms + 0.5*sqft + noise<br>np.random.seed(42)<br>rooms = np.random.randint(1, 5, 100)<br>sqft = np.random.randint(500, 2500, 100)<br>price = 100 + 2*rooms + 0.5*sqft + np.random.normal(0, 50, 100)</pre><pre>X = np.column_stack([rooms, sqft])<br>y = price</pre><pre>model = LinearRegression()<br>model.fit(X, y)</pre><pre>print(&quot;Coefficients:&quot;, model.coef_)<br>print(&quot;Intercept:&quot;, model.intercept_)</pre><h3>12. Ridge/Lasso Regression</h3><p><strong>What It Is</strong></p><ul><li><strong>Ridge Regression</strong> adds an L2 penalty to reduce overfitting by shrinking coefficients.</li><li><strong>Lasso Regression</strong> adds an L1 penalty, which can drive some coefficients to zero, effectively performing feature selection.</li></ul><p><strong>Example Use Case</strong></p><ul><li><strong>High-Dimensional Data</strong>: Gene expression data with many correlated features.</li></ul><p><strong>Code Snippet</strong><br> Using sklearn.linear_model:</p><pre>import numpy as np<br>from sklearn.linear_model import Ridge, Lasso<br>from sklearn.model_selection import train_test_split</pre><pre># Synthetic data<br>np.random.seed(42)<br>X = np.random.rand(100, 10)<br>y = X[:, 0]*5 + X[:, 1]*3 + np.random.normal(0, 0.1, 100)</pre><pre>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)</pre><pre>ridge = Ridge(alpha=1.0).fit(X_train, y_train)<br>lasso = Lasso(alpha=0.1).fit(X_train, y_train)</pre><pre>print(&quot;Ridge coefficients:&quot;, ridge.coef_)<br>print(&quot;Lasso coefficients:&quot;, lasso.coef_)</pre><h3>13. Logistic Regression</h3><p><strong>What It Is</strong><br> Logistic regression is used for binary classification, modeling the probability of a certain class or event existing.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Credit Card Fraud Detection</strong>: Classify transactions as fraudulent or legitimate.</li></ul><p><strong>Code Snippet</strong><br> Using sklearn.linear_model.LogisticRegression:</p><pre>import numpy as np<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.model_selection import train_test_split</pre><pre>np.random.seed(42)<br>X = np.random.rand(100, 5)<br>y = np.random.randint(0, 2, 100)  # Binary labels</pre><pre>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)<br>model = LogisticRegression()<br>model.fit(X_train, y_train)</pre><pre>accuracy = model.score(X_test, y_test)<br>print(&quot;Accuracy:&quot;, accuracy)</pre><h3>14. Mixed Effects Models</h3><p><strong>What It Is</strong><br> Mixed effects models (or hierarchical linear models) include both fixed effects (common to all groups) and random effects (specific to each group). They are commonly used in longitudinal or grouped data.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Educational Data</strong>: Test scores across multiple schools, each school having a random intercept.</li></ul><p><strong>Code Snippet</strong><br> Using statsmodels’ mixedlm:</p><pre>import pandas as pd<br>import numpy as np<br>import statsmodels.formula.api as smf</pre><pre># Synthetic data: Each student belongs to a specific school<br>np.random.seed(42)<br>school_ids = np.repeat(np.arange(10), 20)<br>scores = 50 + 5*np.random.rand(200) + 2*school_ids + np.random.normal(0, 5, 200)</pre><pre>df = pd.DataFrame({&quot;score&quot;: scores, &quot;school&quot;: school_ids})<br>model = smf.mixedlm(&quot;score ~ 1&quot;, df, groups=df[&quot;school&quot;])<br>result = model.fit()<br>print(result.summary())</pre><h3>15. Nonparametric Tests (Mann-Whitney U)</h3><p><strong>What It Is</strong><br> Nonparametric tests do not assume a specific distribution of the data. The Mann-Whitney U test is used to compare two independent samples.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Median Comparison</strong>: Compare the median of sales in two stores without assuming normality.</li></ul><p><strong>Code Snippet</strong><br> Using scipy.stats.mannwhitneyu:</p><pre>import numpy as np<br>from scipy.stats import mannwhitneyu</pre><pre># Synthetic data<br>group_A = np.random.exponential(scale=1.0, size=30)<br>group_B = np.random.exponential(scale=1.2, size=30)</pre><pre>stat, pvalue = mannwhitneyu(group_A, group_B, alternative=&#39;two-sided&#39;)<br>print(f&quot;Statistic: {stat:.2f}, p-value: {pvalue:.4f}&quot;)</pre><h3>16. Monte Carlo Simulation</h3><p><strong>What It Is</strong><br> Monte Carlo simulations use repeated random sampling to estimate the probability of different outcomes under uncertainty.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Risk Analysis</strong>: Predict the probability of project cost overruns given uncertain variables like labor cost, raw material cost, etc.</li></ul><p><strong>Code Snippet</strong><br> Estimating π using a Monte Carlo approach:</p><pre>import numpy as np</pre><pre>np.random.seed(42)<br>n_samples = 10_000_00<br>xs = np.random.rand(n_samples)<br>ys = np.random.rand(n_samples)<br># Points within the unit circle<br>inside_circle = (xs**2 + ys**2) &lt;= 1.0<br>pi_estimate = inside_circle.sum() * 4 / n_samples</pre><pre>print(&quot;Estimated π:&quot;, pi_estimate)</pre><h3>17. Markov Chain Monte Carlo (MCMC)</h3><p><strong>What It Is</strong><br> MCMC methods (e.g., Metropolis-Hastings, Gibbs sampling) are used in Bayesian inference to generate samples from a posterior distribution when direct sampling is difficult.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Parameter Estimation</strong>: Complex hierarchical models where direct integration is not feasible.</li></ul><p><strong>Code Snippet</strong><br> Using PyMC3 to do a simple MCMC:</p><pre>import pymc3 as pm<br>import numpy as np</pre><pre># Synthetic data<br>np.random.seed(42)<br>data = np.random.normal(0, 1, 100)</pre><pre>with pm.Model() as model:<br>    mu = pm.Normal(&#39;mu&#39;, mu=0, sd=10)<br>    sigma = pm.HalfNormal(&#39;sigma&#39;, sd=1)<br>    <br>    likelihood = pm.Normal(&#39;likelihood&#39;, mu=mu, sd=sigma, observed=data)<br>    trace = pm.sample(1000, tune=500, chains=2)</pre><pre>pm.summary(trace)</pre><h3>18. Robust Regression</h3><p><strong>What It Is</strong><br> Robust regression methods (e.g., RANSAC, Huber regression) are less sensitive to outliers compared to ordinary least squares.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Outlier-Prone Data</strong>: Fitting a model to data that contains extreme values, such as in finance.</li></ul><p><strong>Code Snippet</strong><br> Using sklearn.linear_model.RANSACRegressor:</p><pre>import numpy as np<br>from sklearn.linear_model import RANSACRegressor, LinearRegression</pre><pre>np.random.seed(42)<br>X = np.random.rand(100, 1) * 10<br>y = 3 * X.squeeze() + 2 + np.random.normal(0, 2, 100)</pre><pre># Add outliers<br>X_outliers = np.array([[8], [9], [9.5]])<br>y_outliers = np.array([50, 55, 60])<br>X = np.vstack((X, X_outliers))<br>y = np.concatenate((y, y_outliers))</pre><pre>ransac = RANSACRegressor(base_estimator=LinearRegression(), max_trials=100)<br>ransac.fit(X, y)<br>print(&quot;RANSAC Coefficients:&quot;, ransac.estimator_.coef_)<br>print(&quot;RANSAC Intercept:&quot;, ransac.estimator_.intercept_)</pre><h3>19. Copulas</h3><p><strong>What It Is</strong><br> Copulas capture the dependence structure between random variables separately from their marginal distributions. They are popular in finance for modeling joint distributions of asset returns.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Portfolio Risk</strong>: Model the joint behavior of multiple stocks that exhibit non-linear dependence.</li></ul><p><strong>Code Snippet</strong><br> A simple example using a Gaussian copula from the copulas library:</p><pre>!pip install copulas</pre><pre>import numpy as np<br>from copulas.multivariate import GaussianMultivariate</pre><pre># Synthetic data<br>X = np.random.normal(0, 1, (1000, 2))<br>X[:,1] = 0.8 * X[:,0] + np.random.normal(0, 0.6, 1000)  # correlation</pre><pre>model = GaussianMultivariate()<br>model.fit(X)</pre><pre>sample = model.sample(5)<br>print(&quot;Original correlation:&quot;, np.corrcoef(X[:, 0], X[:, 1])[0,1])<br>print(&quot;Sample correlation:&quot;, np.corrcoef(sample[:, 0], sample[:, 1])[0,1])</pre><h3>20. Generalized Additive Models (GAMs)</h3><p><strong>What It Is</strong><br> GAMs extend linear models by allowing for non-linear functions of predictors while maintaining additivity. They’re more flexible than linear regression but still interpretable.</p><p><strong>Example Use Case</strong></p><ul><li><strong>Health Data</strong>: Modeling a patient outcome as a smooth non-linear function of age and other variables.</li></ul><p><strong>Code Snippet</strong><br> Using the pyGAM library:</p><pre>!pip install pygam</pre><pre>import numpy as np<br>from pygam import LinearGAM, s</pre><pre># Synthetic data<br>np.random.seed(42)<br>X = np.random.rand(200, 1) * 10<br>y = 2 + 3*np.sin(X).ravel() + np.random.normal(0, 0.5, 200)</pre><pre>gam = LinearGAM(s(0)).fit(X, y)<br>XX = np.linspace(0, 10, 100)<br>preds = gam.predict(XX)<br>print(&quot;Coefficients:&quot;, gam.summary())</pre><h3>Conclusion</h3><p>From understanding <strong>Bayesian inference</strong> and <strong>MLE</strong>, through advanced concepts like <strong>Copulas</strong> and <strong>GAMs</strong>, these 20 advanced statistical approaches form a comprehensive toolkit for <strong>any data scientist</strong>. The code snippets here are brief illustrations, but each method can be explored in-depth to address complex real-world problems — be it forecasting, inference, or modeling intricate relationships.</p><p><strong>Happy Analyzing!</strong></p><h3>Further Reading</h3><ol><li><a href="https://xcelab.net/rm/statistical-rethinking/">Statistical Rethinking by Richard McElreath</a></li><li><a href="https://web.stanford.edu/~hastie/Papers/ESLII.pdf">Elements of Statistical Learning</a></li><li><a href="https://www.microsoft.com/en-us/research/people/cmbishop/prml-book/">Pattern Recognition and Machine Learning by Christopher Bishop</a></li></ol><p>If you found this post helpful, feel free to share it and leave a comment! And remember, mastering these statistical techniques will not only make you a stronger data scientist but also enable you to interpret and communicate data-driven insights with confidence.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccc70ae4df28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why a Business Grad Can’t Be a Great Software or ML Developer — Even With Generative AI on Their…]]></title>
            <link>https://medium.com/@sarowar.saurav10/why-a-business-grad-cant-be-a-great-software-or-ml-developer-even-with-generative-ai-on-their-c53f6a84a45b?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c53f6a84a45b</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Wed, 27 Nov 2024 14:59:39 GMT</pubDate>
            <atom:updated>2024-11-27T14:59:39.487Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Why a Business Grad Can’t Be a Great Software or ML Developer — Even With Generative AI on Their Side 🐱‍💻</strong></h3><p>In the age of generative AI, it feels like anything is possible. ChatGPT can churn out code for you faster than you can say “Singleton Pattern,” and platforms like GitHub Copilot are basically your coding buddy on demand. So, it’s only natural to wonder: If AI can write code, could a business grad — someone with no formal computer science training — become a great software developer with the right tools at their fingertips?</p><p>The short answer? Not quite. 🙅‍♂️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hix6D55UEtLsTJ2V.jpg" /></figure><p>Here’s why: <strong>coding is more than just writing syntax</strong>. Sure, AI can help generate boilerplate code or implement design patterns like the classic Singleton or Factory patterns, but being a <em>great</em> software developer requires a deeper understanding of problem-solving, systems thinking, and software design that goes far beyond what AI can generate.</p><p>Let’s dive into the key reasons why a business grad — even with all the AI magic — will struggle to become a top-tier developer. 🚀</p><h3>1. Problem-Solving Isn’t About Copy-Pasting Code 💻➡️📄</h3><p>A lot of coding is about understanding the <em>problem</em> you’re solving. Sure, AI can write code snippets based on your prompts, but if you don’t have a strong grasp of algorithms, data structures, and how to approach complex problems, you’ll be lost when things inevitably go off the rails.</p><p>For example, imagine you’re building an e-commerce platform, and you need to implement a custom sorting algorithm. A business grad with a basic understanding of what a sorting algorithm does might have no idea when or why to choose a particular approach (QuickSort vs. MergeSort?) — and even worse, might miss out on edge cases or performance bottlenecks.</p><p>Generative AI can give you code, but it won’t teach you how to think critically about <em>why</em> that code works or whether it’s the best solution. 🚧</p><h3>2. Design Patterns Aren’t Just Buzzwords 🔄</h3><p>You might hear the term “design patterns” thrown around a lot, but understanding them at a deep level is what separates a novice from a pro. Design patterns like Factory, Singleton, or Observer aren’t just things you drop into code because an AI told you to — they represent established solutions to recurring problems.</p><p>A business grad might <em>know</em> what a Singleton pattern is, but do they understand when to apply it effectively? Do they know the trade-offs of using a certain design pattern in a specific scenario, and how it affects scalability, maintainability, and testing? Without these insights, they’re simply following instructions without understanding the “why” behind the code. 🧠</p><p>AI can generate these patterns, but the value of being able to evaluate and adapt them to your unique project context? That’s something AI can’t <em>teach</em>. 🏗️</p><h3>3. Quality Assurance: Testing Is Key 🔍</h3><p>Building software isn’t just about getting the code to run. It’s about ensuring it <em>works</em> — under every possible scenario, edge case, and load. A business grad might be able to use generative AI to write code that runs perfectly on the first try, but without solid knowledge of quality assurance (QA) practices, they’re setting themselves up for disaster.</p><p>Think about it: how will they write proper unit tests? Can they apply Test-Driven Development (TDD) to ensure the code is robust? Generative AI may be able to help with writing test cases, but understanding test coverage, integration tests, and system testing requires a solid foundation in software engineering principles.</p><p>Skipping this step leads to messy, bug-ridden code. And AI isn’t going to tell you that your app is about to crash when 100,000 people log in at the same time. 👀</p><h3>4. Design Thinking: Where Generalists Can Survive👩‍💻❤️</h3><p>This might be a surprise, but software development is not just about writing code — it’s about creating a product that solves real problems for real people. Design thinking is a human-centered approach that business grads (especially those with product or UX experience) might be familiar with, but not every business grad knows how to blend it effectively with technical execution.</p><p>A great developer needs to think like a user: What is the user journey? How do they interact with the app? What frustrates them? A business grad might have excellent product vision, but without an understanding of how to translate that vision into scalable, usable, and performant code, they will miss the mark.</p><p>AI can’t empathize with users, but a good developer can. That’s the magic of integrating both business thinking and tech skills. 🤖 + 💡</p><h3>5. The Value of Academics: The “Gut Feeling” in Coding 🧠⚡</h3><p>There’s no substitute for hands-on experience when it comes to development. Sure, a business grad might have access to a mountain of online resources and AI tools, but they’ll be missing something crucial: the “gut feeling” that comes with years of coding experience.</p><p>Experienced developers understand trade-offs instinctively. They know when to optimize, when to refactor, when to keep things simple, and when to push the boundaries. They’ve seen firsthand what works (and what doesn’t), and they can troubleshoot issues that would send a newbie into a panic.</p><p>AI can certainly help speed up the process, but it doesn’t give you that “gut feeling” for what’s best. Experience — and understanding your system as a whole — matters more than you might think. 🌍</p><h3>6. Communication and Collaboration Skills 🤝</h3><p>Software development is a team sport. Developers need to communicate effectively with designers, product managers, and other stakeholders. While a business grad might excel in communication within a business context, coding requires a different type of collaboration — one rooted in technical language, problem-solving, and code reviews.</p><p>Building a successful software project involves continuous feedback, brainstorming sessions, and iterative improvement. Even if generative AI helps you write code, the collaboration with real people is where the real value comes in.</p><p>AI can’t replace the need for clear communication and teamwork that happens in real-world software development. 👨‍👩‍👧‍👦</p><h3>In Conclusion: Generative AI Is a Tool, Not a Magic Wand ⚡</h3><p>Generative AI is a <em>fantastic</em> tool, but it’s not a shortcut to becoming a great software developer. Being a great developer requires a mix of deep technical knowledge, problem-solving skills, design thinking, and real-world experience. AI can help with the heavy lifting, but it can’t replace the nuanced understanding that comes with years of practice.</p><p>So, while a business grad with access to AI might be able to <em>code</em>, they’re still missing the essential developer skills that turn good code into great software. Think of it like trying to bake a cake using only the recipe — but without understanding the science of ingredients, techniques, and timing. 🍰</p><p>In the end, generative AI is best seen as a tool to assist, not replace, the human developer. And until a business grad can acquire the full breadth of technical skills and experience, they’ll still be a <em>business grad</em> — not a true software engineer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c53f6a84a45b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Basics: 11 Complex Statistical Algorithms to Elevate Data Science Game]]></title>
            <link>https://medium.com/@sarowar.saurav10/beyond-the-basics-11-complex-statistical-algorithms-to-elevate-data-science-game-d4d6cffcd4a9?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4d6cffcd4a9</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-ai]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Fri, 13 Sep 2024 16:00:18 GMT</pubDate>
            <atom:updated>2024-09-13T16:09:51.586Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Science is more than just running standard algorithms or crafting elegant visualizations. It’s about uncovering hidden insights and patterns within complex datasets. While many data scientists are familiar with the typical suspects like Linear Regression, Decision Trees, or k-means clustering, there exists a universe of <strong>advanced statistical algorithms</strong> that remain relatively unknown but are immensely powerful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*Zk4Xqwg1zH5dv_TpfCgsQA.jpeg" /></figure><p>These algorithms can elevate your data science skills by providing robust solutions to real-world challenges, such as handling high-dimensional data, modeling complex dependencies, or making predictions in dynamic systems. Ready to dive in? Let’s explore these hidden gems that every data scientist should know! 🔍</p><h4>1. Theil-Sen Estimator 📈</h4><p>The Theil-Sen Estimator is a robust alternative to the ordinary least squares (OLS) regression. Unlike OLS, which is highly sensitive to outliers, the Theil-Sen Estimator provides a <strong>non-parametric regression technique</strong> that calculates the median of all possible slopes from pairs of points, making it extremely resistant to anomalies. This approach is particularly valuable when your data is plagued by noise or contains unexpected extreme values. 📊</p><p>🔧 <strong>Use Case:</strong> Financial time series analysis, where the data is often non-linear and contains outliers due to sudden market changes. 💹</p><h4>2. Empirical Bayes 🧮</h4><p>Empirical Bayes is a more practical cousin of traditional Bayesian methods. It estimates prior distributions using the data itself, rather than relying on pure prior information, allowing for <strong>better generalization</strong> and <strong>reduced overfitting</strong>. This clever trick merges frequentist and Bayesian approaches, making it useful when you have multiple similar datasets or subgroups.</p><p>🔧 <strong>Use Case:</strong> A/B testing in online platforms, where dynamic updates to prior beliefs based on user interactions lead to more agile, data-driven decisions. 🌐</p><h4>3. Variational Inference (VI) 🔄</h4><p>Variational Inference (VI) turns a complex inference problem into an optimization problem, providing a faster and more scalable alternative to sampling methods like MCMC. By approximating complex posterior distributions through optimization, VI achieves <strong>near Bayesian-level precision</strong> without the computational overhead, making it suitable for large-scale and real-time applications. 🏎️💨</p><p>🔧 <strong>Use Case:</strong> Probabilistic deep learning, where modeling uncertainty in neural networks is crucial, especially for unsupervised and semi-supervised learning tasks. 🧠</p><h4>4. Markov Chain Monte Carlo (MCMC) Simulations 🎲</h4><p>MCMC methods, such as Metropolis-Hastings and Gibbs sampling, are essential tools for approximating complex distributions, especially in high-dimensional spaces. These algorithms are the backbone of <strong>Bayesian statistics</strong> and allow you to perform probabilistic inferences that would otherwise be computationally prohibitive.</p><p>🔧 <strong>Use Case:</strong> Natural Language Processing (NLP) for topic modeling using Latent Dirichlet Allocation (LDA), where the posterior distribution over topics is sampled using MCMC techniques. 📚🗣️</p><h4>5. Hidden Markov Models (HMMs) 🔍</h4><p>Hidden Markov Models (HMMs) are ideal for modeling <strong>time series data</strong> where the states are not directly observable. Unlike traditional time series models, HMMs handle probabilistic sequences of states, making them perfect for datasets that exhibit a sequential or temporal nature. ⏳</p><p>🔧 <strong>Use Case:</strong> Speech recognition systems, where audio signals are modeled as sequences of phonemes, each representing a hidden state. 🗣️🎤</p><h4>6. Copula-based Models 📉</h4><p>Copula models allow you to understand and model dependencies between multiple random variables. Unlike traditional methods, Copulas can capture <strong>non-linear dependencies and tail dependencies</strong>, which are particularly important in fields like finance and environmental sciences.</p><p>🔧 <strong>Use Case:</strong> Financial risk modeling, where assets exhibit complex, non-linear dependencies that simple linear models cannot capture. 💹</p><h4>7. Sparse Principal Component Analysis (Sparse PCA) 📐</h4><p>Sparse PCA is a variant of PCA designed for <strong>high-dimensional data</strong> where you want to enforce sparsity on the principal components. This modification makes it easier to interpret which variables contribute most to the variance, especially in datasets with thousands or millions of features. 🌟</p><p>🔧 <strong>Use Case:</strong> Genomics, where you need to identify key genes among thousands that contribute to specific conditions. 🧬</p><h4>8. Sequential Monte Carlo (SMC) Methods 🔄</h4><p>Also known as <strong>Particle Filters</strong>, Sequential Monte Carlo methods are used for sampling from complex, evolving probability distributions. These methods are highly effective for real-time tracking, filtering, and prediction problems, where the system state is continuously changing.</p><p>🔧 <strong>Use Case:</strong> Object tracking in video streams or robotic navigation, where the environment is dynamically evolving. 🤖🚗</p><h4>9. Hierarchical Linear Models (HLMs) or Multilevel Models 🏗️</h4><p>Hierarchical Linear Models, or Multilevel Models, are ideal when dealing with <strong>nested or hierarchical data structures</strong> (e.g., students within schools, patients within hospitals). They provide a way to model both within-group and between-group variations, making them invaluable for datasets where data points are not independent and identically distributed (i.i.d). 📊</p><p>🔧 <strong>Use Case:</strong> Education research, where student performance data is nested within classes and schools. 🏫</p><h4>10. Tukey’s Ladder of Powers Transformation 🪜</h4><p>Tukey’s Ladder of Powers is a technique to stabilize variance and normalize data distributions. This transformation helps to <strong>reduce the influence of large errors</strong> and outliers, making your data more amenable to statistical tests and models.</p><p>🔧 <strong>Use Case:</strong> Preparing skewed economic or biomedical data for linear regression models, enhancing their robustness and reliability. 🏥📈</p><h4>11. Random Survival Forests (RSF) 🌲</h4><p>Random Survival Forests are an extension of Random Forests designed for <strong>survival analysis</strong>, where the outcome is the time until an event occurs. RSF can handle <strong>non-linear effects</strong> and complex interactions between variables, making them more robust in survival data analysis.</p><p>🔧 <strong>Use Case:</strong> Predicting patient survival rates or time to relapse in healthcare studies. 🏥📊</p><h3>Conclusion 🌟</h3><p>These advanced statistical algorithms provide powerful solutions for complex data science challenges, from handling high-dimensional data and capturing non-linear dependencies to making real-time predictions. By mastering these lesser-known techniques, you can significantly enhance your analytical capabilities and stay ahead in the data science game. So, when faced with a tricky dataset, think beyond the basics and dive into these advanced algorithms to unlock deeper insights! 🚀</p><p>Happy Data Science! 📊🔍</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4d6cffcd4a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sales Insights: From Loyalty Programs to Retention Through RFM Analysis, Customer Segmentation, and…]]></title>
            <link>https://medium.com/@sarowar.saurav10/sales-insights-from-loyalty-programs-to-retention-through-rfm-analysis-customer-segmentation-and-06f96cffd5d0?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/06f96cffd5d0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Tue, 20 Aug 2024 11:01:45 GMT</pubDate>
            <atom:updated>2024-08-20T11:01:45.522Z</atom:updated>
            <content:encoded><![CDATA[<h3>Sales Insights: From Loyalty Programs to Retention Through RFM Analysis, Customer Segmentation, and Forecasting Beyond! 🚀</h3><p>In today’s data-driven world, understanding your customers is key to driving business growth. Whether you’re in e-commerce, SaaS, or retail, harnessing the power of data can unlock a treasure trove of insights. This blog will take you through a journey of customer analytics, diving into techniques like RFM analysis, customer segmentation, trend analysis, and even churn prediction. Buckle up, as we explore how these methods can elevate your understanding of customer behavior and boost your business strategy! 🎢</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*zC79mA_QITZ0f8pMUNmEJg.png" /></figure><h4>Introduction to the Dataset 🌍</h4><p>We’re diving into the <strong>Online Retail II</strong> dataset, a treasure trove of sales information from a UK-based company that sold souvenirs between December 1, 2009, and December 9, 2011. Think of these items as the kind of promotional products that wholesalers love to stock up on!</p><p>This dataset offers a rich look at transactions, giving us a unique view into the company’s customer base, which mainly consists of wholesalers.</p><p><strong>Here’s a quick rundown of what’s inside:</strong></p><p>InvoiceNo | StockCode | Description | Quantity | InvoiceDate | UnitPrice | CustomerID | Country</p><p>Access Link to Dataset : <a href="https://archive.ics.uci.edu/ml/datasets/Online+Retail+II">https://archive.ics.uci.edu/ml/datasets/Online+Retail+II</a></p><h4>1. Data Preprocessing: Setting the Stage 🎬</h4><p>Before diving into the analytics, we need to clean and prep our data. This crucial step ensures that our analyses are based on accurate and meaningful data. Here’s what we did:</p><ul><li><strong>Loaded the data</strong> 📊: We used an online retail dataset that captures transactions over time. Here is the link :</li><li><strong>Cleaned the data</strong> 🧹: Removed returned products (invoices starting with “C”) and handled missing values.</li><li><strong>Set a reference date</strong> 📅: We set our analysis reference date as the last invoice date in the dataset.</li></ul><pre>import pandas as pd<br># Date<br>import datetime as dt<br># Settings<br>import warnings<br><br>df = pd.read_csv(&#39;../input/online-retail-ii-uci/online_retail_II.csv&#39;)<br><br>df.info()<br><br># Removing returned products (Invoice numbers starting with C) from the data set<br>df = df[~df[&quot;Invoice&quot;].str.contains(&quot;C&quot;, na = False)]<br># Removing missing values from the dataset<br>df.dropna(inplace = True)</pre><h4>2. RFM Analysis: The Foundation of Customer Segmentation 🧱</h4><p>RFM stands for Recency, Frequency, and Monetary. It’s a powerful tool to evaluate customers based on how recently they made a purchase, how often they purchase, and how much they spend. Here’s a breakdown:</p><ul><li><strong>Recency (R)</strong>: How recently a customer made a purchase. In order to find the recency value of each customer, we need to determine the last invoice date as the current date and subtract the last purchasing date of each customer from this date.</li></ul><pre>df[&quot;InvoiceDate&quot;].max() # Last invoice date<br>today_date = dt.datetime(2011,12,9) # last invoice date is assigned to today_date variable<br><br># The type of Customer ID variable needs to be turned into an integer for following commands.<br>df[&quot;Customer ID&quot;] = df[&quot;Customer ID&quot;].astype(int) <br><br># The type of InvoiceDate variable needs to be turned into datetime for following commands.<br>df[&quot;InvoiceDate&quot;] = pd.to_datetime(df[&quot;InvoiceDate&quot;])<br><br># Grouping the last invoice dates according to the Customer ID variable, subtracting them from today_date, and assigning them as recency<br>recency = (today_date - df.groupby(&quot;Customer ID&quot;).agg({&quot;InvoiceDate&quot;:&quot;max&quot;}))<br># Rename column name as Recency<br>recency.rename(columns = {&quot;InvoiceDate&quot;:&quot;Recency&quot;}, inplace = True)<br># Change the values to day format<br>recency_df = recency[&quot;Recency&quot;].apply(lambda x: x.days)<br>recency_df.head()</pre><ul><li><strong>Frequency (F)</strong>: How often a customer makes a purchase.In order to find the frequency value of each customer, we need to determine how many times the customers make purchases.</li></ul><pre># Grouping unique values of invoice date according to customer_id variable and assigning them to freq_df variable<br>freq_df = df.groupby(&quot;Customer ID&quot;).agg({&quot;InvoiceDate&quot;:&quot;nunique&quot;}) <br># Rename column name as Frequency<br>freq_df.rename(columns={&quot;InvoiceDate&quot;: &quot;Frequency&quot;}, inplace=True)<br>freq_df.head()</pre><ul><li><strong>Monetary (M)</strong>: How much a customer spends. In order to find the monetary value of each customer, we need to determine how much do the customers spend on purchases</li></ul><pre># Multiplying the prices and quantities of purchased products and assigning them to the total price variable<br>df[&quot;TotalPrice&quot;] = df[&quot;Quantity&quot;] * df[&quot;Price&quot;]<br><br># Grouping and sum up total prices according to each Customer ID<br>monetary_df = df.groupby(&quot;Customer ID&quot;).agg({&quot;TotalPrice&quot;:&quot;sum&quot;})<br># Rename Total Price column as Monetary<br>monetary_df.rename(columns={&quot;TotalPrice&quot;:&quot;Monetary&quot;}, inplace=True)<br>monetary_df.head()<br><br>rfm = pd.concat([recency_df, freq_df, monetary_df],  axis=1)<br>rfm.head()</pre><p>By calculating RFM scores, we can segment customers into different groups. For instance:</p><ul><li><strong>Champions</strong> 🏆: Customers with high RFM scores. They are your most valuable customers.</li><li><strong>Potential Loyalists</strong> 🌟: Customers with recent purchases and moderate frequency, but lower monetary value.</li><li><strong>At Risk</strong> 🚨: Customers who haven’t purchased in a while.</li></ul><pre># Dividing the recency values into recency scores such that the lowest recency value as 5 and the highest as 1<br>rfm[&quot;RecencyScore&quot;] = pd.qcut(rfm[&quot;Recency&quot;], 5, labels = [5, 4 , 3, 2, 1])<br> <br># Dividing the frequency values into frequency scores such that the lowest frequency value as 1 and the highest as 5<br>rfm[&quot;FrequencyScore&quot;]= pd.qcut(rfm[&quot;Frequency&quot;].rank(method=&quot;first&quot;),5, labels=[1,2,3,4,5])<br><br># Dividing the monetary values into monetary scores such that the lowest monetary value as 1 and the highest as 5<br>rfm[&quot;MonetaryScore&quot;] = pd.qcut(rfm[&#39;Monetary&#39;], 5, labels = [1, 2, 3, 4, 5])<br><br># Combining Recency, Frequency, and Monetary Scores in a string format<br>rfm[&quot;RFM_SCORE&quot;] = (rfm[&#39;RecencyScore&#39;].astype(str) + <br>                    rfm[&#39;FrequencyScore&#39;].astype(str) + <br>                    rfm[&#39;MonetaryScore&#39;].astype(str))<br><br><br># Customers with best scores<br>rfm[rfm[&quot;RFM_SCORE&quot;]==&quot;555&quot;].head()<br><br># Customers with worst scores<br>rfm[rfm[&quot;RFM_SCORE&quot;]==&quot;111&quot;].head()</pre><p>This segmentation allows us to tailor marketing strategies to each group, improving customer retention and satisfaction.</p><h4>3. Customer Segmentation with KNN: Taking It Up a Notch 📈</h4><p>While RFM analysis is powerful, we can further refine our segmentation using machine learning. We employed the K-Nearest Neighbors (KNN) algorithm to cluster customers based on their RFM scores. This approach considers the proximity of customers in the RFM space, grouping similar customers together.</p><pre># Mapping of segments according to recency and frequency scores of customers<br>seg_map = {<br>    r&#39;[1-2][1-2]&#39;: &#39;Hibernating&#39;,<br>    r&#39;[1-2][3-4]&#39;: &#39;At Risk&#39;,<br>    r&#39;[1-2]5&#39;: &#39;Can\&#39;t Loose&#39;,<br>    r&#39;3[1-2]&#39;: &#39;About to Sleep&#39;,<br>    r&#39;33&#39;: &#39;Need Attention&#39;,<br>    r&#39;[3-4][4-5]&#39;: &#39;Loyal Customers&#39;,<br>    r&#39;41&#39;: &#39;Promising&#39;,<br>    r&#39;51&#39;: &#39;New Customers&#39;,<br>    r&#39;[4-5][2-3]&#39;: &#39;Potential Loyalists&#39;,<br>    r&#39;5[4-5]&#39;: &#39;Champions&#39;<br>}<br><br># Recency and Frequency scores are turned into string format, combined and assigned to Segment<br>rfm[&#39;Segment&#39;] = rfm[&#39;RecencyScore&#39;].astype(str) + rfm[&#39;FrequencyScore&#39;].astype(str)<br><br># Segments are changed with the definitons of seg_map<br>rfm[&#39;Segment&#39;] = rfm[&#39;Segment&#39;].replace(seg_map, regex=True)<br><br># Mean, median, count statistics of different segments<br>rfm[[&quot;Segment&quot;,&quot;Recency&quot;,&quot;Frequency&quot;, &quot;Monetary&quot;]].groupby(&quot;Segment&quot;).agg([&quot;mean&quot;,&quot;median&quot;,&quot;count&quot;])</pre><p>The result? Even more granular segments that can be targeted with precision. For example, a customer close to a “Champion” in RFM space might be nudged towards higher spending or more frequent purchases.</p><h4>4. Trend Analytics: Uncovering Patterns Over Time 📅</h4><p>Understanding trends in your data can reveal valuable insights about customer behavior and market dynamics. We performed trend analysis to track customer activity over time, identifying patterns like:</p><p>Aggregate Data by Time Periods: Choose the time period for trend analysis (e.g., monthly, quarterly).</p><p>Analyze Key Metrics: Examine metrics like total revenue, number of customers, and average order value for each segment over time.</p><p>Visualize Trends: Use plots to visualize trends in key metrics for each segment.</p><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br><br># Assuming `rfm` DataFrame has been created as per your notebook<br><br># Add a column for month<br>df[&#39;InvoiceMonth&#39;] = df[&#39;InvoiceDate&#39;].dt.to_period(&#39;M&#39;)<br><br># Merge RFM data with original DataFrame to get segments<br>df = df.merge(rfm[[&#39;Segment&#39;]], left_on=&#39;Customer ID&#39;, right_index=True)<br><br># Group by segment and month to get total revenue and number of customers<br>trend_data = df.groupby([&#39;Segment&#39;, &#39;InvoiceMonth&#39;]).agg({<br>    &#39;TotalPrice&#39;: &#39;sum&#39;,<br>    &#39;Customer ID&#39;: &#39;nunique&#39;<br>}).reset_index()<br><br># Rename columns for clarity<br>trend_data.rename(columns={&#39;TotalPrice&#39;: &#39;TotalRevenue&#39;, &#39;Customer ID&#39;: &#39;NumCustomers&#39;}, inplace=True)<br><br># Plot Total Revenue by Segment<br>plt.figure(figsize=(12, 6))<br>for segment in trend_data[&#39;Segment&#39;].unique():<br>    segment_data = trend_data[trend_data[&#39;Segment&#39;] == segment]<br>    plt.plot(segment_data[&#39;InvoiceMonth&#39;].astype(str), segment_data[&#39;TotalRevenue&#39;], label=segment)<br><br>plt.title(&#39;Total Revenue by Customer Segment Over Time&#39;)<br>plt.xlabel(&#39;Month&#39;)<br>plt.ylabel(&#39;Total Revenue&#39;)<br>plt.legend()<br>plt.xticks(rotation=45)<br>plt.tight_layout()<br>plt.show()<br><br># Plot Number of Customers by Segment<br>plt.figure(figsize=(12, 6))<br>for segment in trend_data[&#39;Segment&#39;].unique():<br>    segment_data = trend_data[trend_data[&#39;Segment&#39;] == segment]<br>    plt.plot(segment_data[&#39;InvoiceMonth&#39;].astype(str), segment_data[&#39;NumCustomers&#39;], label=segment)<br><br>plt.title(&#39;Number of Customers by Segment Over Time&#39;)<br>plt.xlabel(&#39;Month&#39;)<br>plt.ylabel(&#39;Number of Customers&#39;)<br>plt.legend()<br>plt.xticks(rotation=45)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*P4vQpz6d9CAlomHaBVGwpg.png" /></figure><p>These insights help in planning inventory, marketing campaigns, and even forecasting future sales.</p><h4>5. Cohort Analysis: Understanding Customer Lifecycles ⏳</h4><p>Cohort analysis allows us to group customers based on shared characteristics, typically the time of their first purchase. By tracking these cohorts over time, we can observe how different groups evolve, providing insights into customer retention and the effectiveness of marketing strategies.</p><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br># Assuming `df` is your DataFrame<br><br># Convert InvoiceDate to datetime<br>df[&#39;InvoiceDate&#39;] = pd.to_datetime(df[&#39;InvoiceDate&#39;])<br><br># Create a column for the month of the first purchase for each customer<br>df[&#39;CohortMonth&#39;] = df.groupby(&#39;Customer ID&#39;)[&#39;InvoiceDate&#39;].transform(&#39;min&#39;).dt.to_period(&#39;M&#39;)<br><br># Create a column for the month of each purchase<br>df[&#39;PurchaseMonth&#39;] = df[&#39;InvoiceDate&#39;].dt.to_period(&#39;M&#39;)<br><br># Calculate the cohort index (months since first purchase)<br>df[&#39;CohortIndex&#39;] = (df[&#39;PurchaseMonth&#39;] - df[&#39;CohortMonth&#39;]).apply(lambda x: x.n)<br><br># Group by Country, CohortMonth, and CohortIndex to get the number of unique customers<br>cohort_data = df.groupby([&#39;Country&#39;, &#39;CohortMonth&#39;, &#39;CohortIndex&#39;]).agg({&#39;Customer ID&#39;: &#39;nunique&#39;}).reset_index()<br><br># Pivot the data to make CohortIndex the columns<br>cohort_pivot = cohort_data.pivot_table(index=[&#39;Country&#39;, &#39;CohortMonth&#39;], columns=&#39;CohortIndex&#39;, values=&#39;Customer ID&#39;)<br><br># Plot the cohort analysis for a specific country<br>country = &#39;United Kingdom&#39;  # Change this to any country you want to analyze<br><br>plt.figure(figsize=(12, 8))<br>sns.heatmap(cohort_pivot.loc[country], annot=True, fmt=&quot;.0f&quot;, cmap=&quot;Blues&quot;)<br>plt.title(f&#39;Cohort Analysis for {country}&#39;)<br>plt.xlabel(&#39;Cohort Index (Months since first purchase)&#39;)<br>plt.ylabel(&#39;Cohort Month&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*SD7C6F0qWKGB6t071S48yQ.png" /></figure><p>For instance, we might find that customers acquired during a holiday season have a higher churn rate. Armed with this knowledge, we can tailor our approach to improve retention for future cohorts.</p><h4>6. CLV Analysis: The Golden Metric 💰</h4><p>Customer Lifetime Value (CLV) is the total revenue a business can expect from a single customer throughout their relationship. It’s a crucial metric for understanding the long-term value of your customer base.</p><pre>import pandas as pd<br><br># Assuming `df` is your DataFrame<br># Calculate total revenue<br>total_revenue = df[&#39;TotalPrice&#39;].sum()<br><br># Calculate number of orders and unique customers<br>num_orders = df[&#39;Invoice&#39;].nunique()<br>num_customers = df[&#39;Customer ID&#39;].nunique()<br><br># Calculate Average Order Value (AOV)<br>aov = total_revenue / num_orders<br><br># Calculate Purchase Frequency (PF)<br>pf = num_orders / num_customers<br><br># Calculate Customer Value (CV)<br>cv = aov * pf<br><br># Calculate Average Customer Lifespan (ACL)<br># Assuming an average lifespan of 1 year for simplicity<br>acl = 1<br><br># Calculate CLV<br>clv = cv * acl<br><br>print(f&#39;Average Order Value (AOV): {aov:.2f}&#39;)<br>print(f&#39;Purchase Frequency (PF): {pf:.2f}&#39;)<br>print(f&#39;Customer Value (CV): {cv:.2f}&#39;)<br>print(f&#39;Customer Lifetime Value (CLV): {clv:.2f}&#39;)<br><br>**Output**<br><br>Average Order Value (AOV): 479.88<br>Purchase Frequency (PF): 6.29<br>Customer Value (CV): 3017.08<br>Customer Lifetime Value (CLV): 3017.08</pre><p>By calculating CLV, we can:</p><ul><li><strong>Identify high-value customers</strong> 🤑: Focus marketing efforts on retaining and nurturing these customers.</li><li><strong>Optimize acquisition costs</strong> 💼: Ensure that the cost of acquiring a customer is justified by their lifetime value.</li></ul><h4>7. Churn Prediction: Staying Ahead of the Curve 🏃‍♂️</h4><p>Churn prediction is all about identifying customers at risk of leaving. We used machine learning models to predict which customers are likely to churn based on their past behavior. By analyzing factors like purchase frequency, recency, and spending patterns, we can proactively address churn and improve customer retention.</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import classification_report<br><br># Assuming `df` is your DataFrame<br># Define churn: customers who haven&#39;t purchased in the last 6 months<br>df[&#39;LastPurchase&#39;] = df.groupby(&#39;Customer ID&#39;)[&#39;InvoiceDate&#39;].transform(&#39;max&#39;)<br>df[&#39;Churn&#39;] = (df[&#39;LastPurchase&#39;] &lt; &#39;2011-06-01&#39;).astype(int)<br><br># Feature engineering<br>features = df.groupby(&#39;Customer ID&#39;).agg({<br>    &#39;InvoiceDate&#39;: lambda x: (x.max() - x.min()).days,<br>    &#39;Invoice&#39;: &#39;nunique&#39;,<br>    &#39;TotalPrice&#39;: &#39;sum&#39;<br>}).rename(columns={<br>    &#39;InvoiceDate&#39;: &#39;Recency&#39;,<br>    &#39;Invoice&#39;: &#39;Frequency&#39;,<br>    &#39;TotalPrice&#39;: &#39;Monetary&#39;<br>})<br><br># Add the target variable<br>features[&#39;Churn&#39;] = df.groupby(&#39;Customer ID&#39;)[&#39;Churn&#39;].first()<br><br># Split the data<br>X = features.drop(&#39;Churn&#39;, axis=1)<br>y = features[&#39;Churn&#39;]<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># Train a Random Forest model<br>model = RandomForestClassifier(random_state=42)<br>model.fit(X_train, y_train)<br><br># Evaluate the model<br>y_pred = model.predict(X_test)<br>print(classification_report(y_test, y_pred))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*riUwXxkl4ZjXDRqbYNVrQQ.png" /></figure><h4>8. Looking Forward: Sales Forecasting with SARIMA 🔮</h4><p>The journey doesn’t end here! To take your analytics to the next level, consider extending your analysis to include sales forecasting using the SARIMA (Seasonal AutoRegressive Integrated Moving Average) model. This advanced time-series model can help you predict future sales, enabling better inventory management and financial planning.</p><h4>Conclusion: The Power of Data-Driven Decisions 💡</h4><p>By leveraging these analytics techniques, you can transform raw data into actionable insights. Whether you’re segmenting customers, predicting churn, or forecasting sales, the key is to use data to drive your decisions. So, what’s next on your data journey?</p><p>Let’s continue to explore, analyze, and unlock the secrets hidden in our data! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06f96cffd5d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL Cheat Sheet for Data Science Enthusiasts: 32 Advanced & Complex Queries You Must Know ]]></title>
            <link>https://medium.com/@sarowar.saurav10/sql-cheat-sheet-for-data-science-enthusiasts-32-advanced-complex-queries-you-must-know-10c5f20caacc?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/10c5f20caacc</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[cheatsheet]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Fri, 28 Jun 2024 14:01:36 GMT</pubDate>
            <atom:updated>2024-06-28T14:01:36.603Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey Data Science Enthusiasts! 👩‍💻👨‍💻</p><p>Welcome to an advanced SQL cheat sheet designed to boost your data analytics skills. Whether you’re diving deep into data warehouses or crunching big datasets, these advanced SQL queries will help you extract, manipulate, and analyze data like a pro. Let’s dive right in! 🏊‍♂️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*7M0HcP3RDPhUGTYb6Jrn2Q.png" /></figure><h4>1. Recursive Common Table Expressions (CTEs) 🌳</h4><p>Recursive CTEs are a powerful feature for working with hierarchical data such as organizational structures or graph data.</p><pre>WITH RECURSIVE EmployeeHierarchy AS (<br>    SELECT employee_id, name, manager_id, 1 AS level<br>    FROM employees<br>    WHERE manager_id IS NULL<br>    UNION ALL<br>    SELECT e.employee_id, e.name, e.manager_id, eh.level + 1<br>    FROM employees e<br>    JOIN EmployeeHierarchy eh ON e.manager_id = eh.employee_id<br>)<br>SELECT * FROM EmployeeHierarchy;</pre><h4>2. Window Functions: ROW_NUMBER(), RANK(), DENSE_RANK() 📊</h4><p>Window functions are incredibly useful for ranking and windowed aggregations.</p><pre>SELECT employee_id, salary,<br>       ROW_NUMBER() OVER (ORDER BY salary DESC) AS row_num,<br>       RANK() OVER (ORDER BY salary DESC) AS rank,<br>       DENSE_RANK() OVER (ORDER BY salary DESC) AS dense_rank<br>FROM employees;</pre><h4>3. Pivot Tables for Dynamic Reporting 🔄</h4><p>Pivot tables allow you to dynamically transform rows into columns.</p><pre>SELECT *<br>FROM (<br>    SELECT year, product, revenue<br>    FROM sales<br>) s<br>PIVOT (<br>    SUM(revenue)<br>    FOR year IN ([2019], [2020], [2021])<br>) AS p;</pre><h4>4. Advanced JOINs: CROSS JOIN, SELF JOIN 🔗</h4><p>Understand advanced joins to combine data in complex ways.</p><pre>-- CROSS JOIN<br>SELECT a.employee_id, b.department_id<br>FROM employees a<br>CROSS JOIN departments b;<br><br>-- SELF JOIN<br>SELECT e1.employee_id, e1.name AS employee, e2.name AS manager<br>FROM employees e1<br>JOIN employees e2 ON e1.manager_id = e2.employee_id;</pre><h4>5. Grouping Sets for Multiple Aggregations 📐</h4><p>Grouping sets allow you to define multiple groupings in a single query.</p><pre>SELECT department_id, job_id, AVG(salary)<br>FROM employees<br>GROUP BY GROUPING SETS (<br>    (department_id, job_id),<br>    (department_id),<br>    (job_id),<br>    ()<br>);</pre><h4>6. JSON Functions for Semi-Structured Data 📦</h4><p>Work with JSON data directly within your SQL queries.</p><pre>SELECT <br>    employee_id,<br>    JSON_VALUE(employee_info, &#39;$.address.city&#39;) AS city<br>FROM employees<br>WHERE JSON_VALUE(employee_info, &#39;$.skills[0]&#39;) = &#39;SQL&#39;;</pre><h4>7. Temporal Tables for Time-Travel Queries 🕰️</h4><p>Temporal tables let you query historical data.</p><pre>SELECT * FROM employees FOR SYSTEM_TIME AS OF &#39;2023-01-01 00:00:00&#39;;</pre><h4>8. Hierarchical Queries Using CONNECT BY 🔗</h4><p>For databases that support it, hierarchical queries can navigate parent-child relationships.</p><pre>SELECT employee_id, name, manager_id, LEVEL<br>FROM employees<br>CONNECT BY PRIOR employee_id = manager_id<br>START WITH manager_id IS NULL;</pre><h4>9. Full-Text Search 🔍</h4><p>Use full-text search to efficiently find text within large datasets.</p><pre>SELECT employee_id, name<br>FROM employees<br>WHERE CONTAINS(name, &#39;Data Scientist&#39;);</pre><h4>10. Window Functions: LAG() and LEAD() ⏭️</h4><p>Analyze data trends by accessing previous and next rows within a window.</p><pre>SELECT employee_id, salary,<br>       LAG(salary, 1) OVER (ORDER BY hire_date) AS previous_salary,<br>       LEAD(salary, 1) OVER (ORDER BY hire_date) AS next_salary<br>FROM employees;</pre><h4>11. CUBE and ROLLUP for Comprehensive Aggregations 🎲</h4><p>Advanced grouping with CUBE and ROLLUP for multidimensional analysis.</p><pre>-- CUBE<br>SELECT department_id, job_id, SUM(salary)<br>FROM employees<br>GROUP BY CUBE (department_id, job_id);<br><br>-- ROLLUP<br>SELECT department_id, job_id, SUM(salary)<br>FROM employees<br>GROUP BY ROLLUP (department_id, job_id);</pre><h4>12. Geospatial Queries for Location Data 🌍</h4><p>Analyze spatial data directly within SQL.</p><pre>SELECT name, ST_Distance(location, &#39;POINT(-73.935242 40.730610)&#39;) <br>AS distance FROM places<br>WHERE ST_DWithin(location, &#39;POINT(-73.935242 40.730610)&#39;, 1000);</pre><h4>13. Materialized Views for Optimization 🏎️</h4><p>Materialized views store query results for faster access.</p><pre>CREATE MATERIALIZED VIEW EmployeeSalaries AS<br>SELECT department_id, AVG(salary) AS avg_salary<br>FROM employees<br>GROUP BY department_id;<br><br>REFRESH MATERIALIZED VIEW EmployeeSalaries;</pre><h4>14. Recursive Queries for Graph Data 🌐</h4><p>Recursive queries are great for working with graph data structures.</p><pre>WITH RECURSIVE graph AS (<br>    SELECT id, name, parent_id<br>    FROM nodes<br>    WHERE parent_id IS NULL<br>    UNION ALL<br>    SELECT n.id, n.name, n.parent_id<br>    FROM nodes n<br>    JOIN graph g ON n.parent_id = g.id<br>)<br>SELECT * FROM graph;</pre><h4>15. Advanced String Manipulation ✂️</h4><p>Powerful string functions to manipulate and analyze text data.</p><pre>SELECT employee_id, name,<br>       SUBSTRING(name, 1, 5) AS short_name,<br>       CHARINDEX(&#39; &#39;, name) AS first_space,<br>       LEFT(name, CHARINDEX(&#39; &#39;, name) - 1) AS first_name<br>FROM employees;</pre><h4>16. User-Defined Functions (UDFs) 🔧</h4><p>Create custom functions to reuse logic in your queries.</p><pre>CREATE FUNCTION CalculateAge(@birthdate DATE)<br>RETURNS INT<br>AS<br>BEGIN<br>    RETURN DATEDIFF(YEAR, @birthdate, GETDATE());<br>END;<br><br>SELECT employee_id, name, dbo.CalculateAge(birthdate) AS age<br>FROM employees;</pre><h4>17. Correlated Subqueries 🕵️</h4><p>Use correlated subqueries for row-by-row operations.</p><pre>SELECT e1.employee_id, e1.name, e1.salary<br>FROM employees e1<br>WHERE e1.salary &gt; (<br>    SELECT AVG(e2.salary)<br>    FROM employees e2<br>    WHERE e2.department_id = e1.department_id<br>);</pre><h4>18. Common Table Expressions (CTEs) for Modular Queries 📜</h4><p>CTEs make complex queries more readable and maintainable.</p><pre>WITH HighEarners AS (<br>    SELECT employee_id, name, salary<br>    FROM employees<br>    WHERE salary &gt; 100000<br>)<br>SELECT * FROM HighEarners;</pre><h4>19. Indexing for Performance Boost 🚀</h4><p>Improve query performance with appropriate indexing.</p><pre>CREATE INDEX idx_employee_name ON employees(name);<br><br>SELECT employee_id, name<br>FROM employees<br>WHERE name LIKE &#39;Saurav%&#39;;</pre><h3>20. Working with XML Data 📜</h3><p>Query and manipulate XML data directly in SQL.</p><pre>SELECT <br>    employee_id,<br>    employee_info.value(&#39;(/Employee/Address/City)[1]&#39;, &#39;NVARCHAR(100)&#39;) AS city<br>FROM employees<br>WHERE employee_info.exist(&#39;/Employee/Skills/Skill[text()=&quot;SQL&quot;]&#39;) = 1;</pre><h4>21. Analytical Functions: PERCENT_RANK(), NTILE() 📊</h4><p>Analytical functions help with data analysis tasks.</p><pre>SELECT employee_id, salary,<br>       PERCENT_RANK() OVER (ORDER BY salary DESC) AS percent_rank,<br>       NTILE(4) OVER (ORDER BY salary DESC) AS quartile<br>FROM employees;</pre><h4>22. Conditional Aggregations with CASE Statements 🧮</h4><p>Use CASE statements for conditional aggregations.</p><pre>SELECT department_id,<br>       SUM(CASE WHEN gender = &#39;M&#39; THEN salary ELSE 0 END) AS male_salary,<br>       SUM(CASE WHEN gender = &#39;F&#39; THEN salary ELSE 0 END) AS female_salary<br>FROM employees<br>GROUP BY department_id;</pre><h4><strong>23. Data Sampling for Quick Insights 🔍</strong></h4><p>Sample to quickly gather insights without processing the entire dataset.</p><pre>SELECT *<br>FROM employees<br>TABLESAMPLE (10 PERCENT);</pre><h4>24. Advanced Date Functions 🗓️</h4><p>Work with dates and times more effectively.</p><pre>SELECT employee_id, name,<br>       DATEADD(YEAR, 1, hire_date) AS next_year_anniversary,<br>       DATEDIFF(DAY, hire_date, GETDATE()) AS days_with_company<br>FROM employees;</pre><h4>25. Advanced Error Handling and Transaction Control 🛡️</h4><p>Ensure data integrity with transactions.</p><pre>BEGIN TRANSACTION;<br><br>BEGIN TRY<br>    UPDATE accounts SET balance = balance - 100 WHERE account_id = 1;<br>    UPDATE accounts SET balance = balance + 100 WHERE account_id = 2;<br>    COMMIT TRANSACTION;<br>END TRY<br>BEGIN CATCH<br>    ROLLBACK TRANSACTION;<br>    THROW;<br>END CATCH;</pre><h4>26. Statistical Functions for Data Analysis 📈</h4><p>Perform statistical analysis directly in SQL.</p><pre>SELECT <br>    department_id,<br>    AVG(salary) AS avg_salary,<br>    STDDEV(salary) AS salary_stddev,<br>    VARIANCE(salary) AS salary_variance,<br>    PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY salary) AS median_salary<br>FROM <br>    employees<br>GROUP BY <br>    department_id;</pre><h4>27. Dynamic SQL for Flexible Querying 🧩</h4><p>Dynamic SQL allows you to build and execute SQL queries dynamically.</p><pre>DECLARE @sql NVARCHAR(MAX);<br>SET @sql = &#39;SELECT employee_id, name, salary FROM employees <br>WHERE department_id = &#39; + CAST(@dept_id AS NVARCHAR);<br>EXEC sp_executesql @sql;</pre><h4>28. Advanced Subqueries: EXISTS, NOT EXISTS 🔄</h4><p>Leverage advanced subqueries for complex filtering.</p><pre>-- EXISTS<br>SELECT employee_id, name<br>FROM employees e<br>WHERE EXISTS (<br>    SELECT 1<br>    FROM projects p<br>    WHERE p.employee_id = e.employee_id<br>);<br><br>-- NOT EXISTS<br>SELECT employee_id, name<br>FROM employees e<br>WHERE NOT EXISTS (<br>    SELECT 1<br>    FROM projects p<br>    WHERE p.employee_id = e.employee_id<br>);</pre><h4>29. Advanced UPDATE and DELETE with JOINs 🗑️</h4><p>Use JOINs in UPDATE and DELETE statements for complex operations.</p><pre>-- UPDATE with JOIN<br>UPDATE e<br>SET e.salary = e.salary * 1.10<br>FROM employees e<br>JOIN departments d ON e.department_id = d.department_id<br>WHERE d.department_name = &#39;Sales&#39;;<br><br>-- DELETE with JOIN<br>DELETE e<br>FROM employees e<br>JOIN departments d ON e.department_id = d.department_id<br>WHERE d.department_name = &#39;HR&#39;;</pre><h4>30. Row Pattern Recognition with MATCH_RECOGNIZE 🕵️</h4><p>Detect patterns within sequences of rows.</p><pre>SELECT *<br>FROM orders<br>MATCH_RECOGNIZE (<br>    PARTITION BY customer_id<br>    ORDER BY order_date<br>    MEASURES<br>        MATCH_NUMBER() AS match_num,<br>        CLASSIFIER() AS match_type<br>    PATTERN (strt ord+)<br>    DEFINE<br>        ord AS (order_date - prev(order_date)) &lt;= 7<br>);</pre><h4>31. Hierarchical Queries with PATH Expressions 🌲</h4><p>For databases that support path expressions, you can navigate and visualize hierarchical data.</p><pre>WITH RECURSIVE EmployeePaths AS (<br>    SELECT employee_id, name, manager_id, name AS path<br>    FROM employees<br>    WHERE manager_id IS NULL<br>    UNION ALL<br>    SELECT e.employee_id, e.name, e.manager_id, CONCAT(ep.path, &#39; -&gt; &#39;, e.name)<br>    FROM employees e<br>    JOIN EmployeePaths ep ON e.manager_id = ep.employee_id<br>)<br>SELECT * FROM EmployeePaths;</pre><h4>32. Temporal Analytics with INTERVALS ⏳</h4><p>Perform temporal analytics using intervals to calculate trends over time.</p><pre>SELECT<br>    DATE_TRUNC(&#39;month&#39;, order_date) AS month,<br>    COUNT(order_id) AS orders,<br>    LAG(COUNT(order_id), 1) OVER (ORDER BY DATE_TRUNC(&#39;month&#39;, order_date)) AS previous_month_orders,<br>    COUNT(order_id) - LAG(COUNT(order_id), 1) OVER (ORDER BY DATE_TRUNC(&#39;month&#39;, order_date)) AS order_change<br>FROM orders<br>GROUP BY DATE_TRUNC(&#39;month&#39;, order_date)<br>ORDER BY month;</pre><h3>Conclusion 🎉</h3><p>Congratulations, data professionals! 🧙‍♂️🧙‍♀️ You have now equipped yourself with 32 advanced SQL queries designed to elevate your data analytics skills to new heights. From recursive CTEs and window functions to temporal analytics and statistical functions, these sophisticated tools enable you to extract, analyze, and manipulate data with exceptional precision and efficiency.</p><p>With these queries in your toolkit, you are well-prepared to tackle complex data challenges, uncover hidden patterns, and derive actionable insights from your datasets. Whether you are optimizing performance, conducting in-depth analysis, or creating dynamic reports, these SQL techniques will significantly enhance your capabilities as a data scientist.</p><p>Mastering SQL requires continuous practice and exploration. Do not hesitate to experiment with different queries and datasets. The more you engage with data, the more proficient you will become at solving intricate problems and making data-driven decisions.</p><p>Continue pushing the boundaries of your SQL knowledge, maintain your curiosity, and advance on your journey towards data excellence. If you found this cheat sheet valuable, please give it a clap and share it with your fellow data enthusiasts. Together, we can foster a community of skilled data scientists ready to transform the world through data, one query at a time. 🚀</p><p>Thank you for reading, and happy querying! 🌟</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10c5f20caacc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turning Your CSV into a Secure Accessible Endpoint API with FastAPI and JWT Authentication ]]></title>
            <link>https://medium.com/@sarowar.saurav10/turning-your-csv-into-a-secure-api-with-fastapi-and-jwt-authentication-ef2a53b01233?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef2a53b01233</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[fastapi]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Sat, 15 Jun 2024 05:54:49 GMT</pubDate>
            <atom:updated>2024-06-15T06:00:26.634Z</atom:updated>
            <content:encoded><![CDATA[<p>Ever wondered how you can take a basic CSV file and transform it into a blazing-fast and secure API? With FastAPI, this task becomes not only possible but downright enjoyable. In this blog, we’ll guide you through the process step-by-step, complete with code snippets and a test case. Let’s dive in! 🏊‍♂️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*Vuz79bVyd8LUtxm1.png" /></figure><h3>Why FastAPI? 🤔</h3><p>FastAPI is a modern, fast (high-performance) web framework for building APIs with Python 3.7+ based on standard Python type hints. It is designed to be easy to use and to make your code easy to understand and maintain. FastAPI is gaining popularity because of its:</p><ul><li><strong>Speed</strong>: It’s one of the fastest Python frameworks.</li><li><strong>Ease of Use</strong>: It comes with automatic interactive API documentation.</li><li><strong>Out-of-the-Box Features</strong>: FastAPI handles everything from request validation to interactive documentation with Swagger and ReDoc.</li></ul><h3>Step-by-Step Guide 🛠️</h3><h3>Step 1: Install FastAPI and Uvicorn</h3><p>First, you need to install FastAPI and Uvicorn (a lightning-fast ASGI server) using pip:</p><pre>pip install fastapi uvicorn pandas pyjwt</pre><h3>Step 2: Create Your CSV File</h3><p>Let’s start with a simple CSV file named data.csv:</p><pre>id,name,age,city<br>1,Shunno Sarkar,30,Dhaka<br>2,Lord Liton,25,Rangpur<br>3,Tamim Al Hasan,35,Khulna</pre><h3>Step 3: Set Up Your FastAPI Project</h3><p>Create a new Python file, main.py, and start by importing the necessary libraries:</p><pre>from fastapi import FastAPI, Depends, HTTPException, status<br>from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm<br>from pydantic import BaseModel<br>import pandas as pd<br>import jwt<br>from datetime import datetime, timedelta<br><br>app = FastAPI()<br><br># Secret key to encode and decode JWT tokens<br>SECRET_KEY = &quot;mysecretkey&quot;<br>ALGORITHM = &quot;HS256&quot;<br><br>class Person(BaseModel):<br>    id: int<br>    name: str<br>    age: int<br>    city: str<br><br>class User(BaseModel):<br>    username: str<br>    password: str<br><br># In-memory user store for example purposes<br>users_db = {<br>    &quot;user1&quot;: {<br>        &quot;username&quot;: &quot;user1&quot;,<br>        &quot;password&quot;: &quot;password&quot;<br>    }<br>}<br><br># Load CSV data into a DataFrame<br>df = pd.read_csv(&quot;data.csv&quot;)<br><br>oauth2_scheme = OAuth2PasswordBearer(tokenUrl=&quot;token&quot;)<br><br>def create_access_token(data: dict, expires_delta: timedelta = None):<br>    to_encode = data.copy()<br>    if expires_delta:<br>        expire = datetime.utcnow() + expires_delta<br>    else:<br>        expire = datetime.utcnow() + timedelta(minutes=15)<br>    to_encode.update({&quot;exp&quot;: expire})<br>    encoded_jwt = jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)<br>    return encoded_jwt<br><br>def verify_token(token: str):<br>    try:<br>        payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])<br>        return payload<br>    except jwt.PyJWTError:<br>        raise HTTPException(<br>            status_code=status.HTTP_401_UNAUTHORIZED,<br>            detail=&quot;Could not validate credentials&quot;,<br>            headers={&quot;WWW-Authenticate&quot;: &quot;Bearer&quot;},<br>        )<br><br>async def get_current_user(token: str = Depends(oauth2_scheme)):<br>    payload = verify_token(token)<br>    username = payload.get(&quot;sub&quot;)<br>    if username is None:<br>        raise HTTPException(status_code=401, detail=&quot;Invalid authentication credentials&quot;)<br>    return username<br><br>@app.post(&quot;/token&quot;)<br>async def login(form_data: OAuth2PasswordRequestForm = Depends()):<br>    user_dict = users_db.get(form_data.username)<br>    if not user_dict or user_dict[&quot;password&quot;] != form_data.password:<br>        raise HTTPException(<br>            status_code=status.HTTP_401_UNAUTHORIZED,<br>            detail=&quot;Incorrect username or password&quot;,<br>            headers={&quot;WWW-Authenticate&quot;: &quot;Bearer&quot;},<br>        )<br>    access_token_expires = timedelta(minutes=30)<br>    access_token = create_access_token(<br>        data={&quot;sub&quot;: form_data.username}, expires_delta=access_token_expires<br>    )<br>    return {&quot;access_token&quot;: access_token, &quot;token_type&quot;: &quot;bearer&quot;}<br><br>@app.get(&quot;/people&quot;, response_model=list[Person])<br>async def get_people(current_user: str = Depends(get_current_user)):<br>    people = df.to_dict(orient=&quot;records&quot;)<br>    return people<br><br>@app.get(&quot;/people/{person_id}&quot;, response_model=Person)<br>async def get_person(person_id: int, current_user: str = Depends(get_current_user)):<br>    person = df[df[&quot;id&quot;] == person_id].to_dict(orient=&quot;records&quot;)<br>    if person:<br>        return person[0]<br>    else:<br>        raise HTTPException(status_code=404, detail=&quot;Person not found&quot;)</pre><h3>Step 4: Run Your Application</h3><p>Run your FastAPI application using Uvicorn:</p><pre>uvicorn main:app --reload</pre><h3>Step 5: Test Your API</h3><p>First, you need to get a token by logging in. Open your browser and navigate to http://127.0.0.1:8000/docs. Click on the POST /token endpoint and use the default user credentials (user1/password) to obtain an access token.</p><p>Then, you can test the other endpoints by providing the token in the Authorize section. For example, navigate to http://127.0.0.1:8000/people to see all records, or http://127.0.0.1:8000/people/1 to see the record for the person with ID 1.</p><h3>Example Output 🎉</h3><p>When you visit http://127.0.0.1:8000/people with a valid token, you should see:</p><pre>[<br>    {<br>        &quot;id&quot;: 1,<br>        &quot;name&quot;: &quot;Shunno Sarkar&quot;,<br>        &quot;age&quot;: 30,<br>        &quot;city&quot;: &quot;Dhaka&quot;<br>    },<br>    {<br>        &quot;id&quot;: 2,<br>        &quot;name&quot;: &quot;Lord Liton&quot;,<br>        &quot;age&quot;: 25,<br>        &quot;city&quot;: &quot;Rangpur&quot;<br>    },<br>    {<br>        &quot;id&quot;: 3,<br>        &quot;name&quot;: &quot;Tamim Al Hasan&quot;,<br>        &quot;age&quot;: 35,<br>        &quot;city&quot;: &quot;Khulna&quot;<br>    }<br>]</pre><h3>Conclusion 🏁</h3><p>Congratulations! You’ve just transformed a simple CSV file into a powerful and secure API using FastAPI. This is just the beginning; you can expand this by adding more endpoints, filtering data, or even integrating with a database. FastAPI makes it easy and fun to build and scale your applications. Happy coding! 💻✨</p><p>Feel free to share your experiences or ask any questions in the comments below. Let’s keep learning together! 🤗</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef2a53b01233" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Cheat Sheet: 22 Python Code Snippets Every Data Enthusiast Should Know ]]></title>
            <link>https://medium.com/@sarowar.saurav10/22-python-code-snippets-every-data-enthusiast-should-know-a7cea27da3aa?source=rss-991b7159b87b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7cea27da3aa</guid>
            <category><![CDATA[code]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[cheatsheet]]></category>
            <dc:creator><![CDATA[Sarowar Jahan Saurav]]></dc:creator>
            <pubDate>Fri, 14 Jun 2024 16:02:18 GMT</pubDate>
            <atom:updated>2025-02-09T08:33:32.654Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello, data enthusiasts! 👋 Have you ever found yourself bogged down by repetitive data manipulation tasks, wishing there was a more efficient way to get things done? Well, you’re in luck! This cheat sheet is your ultimate guide to tackling everyday data analytics tasks with Python.</p><p>This cheat sheet wasn’t just born — it was crafted out of necessity and experience.I realized the importance of having a handy reference for the tasks and features I use most often. I aimed to capture the essential 20% of Python that addresses 80% of the typical programming needs in data analytics. The result? A curated collection of impactful Python code snippets that are immediately applicable to your projects and challenges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_SREz663efAOFNG7" /></figure><h3>Why This Cheat Sheet? 🚀</h3><p>This guide is the culmination of my journey back into Python, offering the most practical knowledge, insights, and useful libraries I’ve encountered along the way. It’s designed to be a quick-reference toolkit for data analysts, whether you’re a seasoned professional or a newcomer. The snippets are organized into logical sections, allowing you to jump directly to the area you need, be it data manipulation, visualization, modeling, or other essential tasks.</p><h3>Join the Conversation! 🗣️</h3><p>If you think I’ve missed anything that should be included in this cheat sheet, please let me know in the comments. Your feedback is invaluable, and I’ll update the list with any great suggestions.</p><p>So, without further ado, let’s dive into these 30 Python code snippets that will transform your daily data analytics tasks! Happy coding and analyzing! 📈🐍</p><h4><strong>1. </strong>Forecasting Data</h4><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import statsmodels.api as sm<br>from statsmodels.tsa.statespace.sarimax import SARIMAX<br>from sklearn.metrics import mean_squared_error<br><br># Step 1: Load and Prepare Data<br># Assume &#39;monthly_sales.csv&#39; has columns &#39;date&#39; and &#39;sales&#39;<br>df = pd.read_csv(&#39;monthly_sales.csv&#39;, parse_dates=[&#39;date&#39;], index_col=&#39;date&#39;)<br><br># Step 2: Visualize the Data<br>df[&#39;sales&#39;].plot(figsize=(12, 6))<br>plt.title(&#39;Monthly Sales Over Time&#39;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.ylabel(&#39;Sales&#39;)<br>plt.show()<br><br># Step 3: Check for Stationarity (using ADF test)<br>from statsmodels.tsa.stattools import adfuller<br><br>result = adfuller(df[&#39;sales&#39;])<br>print(f&#39;ADF Statistic: {result[0]}&#39;)<br>print(f&#39;p-value: {result[1]}&#39;)<br><br># If the p-value is less than 0.05, the time series is stationary<br># Otherwise, you might need to difference the data<br><br># Step 4: Fit the SARIMA Model<br># SARIMA(p, d, q)(P, D, Q, s)<br># where (P, D, Q, s) are the seasonal parameters and s is the length of the seasonal cycle<br># Let&#39;s assume initial parameters: (1, 1, 1)(1, 1, 1, 12)<br>model = SARIMAX(df[&#39;sales&#39;], order=(1, 1, 1), seasonal_order=(1, 1, 1, 12))<br>model_fit = model.fit(disp=False)<br><br># Print model summary<br>print(model_fit.summary())<br><br># Step 5: Make Forecasts<br># Forecast the next 12 months<br>forecast = model_fit.get_forecast(steps=12)<br>forecast_index = pd.date_range(start=df.index[-1], periods=12, freq=&#39;M&#39;)<br>forecast_series = pd.Series(forecast.predicted_mean, index=forecast_index)<br><br># Plot the forecast<br>plt.figure(figsize=(12, 6))<br>plt.plot(df[&#39;sales&#39;], label=&#39;Observed&#39;)<br>plt.plot(forecast_series, label=&#39;Forecast&#39;, color=&#39;red&#39;)<br>plt.title(&#39;Sales Forecast&#39;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.ylabel(&#39;Sales&#39;)<br>plt.legend()<br>plt.show()<br><br># Step 6: Evaluate the Model<br># If you have a test set, you can compare the forecast against the actual values<br># For this example, let&#39;s use the last 12 months as a test set<br>train = df[&#39;sales&#39;][:-12]<br>test = df[&#39;sales&#39;][-12:]<br><br># Fit the model on the training set<br>model = SARIMAX(train, order=(1, 1, 1), seasonal_order=(1, 1, 1, 12))<br>model_fit = model.fit(disp=False)<br><br># Forecast the test set period<br>forecast = model_fit.get_forecast(steps=12)<br>forecast_series = pd.Series(forecast.predicted_mean, index=test.index)<br><br># Plot the results<br>plt.figure(figsize=(12, 6))<br>plt.plot(train, label=&#39;Training&#39;)<br>plt.plot(test, label=&#39;Test&#39;, color=&#39;green&#39;)<br>plt.plot(forecast_series, label=&#39;Forecast&#39;, color=&#39;red&#39;)<br>plt.title(&#39;Sales Forecast vs Actual&#39;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.ylabel(&#39;Sales&#39;)<br>plt.legend()<br>plt.show()<br><br># Calculate RMSE<br>rmse = np.sqrt(mean_squared_error(test, forecast_series))<br>print(f&#39;Root Mean Squared Error: {rmse}&#39;)</pre><h4>2. Predicting a Churn</h4><pre>import pandas as pd<br>import numpy as np<br>from sklearn.model_selection import train_test_split<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import accuracy_score, confusion_matrix, classification_report<br><br># Load Data<br>df = pd.read_csv(&#39;telecom_churn_data.csv&#39;)<br><br># Data Cleaning<br>df.dropna(inplace=True)<br><br># Feature Engineering<br>df[&#39;total_calls&#39;] = df[&#39;day_calls&#39;] + df[&#39;evening_calls&#39;] + df[&#39;night_calls&#39;] + df[&#39;international_calls&#39;]<br>df[&#39;total_charge&#39;] = df[&#39;day_charge&#39;] + df[&#39;evening_charge&#39;] + df[&#39;night_charge&#39;] + df[&#39;international_charge&#39;]<br><br># Encoding categorical variables<br>df = pd.get_dummies(df, columns=[&#39;state&#39;, &#39;area_code&#39;, &#39;international_plan&#39;, &#39;voice_mail_plan&#39;])<br><br># Split Data<br>X = df.drop(&#39;churn&#39;, axis=1)<br>y = df[&#39;churn&#39;]<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)<br><br># Model Training<br>model = RandomForestClassifier(n_estimators=100, random_state=42)<br>model.fit(X_train, y_train)<br><br># Predictions<br>y_pred = model.predict(X_test)<br><br># Evaluation<br>print(f&#39;Accuracy: {accuracy_score(y_test, y_pred)}&#39;)<br>print(confusion_matrix(y_test, y_pred))<br>print(classification_report(y_test, y_pred))</pre><h4>3. Time Series Forecasting</h4><pre>import pandas as pd<br>import numpy as np<br>from statsmodels.tsa.seasonal import seasonal_decompose<br>from statsmodels.tsa.arima.model import ARIMA<br>import matplotlib.pyplot as plt<br><br># Load Data<br>df = pd.read_csv(&#39;monthly_sales.csv&#39;, parse_dates=[&#39;date&#39;], index_col=&#39;date&#39;)<br><br># Decompose Time Series<br>result = seasonal_decompose(df[&#39;sales&#39;], model=&#39;additive&#39;)<br>result.plot()<br>plt.show()<br><br># Train ARIMA Model<br>model = ARIMA(df[&#39;sales&#39;], order=(1, 1, 1))<br>model_fit = model.fit()<br><br># Forecast<br>forecast = model_fit.forecast(steps=12)<br>forecast.plot()<br>plt.show()</pre><h4>4. Market Basket Analysis</h4><pre>import pandas as pd<br>from mlxtend.frequent_patterns import apriori, association_rules<br><br># Load Data<br>df = pd.read_csv(&#39;ecommerce_transactions.csv&#39;)<br><br># Preprocess Data<br>basket = (df.groupby([&#39;transaction_id&#39;, &#39;product_name&#39;])[&#39;quantity&#39;]<br>          .sum().unstack().reset_index().fillna(0)<br>          .set_index(&#39;transaction_id&#39;))<br><br>def encode_units(x):<br>    return 1 if x &gt;= 1 else 0<br><br>basket = basket.applymap(encode_units)<br><br># Apriori Algorithm<br>frequent_itemsets = apriori(basket, min_support=0.01, use_colnames=True)<br>rules = association_rules(frequent_itemsets, metric=&quot;lift&quot;, min_threshold=1)<br><br># Display Rules<br>print(rules[[&#39;antecedents&#39;, &#39;consequents&#39;, &#39;support&#39;, &#39;confidence&#39;, &#39;lift&#39;]])</pre><h4>5. Sentiment Analysis on Social Media</h4><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>from sklearn.feature_extraction.text import CountVectorizer<br>from sklearn.naive_bayes import MultinomialNB<br>from sklearn.metrics import accuracy_score, confusion_matrix, classification_report<br>import re<br>import nltk<br>nltk.download(&#39;stopwords&#39;)<br>from nltk.corpus import stopwords<br><br># Load Data<br>df = pd.read_csv(&#39;social_media_reviews.csv&#39;)<br><br># Clean Data<br>def clean_text(text):<br>    text = re.sub(r&#39;\W&#39;, &#39; &#39;, text)<br>    text = re.sub(r&#39;\s+&#39;, &#39; &#39;, text)<br>    text = text.lower()<br>    text = &#39; &#39;.join([word for word in text.split() if word not in stopwords.words(&#39;english&#39;)])<br>    return text<br><br>df[&#39;cleaned_review&#39;] = df[&#39;review&#39;].apply(clean_text)<br><br># Preprocess Text<br>vectorizer = CountVectorizer(max_features=1000)<br>X = vectorizer.fit_transform(df[&#39;cleaned_review&#39;]).toarray()<br>y = df[&#39;sentiment&#39;]<br><br># Split Data<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)<br><br># Train Model<br>model = MultinomialNB()<br>model.fit(X_train, y_train)<br><br># Predictions<br>y_pred = model.predict(X_test)<br><br># Evaluation<br>print(f&#39;Accuracy: {accuracy_score(y_test, y_pred)}&#39;)<br>print(confusion_matrix(y_test, y_pred))<br>print(classification_report(y_test, y_pred))</pre><h4>6. Anomaly Detection in Network Traffic</h4><pre>import pandas as pd<br>import numpy as np<br>from sklearn.ensemble import IsolationForest<br>import matplotlib.pyplot as plt<br><br># Load Data<br>df = pd.read_csv(&#39;network_traffic.csv&#39;)<br><br># Feature Selection<br>features = [&#39;duration&#39;, &#39;src_bytes&#39;, &#39;dst_bytes&#39;, &#39;count&#39;, &#39;srv_count&#39;]<br><br>X = df[features]<br><br># Train Isolation Forest<br>model = IsolationForest(n_estimators=100, contamination=0.01, random_state=42)<br>model.fit(X)<br><br># Predict Anomalies<br>df[&#39;anomaly&#39;] = model.predict(X)<br><br># Visualize Anomalies<br>anomalies = df[df[&#39;anomaly&#39;] == -1]<br><br>plt.scatter(df[&#39;src_bytes&#39;], df[&#39;dst_bytes&#39;], c=df[&#39;anomaly&#39;], cmap=&#39;coolwarm&#39;)<br>plt.xlabel(&#39;Source Bytes&#39;)<br>plt.ylabel(&#39;Destination Bytes&#39;)<br>plt.title(&#39;Anomaly Detection in Network Traffic&#39;)<br>plt.show()</pre><h4>7. Load a CSV File 📂</h4><pre>df = pd.read_csv(&#39;data.csv&#39;)</pre><h4>8. Mutate Strings</h4><pre># Create a string named str<br>str = &quot;Rahim and Karim&quot;<br><br># Convert a string to uppercase<br>str.upper() # &#39;RAHIM AND KARIM&#39;<br><br># Convert a string to lowercase<br>str.lower() # &#39;rahim and karim&#39;<br><br># Convert a string to title case<br>str.title() # &#39;Rahim And Karim&#39; <br><br># Replaces matches of a substring with another<br>str.replace(&quot;J&quot;, &quot;P&quot;) # &#39;Kahim and Rarim&#39;</pre><h4>9. Handle Missing Values &amp; Remove Duplicates❓</h4><pre>df.dropna(inplace=True)  # Drop missing values<br>df.fillna(df.mean(), inplace=True)  # Fill missing values with mean<br><br>df.drop_duplicates(inplace=True)</pre><h4>10. Rename Columns 📝</h4><pre>df.rename(columns={&#39;old_name&#39;: &#39;new_name&#39;}, inplace=True)</pre><h4>11. Group By and Aggregate 🔄</h4><pre>grouped_df = df.groupby(&#39;column&#39;).agg({&#39;col1&#39;: &#39;sum&#39;, &#39;col2&#39;: &#39;mean&#39;})</pre><h4><strong>12. Merge DataFrames 🔗</strong></h4><pre>merged_df = pd.merge(df1, df2, on=&#39;common_column&#39;, how=&#39;inner&#39;)</pre><h4>13. Pivot Table 🕹️</h4><pre>pivot = df.pivot_table(values=&#39;value&#39;, index=&#39;index&#39;, columns=&#39;columns&#39;, aggfunc=&#39;mean&#39;)</pre><h4>14. Apply Function to Column 🧮</h4><pre>df[&#39;new_column&#39;] = df[&#39;column&#39;].apply(lambda x: x*2)</pre><h4>15. Model Evaluation — MSE and R²</h4><pre>mse = mean_squared_error(y_test, predictions)<br>r2 = r2_score(y_test, predictions)<br>print(f&#39;MSE: {mse}, R²: {r2}&#39;)</pre><h4>16. Save Model 💾</h4><pre>import joblib<br>joblib.dump(model, &#39;model.pkl&#39;)</pre><h4>17. Time Series — Convert to Datetime 📅</h4><pre>df[&#39;date&#39;] = pd.to_datetime(df[&#39;date&#39;])<br>df.set_index(&#39;date&#39;, inplace=True)</pre><h4>18. Rolling Window Calculation 🔄</h4><pre>df[&#39;rolling_mean&#39;] = df[&#39;column&#39;].rolling(window=12).mean()</pre><h4>19. Principal Component Analysis (PCA)</h4><pre>from sklearn.decomposition import PCA<br>pca = PCA(n_components=2)<br>principal_components = pca.fit_transform(df_scaled)</pre><h4>20. K-Means Clustering 🎯</h4><pre>from sklearn.cluster import KMeans<br>kmeans = KMeans(n_clusters=3)<br>kmeans.fit(df_scaled)<br>df[&#39;cluster&#39;] = kmeans.labels_</pre><h4>21. Perform SQL Query on DataFrame 🗃️</h4><pre>import pandasql as psql<br>query = &quot;SELECT * FROM df WHERE column &gt; value&quot;<br>result = psql.sqldf(query, locals())</pre><h4>22. Create a Dashboard with Plotly 📊</h4><pre>import plotly.express as px<br>fig = px.line(df, x=&#39;date&#39;, y=&#39;value&#39;)<br>fig.show()</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7cea27da3aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>