<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rushabh Vora on Medium]]></title>
        <description><![CDATA[Stories by Rushabh Vora on Medium]]></description>
        <link>https://medium.com/@rushhabhh?source=rss-401aabbf88fd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RRcEkpTsYj2oqh0MQJAUKA.jpeg</url>
            <title>Stories by Rushabh Vora on Medium</title>
            <link>https://medium.com/@rushhabhh?source=rss-401aabbf88fd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:10:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rushhabhh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Can AI Detect Cancer? Exploring ML Models on Breast Tumor Data]]></title>
            <link>https://rushhabhh.medium.com/can-ai-detect-cancer-exploring-ml-models-on-breast-tumor-data-936f63edbe55?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/936f63edbe55</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[healthcare-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cancer-detection]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Sat, 19 Apr 2025 20:40:47 GMT</pubDate>
            <atom:updated>2025-04-19T20:43:28.388Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZbGBrZQRhnG48RkL" /><figcaption>Photo by <a href="https://unsplash.com/@ang10ze?utm_source=medium&amp;utm_medium=referral">Angiola Harry</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the era of AI-powered healthcare, early detection of diseases like breast cancer can significantly improve patient outcomes. Breast cancer remains one of the leading causes of death among women globally, and accurate diagnosis is a critical step toward effective treatment.</p><p>In this project, I explored the <strong>Wisconsin Diagnostic Breast Cancer (WDBC)</strong> dataset to build and compare machine learning models that can classify tumors as <strong>malignant or benign</strong> with high precision and recall.</p><p>This article will take you through:</p><ul><li>How I approached the diagnosis prediction problem</li><li>How I visualized patterns in the data</li><li>How I trained and compared six machine learning models</li><li>And finally, how I identified the most effective model for breast cancer classification</li></ul><h3>The Dataset</h3><ul><li><strong>Source</strong>: UCI Machine Learning Repository — <a href="https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)">WDBC Dataset</a></li><li><strong>Size</strong>: 569 records (212 Malignant, 357 Benign)</li><li><strong>Features</strong>:</li><li>30 numerical features related to tumor cell nuclei (e.g., radius, texture, concavity)</li><li>Diagnosis: Target variable (M = Malignant, B = Benign)</li></ul><p><strong>GitHub: </strong><a href="https://github.com/rushhabhh/Breast-Cancer-Detection"><strong><em>Breast Cancer Detection</em></strong></a></p><p>The dataset provides a great foundation for classification tasks, especially because the features are continuous, and the classes while slightly imbalanced are reasonably distributed.</p><h3>Exploratory Data Analysis (EDA)</h3><p>I began with a thorough EDA to understand the nature of the data and how the classes differ visually.</p><ul><li>Count plots to view class distribution</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*OnNN3HHBteXitf7R8Rn_ww.png" /></figure><ul><li>Boxplots and KDE plots for feature-wise comparison between classes</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*DWsICwNEkdGgaxvrKxS71Q.png" /></figure><ul><li>PCA scatter plot to see how separable the two classes are in reduced dimensions</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*kCe5q1X_hnKSulxP_w1bCQ.png" /></figure><ul><li>A pairplot of the <strong>top 5 most informative features</strong> shows distinct clusters between malignant and benign cases, indicating strong feature separability</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lTfKbWwjzW_n4zAgTlav7Q.png" /></figure><p><strong>Insights</strong>:</p><ul><li>Malignant tumors generally had higher values for features like radius_mean and concave_points_mean</li><li>PCA revealed strong separation between classes, suggesting machine learning models should perform well</li></ul><h3>Preprocessing</h3><ul><li>Encoded the target (Diagnosis) using map({‘M’: 1, ‘B’: 0})</li><li>Scaled features using StandardScaler for models like SVM and Naive Bayes</li><li>Split the data into training and testing sets</li></ul><h3>Model Building</h3><p>I trained and evaluated <strong>six classification models</strong>:</p><ol><li><strong>Logistic Regression: </strong>Simple, interpretable, and surprisingly strong as a baseline.</li><li><strong>Gaussian Naive Bayes: </strong>Works best with scaled data due to its Gaussian assumption.</li><li><strong>Support Vector Machine (SVM): </strong>Effective for small-to-medium datasets with clear margins of separation.</li><li><strong>Decision Tree: </strong>Easy to interpret but prone to overfitting.</li><li><strong>Random Forest: </strong>Ensemble of decision trees that improves generalization.</li><li><strong>XGBoost: </strong>State-of-the-art gradient boosting model, especially powerful for structured data</li></ol><h3>Model Comparison</h3><p>All top models achieved <strong>above 95% accuracy</strong>, but AUC and F1-score helped identify the real winners:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53248d2f2a8ce7ca648a44209bf428bb/href">https://medium.com/media/53248d2f2a8ce7ca648a44209bf428bb/href</a></iframe><p>ROC and PR curves further confirmed that Random Forest and XGBoost offered the most consistent performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*Uc8s82OtfKaIRP5Z92w6DA.png" /></figure><h3>Feature Importance</h3><p>Using Random Forest and XGBoost, I identified key features such as:</p><ul><li>radius_mean</li><li>perimeter_worst</li><li>concave_points_mean</li></ul><p>These features are consistent with medical research, reinforcing the interpretability and trust in the models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*s2VVSMhyKcHjbfDiZXqt0Q.png" /></figure><h3>Key Takeaways</h3><ul><li>Early breast cancer diagnosis can be significantly improved with machine learning</li><li>Scaling and preprocessing matter, especially for models like SVM and Naive Bayes</li><li><strong>Random Forest and XGBoost</strong> provided the best performance overall</li><li><strong>Logistic Regression</strong> is a surprisingly strong and interpretable baseline</li><li>Visualizations like PCA and feature importance enhance interpretability</li></ul><h3>Final Thoughts</h3><p>This project gave me valuable experience in:</p><ul><li>Training and comparing multiple ML models</li><li>Evaluating performance beyond accuracy</li><li>Working with real-world medical datasets</li><li>Communicating insights through visualizations and metrics</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=936f63edbe55" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Supervised Learning — Classification (Part 2)]]></title>
            <link>https://rushhabhh.medium.com/understanding-supervised-learning-classification-part-2-7f53ac6a5952?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f53ac6a5952</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 00:54:26 GMT</pubDate>
            <atom:updated>2025-04-11T00:54:26.505Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding Supervised Learning — Classification (Part 2)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uHWwVZ57O3dxq_xG" /><figcaption>Photo by <a href="https://unsplash.com/@kommumikation?utm_source=medium&amp;utm_medium=referral">Mika Baumeister</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Introduction</h4><p>Machine Learning is all about teaching machines to learn from data. One of the most essential techniques in ML is <strong>Supervised Learning</strong>, where the model learns from labeled data. Within supervised learning, <strong>Classification</strong> is the method used when the output variable is <strong>categorical</strong>, i.e., it belongs to distinct classes or categories.</p><h4>But, What is Classification?</h4><p><strong>Classification</strong> is a type of supervised learning where the goal is to predict the <strong>label</strong> (or class) of input data.</p><ul><li>Output: Discrete/Categorical (e.g., 0 or 1, Red or Blue, Class A or B)</li><li>Input: Features or attributes of data</li><li>Goal: Find patterns in input data that help assign it to one of the predefined categories</li></ul><p>Think spam vs. non-spam, cat vs. dog, or customer will churn vs. will not churn — these are all classification problems.</p><h4>Types of Classification Problems</h4><ol><li><strong>Binary Classification</strong>: Two classes (e.g., True/False, Spam/Not Spam)</li><li><strong>Multiclass Classification</strong>: More than two classes (e.g., predicting the species of a flower)</li><li><strong>Multilabel Classification</strong>: Each instance can belong to multiple classes at once (e.g., a movie can be both “Action” and “Comedy”)</li></ol><h4>What are the types of Classification algorithms?</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb7be142ee5177ac5a615d7f97076d81/href">https://medium.com/media/fb7be142ee5177ac5a615d7f97076d81/href</a></iframe><h4>How Does It Work?</h4><ol><li><strong>Training Phase</strong>: Feed the algorithm labeled data so it can learn patterns.</li><li><strong>Prediction Phase</strong>: Input new data and get a predicted label.</li><li><strong>Evaluation</strong>: Measure performance using various metrics.</li></ol><h4>Evaluation Metrics</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a7a1dfceaf139955f304e334919828a/href">https://medium.com/media/7a7a1dfceaf139955f304e334919828a/href</a></iframe><h4>Common Loss Functions in Classification</h4><ul><li><strong>Binary Cross-Entropy</strong>: Used in binary classification problems</li><li><strong>Categorical Cross-Entropy</strong>: Used in multiclass classification with one-hot encoded labels</li><li><strong>Hinge Loss</strong>: Used for Support Vector Machines</li></ul><p>These loss functions guide the model during training to make better predictions.</p><h4>Where are classification algorithms used?</h4><ul><li><strong>Healthcare</strong>: Diagnosing diseases based on symptoms</li><li><strong>Finance</strong>: Fraud detection in transactions</li><li><strong>Marketing</strong>: Predicting customer churn</li><li><strong>Retail</strong>: Product recommendation systems</li><li><strong>Cybersecurity</strong>: Detecting phishing emails or malware</li></ul><h4>Example of Classification in Python</h4><pre>from sklearn.datasets import load_iris<br>from sklearn.model_selection import train_test_split<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import accuracy_score<br><br># Load dataset<br>iris = load_iris()<br>X = iris.data<br>y = iris.target<br><br># Train-test split<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># Model training<br>model = RandomForestClassifier()<br>model.fit(X_train, y_train)<br><br># Prediction and evaluation<br>y_pred = model.predict(X_test)<br>print(&quot;Accuracy:&quot;, accuracy_score(y_test, y_pred))</pre><h4>Conclusion</h4><p>Classification is a cornerstone of supervised learning, enabling machines to categorize data into meaningful groups. From filtering spam to diagnosing diseases, classification is everywhere. Understanding its core principles, algorithms, and evaluation metrics is key to building effective predictive models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f53ac6a5952" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gradient Descent: Heartbeat of Machine Learning?]]></title>
            <link>https://rushhabhh.medium.com/gradient-descent-heartbeat-of-machine-learning-099f9e9d5272?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/099f9e9d5272</guid>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Tue, 08 Apr 2025 03:42:25 GMT</pubDate>
            <atom:updated>2025-04-08T03:42:25.766Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FPRaHj9amUo1Yj1e75pEjA.png" /></figure><h3>Introduction</h3><p>Machine Learning is transforming how we analyze data and make decisions, with optimization being at the core of every model. One of the most fundamental optimization techniques in machine learning is <strong>Gradient Descent</strong>. It’s used everywhere from predicting stock prices to training deep learning models. This article will guide you through the concept of Gradient Descent, why it matters, how it works, and where it’s used.</p><h3>First thing First, What is Gradient Descent?</h3><p>Gradient Descent is an <strong>optimization algorithm</strong> used to minimize a <strong>cost function</strong> by iteratively adjusting model parameters. Think of it as the engine that powers model learning.</p><p>In simple terms, it helps a machine learn by <strong>finding the best values for the parameters</strong> of a model so that the error (or cost) is as small as possible.</p><h4><strong>Analogy: Hiking Down a Mountain</strong></h4><p>Imagine you’re on top of a foggy hill and want to reach the lowest point. You can’t see the whole landscape, so you feel the slope beneath your feet and step downhill. Repeating this process eventually brings you to the bottom. That’s exactly what Gradient Descent does, step by step.</p><h3>Why Do We Use Gradient Descent?</h3><p>In supervised learning, especially regression tasks, we try to find the best-fit line (or curve) through data points. To do this, we define a cost function (like Mean Squared Error) that measures how far off our predictions are. Gradient Descent minimizes this cost by tweaking the model parameters until the predictions are as close as possible to the actual values.</p><h3>How Does Gradient Descent Work?</h3><p>Let’s break it down:</p><ol><li>Start with initial random values for the model parameters (weights and bias).</li><li>Calculate the <strong>gradient</strong> (partial derivatives) of the cost function with respect to each parameter.</li><li>Update the parameters by moving them in the opposite direction of the gradient (because we want to minimize the cost).</li><li>Repeat until convergence (i.e., the cost stops decreasing significantly).</li></ol><h4>Update Rule:</h4><p>For a parameter θ, the update rule is: θ = θ — α * ∇ J (θ)</p><p>Where:</p><ul><li>α = learning rate (controls step size)</li><li>∇ J (θ) = gradient of cost function with respect to θ</li></ul><h3>The Role of the Learning Rate</h3><p>The learning rate α is crucial:</p><ul><li><strong>Too small</strong>: Takes forever to converge</li><li><strong>Too large</strong>: Can overshoot or even diverge</li><li><strong>Just right</strong>: Smooth and fast convergence</li></ul><h3>Visualizing Gradient Descent</h3><p>Imagine a U-shaped curve (like<strong><em> </em></strong><em>y = x²</em>).</p><ul><li>A ball (your parameter) rolls down the curve.</li><li>With each step (iteration), it gets closer to the lowest point (minimum cost).</li></ul><p>Each step size depends on the slope at that point, which is exactly what the gradient tells us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DApThJD7VDYogEo9LXXGXQ.png" /><figcaption>Generated by ChatGPT</figcaption></figure><h3>Types of Gradient Descent</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/288a00603572a5af0b7588a8b118d740/href">https://medium.com/media/288a00603572a5af0b7588a8b118d740/href</a></iframe><h3>Real-World Example</h3><p>Let’s say you’re predicting housing prices based on square footage. You start with random weights for your regression line. Using gradient descent, you iteratively adjust these weights to reduce the prediction error until you find the best-fit line.</p><h3>Python Code: A Simple Gradient Descent from Scratch</h3><pre>def df(x): <br>  return x ** 2<br><br>def df(x): <br>  return 2 * x<br><br>x = 10  # Starting point<br>alpha = 0.1  # Learning rate<br><br>for i in range(20):<br>    x = x - alpha * df(x)<br>    print(f&quot;Step {i+1}: x = {x:.4f}, f(x) = {f(x):.4f}&quot;)</pre><p>This will show x converging to 0, the minimum of the function <em>f(x) = x²</em>.</p><h3>Where is Gradient Descent Used?</h3><ul><li>Linear &amp; Logistic Regression</li><li>Neural Networks</li><li>Natural Language Processing (NLP)</li><li>Computer Vision</li><li>Recommender Systems</li></ul><h3>Wrapping Up</h3><p>Gradient Descent isn’t just an algorithm. It’s a mindset of continuous improvement trying, failing, correcting, and trying again. Whether you’re building a simple linear model or training a massive neural network, understanding gradient descent helps you fine-tune your model with confidence.</p><h3>Acknowledgements</h3><ul><li><em>Data Science from Scratch</em> by Joel Grus</li><li>Codebasics: Master Machine Learning for Data Science &amp; AI</li><li>ChatGPT</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=099f9e9d5272" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built a Fraud Detection System with XGBoost and SMOTE]]></title>
            <link>https://rushhabhh.medium.com/how-i-built-a-fraud-detection-system-with-xgboost-and-smote-13cf2f3f7b18?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/13cf2f3f7b18</guid>
            <category><![CDATA[financial-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Mon, 31 Mar 2025 21:22:44 GMT</pubDate>
            <atom:updated>2025-03-31T21:25:09.726Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IL6WY9RJxE207_YT" /><figcaption>Photo by <a href="https://unsplash.com/@averye457?utm_source=medium&amp;utm_medium=referral">Avery Evans</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the digital era, credit card fraud is more than just a financial risk, it’s a persistent threat to consumer trust and institutional stability. Financial organizations process millions of transactions daily, and even a small fraction of fraud can lead to major losses.</p><p>In this project, I tackled a <strong>real-world credit card fraud dataset</strong> where only <strong>0.17% of the transactions were fraudulent</strong>. My goal? To build an effective machine learning system that can detect fraud <strong>with high precision and recall</strong>, without flagging too many legit transactions.</p><p>This article will take you through:</p><ul><li>How I approached the problem</li><li>How I visualized the patterns</li><li>How I handled extreme class imbalance</li><li>How I trained and compared three machine learning models</li><li>And finally, how I identified the best model for fraud detection</li></ul><h4>The Dataset</h4><ul><li><strong>Source</strong>: <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Kaggle — Credit Card Fraud Detection</a></li><li><strong>GitHub: </strong><a href="https://github.com/rushhabhh/Credit-Card-Fraud-Detection">Credit Card Fraud Detection</a></li><li><strong>Size</strong>: 284,807 transactions, of which only 492 are fraud (0.17%)</li><li><strong>Features</strong>:</li><li>V1 to V28: PCA-transformed features to preserve confidentiality</li><li>Time and Amount: Raw features</li><li>Class: Target variable (0 = Legit, 1 = Fraud)</li></ul><p>Due to the sensitive nature of financial data, the dataset was transformed using <strong>Principal Component Analysis (PCA)</strong>. This means we can’t directly interpret what V1, V2, etc. represent, but patterns can still be learned by models.</p><h4>Exploratory Data Analysis (EDA)</h4><p>I began with a thorough EDA to understand the distribution of fraud vs legit transactions.</p><ul><li>Visualized the <strong>class imbalance</strong> using count plots</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*47W4s8GL275UH5GlyjfIeA.png" /></figure><ul><li>Explored how <strong>transaction amount</strong> and <strong>time</strong> varied across classes</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*faSeTtsQmrCWGDeQOEbpnA.png" /></figure><ul><li>Plotted <strong>histograms</strong> for all V features to examine feature distribution</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aGgCnPHyg1ty2svCXWBZOg.png" /><figcaption>Only first six features are displayed</figcaption></figure><p>Insights:</p><ul><li>Fraudulent transactions had different amount ranges and timing</li><li>Many PCA features showed distinguishable distributions between fraud and legit</li></ul><h4>Handling Class Imbalance with SMOTE</h4><p>With only 0.17% fraud cases, training a model as-is would lead to biased predictions.</p><p>I used <strong>SMOTE (Synthetic Minority Oversampling Technique)</strong> to generate synthetic fraud samples in the training set.</p><p>Why SMOTE?</p><ul><li>Prevents overfitting caused by random oversampling</li><li>Helps models learn minority class patterns better</li></ul><p>Important: SMOTE was only applied to the <strong>training set</strong>, preserving real-world distribution in the test set.</p><pre>Before SMOTE: Class<br>0    199020<br>1       344<br><br>After SMOTE: Class<br>0    199020<br>1    199020</pre><h4>Model Building</h4><p>I trained and evaluated three classification models:</p><p><strong>1. Logistic Regression</strong></p><ul><li>Baseline model with class_weight=&#39;balanced&#39;</li><li>Simple, fast, and interpretable</li></ul><p><strong>2. Random Forest</strong></p><ul><li>Ensemble of decision trees</li><li>Handles non-linear patterns and is robust to outliers</li></ul><p><strong>3. XGBoost</strong></p><ul><li>Gradient boosting algorithm</li><li>Known for performance on structured data and imbalanced problems</li></ul><h4>Evaluation Metrics</h4><p>Because accuracy isn’t reliable in imbalanced datasets, I focused on:</p><ul><li><strong>Precision</strong>: How many predicted frauds were actually fraud?</li><li><strong>Recall</strong>: How many actual frauds were detected?</li><li><strong>F1-Score</strong>: Balance between precision and recall</li><li><strong>ROC-AUC</strong>: Overall discriminatory ability</li></ul><h4>Model Comparison</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2e9e0c5a30bb13fabb13956b4300527/href">https://medium.com/media/b2e9e0c5a30bb13fabb13956b4300527/href</a></iframe><p><strong>Conclusion</strong>:</p><ul><li>Logistic Regression had the highest ROC-AUC but poor precision</li><li>Random Forest and XGBoost performed the best in real-world metrics</li><li>XGBoost slightly edged out due to better recall</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VnIVmliAQkwEpcgABOfPOw.png" /></figure><h4>Key Takeaways</h4><ul><li>Fraud detection is a <strong>heavily imbalanced classification problem</strong></li><li>Techniques like <strong>SMOTE</strong> can help balance data without losing meaning</li><li><strong>XGBoost and Random Forest</strong> are powerful models for this task</li><li>Precision and recall are far more important than accuracy</li></ul><h4>Final Thoughts</h4><p>This project gave me hands-on experience with:</p><ul><li>Handling real-world imbalanced data</li><li>Building interpretable yet powerful models</li><li>Evaluating performance meaningfully in high-stakes domains</li></ul><p>Let me know your thoughts, and feel free to connect or reach out with feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13cf2f3f7b18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Supervised Learning — Regression (Part 1)]]></title>
            <link>https://rushhabhh.medium.com/understanding-supervised-learning-regression-part-1-b584b47fc889?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/b584b47fc889</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 14:33:02 GMT</pubDate>
            <atom:updated>2025-03-13T14:33:02.682Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding Supervised Learning — Regression (Part 1)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CS7kvqLrEHVxazyT" /><figcaption>Photo by <a href="https://unsplash.com/@askkell?utm_source=medium&amp;utm_medium=referral">Andy Kelly</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Machine Learning has revolutionized data science, data analysis, and decision-making, with supervised learning being one of its most powerful techniques. Among other various types of supervised learning, regression plays a crucial role in predicting continous numerical values. This article will take you through the fundamental concepts of regression, its types, real-world application, and practical implementation.</p><h3>Okay, but What is Supervised Learning?</h3><p>Supervised Learning is a Machine Learning approach where an algorithm learns from labeled data. In this paradigm, the model is trained on input-output pairs, meaning it knows the correct answer (output) for wach input during training. Supervised Learning is classified into two types:</p><ol><li><strong>Regression: </strong>Used for predicting continuous values (e.g. House prices, Stock prices)</li><li><strong>Classification: </strong>Used for predicting discrete categories (e.g. Spam or not spam, Disease diagnosis)</li></ol><p>In this article, we will be focusing on <strong>Regression, </strong>a technique used to model relationships between variables and make continuous predictions.</p><h3>Regression: The Art of Making Predictions</h3><p>Regression is a statistical and machine learning technique used to estimate relationships between a dependent variable and(target) and one or more independent variable (features). The goal is to find a function that represents the relationship between input features and the output variable, allowing accurate predictions on new data.</p><h4>Example of Regression:</h4><p>Now, imagine if you were a real estate analyst predicting housing prices in New York based on features such as square footage, location, number of bedrooms, and nearby station. Regression algorithms will help establish relationships between features and house prices, making it easier to estimate the price of a new property.</p><h3>Meet the Regression Family: Different Ways to Predict</h3><p>There are several types of regression techniques, each suited for different types of data and use cases. let’s explore some of the most common ones.</p><h4>1. Linear Regression: The Straight-talker</h4><p>Linear Regression is the simplest form of regression, where the relationship between input variable and the target variable is modeled as a linear equation.</p><p><strong>Formula:</strong></p><p><em>y = (m * x) + b</em> Where:</p><p><strong>y</strong> = predicted output</p><p><strong>m </strong>= slope of the line (coefficient)</p><p><strong>x </strong>= input features</p><p><strong>b </strong>= intercept (bias)</p><p><strong>Use Case: </strong>Predicting house prices based on square footage</p><h4>2. Multiple Linear Regression: Juggling Multiple Factors</h4><p>An extended version of simple linear regression, multiple linear regression considers multiple input variables to predict the target variable.</p><p><strong>Formula:</strong></p><p><em>y = b₀ + b₁x₁ + b₂x₂ + … + bₙxₙ </em>Where:</p><p><strong>b₀</strong><em> = </em>intercept</p><p><strong>bₙ </strong><em>= </em>coefficients of independent variables <strong>xₙ</strong></p><p><strong>Use Case: </strong>Predicting a car’s fuel efficiency based on engine size, weigh, and horsepower</p><h4>3. Polynomial Regression: When Things get Curvy</h4><p>When the relationship between variables is not linear, polynomial regression helps by fitting a higher-degree polynomial equation to the data.</p><p><strong>Formula:</strong></p><p><em>y = b₀ + b₁x + b₂x² + b₃x³ + … + bₙxⁿ</em></p><p><strong>Use Case: </strong>Predicting the growth of a bacterial population over time</p><h4>4. Lasso &amp; Ridge Regression: The Overfitting Fighters</h4><p>These are regularized versions of linear regression that prevent overfitting by adding penalty terms,</p><ul><li><strong>Lasso Regression: </strong>Uses L1 Regularization (Sum of absolute coefficients),</li><li><strong>Ridge Regression:</strong> Uses L2 Regularization (Sum of squared coefficients).</li></ul><p><strong>Use Case:</strong> Stock price predictions where feature selection is important</p><h3>How Good is your Model? Metrics that Matter</h3><p>To assess the performance of regression models, we use metrics such as:</p><ol><li><strong>Mean Squared Error (MSE):</strong> Measures the average squared difference between actual and predicted values.</li><li><strong>Root Mean Squared Error (RMSE)</strong>: Square root of MSE, giving error in actual units.</li><li><strong>Mean Absolute Error (MAE):</strong> Measures the average absolute difference between actual and predicted values.</li><li><strong>R-Squared (R²):</strong> Indicated how well the model explains variance in the data (0 to 1 scale)</li></ol><h3>Where Regression works its Magic</h3><p>Regression models are widely used across industries:</p><ul><li><strong>Finance:</strong> Predicting stock prices</li><li><strong>Healthcare:</strong> Estimating patient recovery time</li><li><strong>Retail:</strong> Forecasting sales based on customer behavior</li><li><strong>Manufacturing:</strong> Predicting machine failure times</li></ul><h3>Let’s Code! Regression in Python</h3><p>Here’s a quick example of how to implement Linear Regression using Python’s <strong>scikit-learn</strong> library.</p><pre>import numpy as np<br>import pandas as pd<br># Importing scikit-learn library<br>from sklearn.model_selection import train_test_split<br>from sklearn.linear_model import LinearRegression<br>from sklearn.metrics import mean_squared_error<br><br># Sample dataset<br>data = {<br>    &#39;Square_Feet&#39;: [800, 1000, 1200, 1500, 1800],<br>    &#39;Price&#39;: [200000, 250000, 300000, 350000, 400000]<br>}<br>df = pd.DataFrame(data)<br><br># Splitting data into training and testing sets<br>X = df[[&#39;Square_Feet&#39;]]<br>y = df[&#39;Price&#39;]<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42)<br><br># Training Linear Regression model<br>model = LinearRegression()<br>model.fit(X_train, y_train)<br><br># Making predictions<br>y_pred = model.predict(X_test)<br><br># Evaluating the model<br>mse = mean_squared_error(y_test, y_pred)<br>print(f&#39;Mean Squared Error: {mse}&#39;)</pre><h3>Wrapping Up</h3><p>Regression isn’t just about numbers and equations, it’s about making informed decisions, spotting trends, and predicting the future! Regression is a fundamental supervised learning technique that enables data-driven decision-making. Whether predicting house prices, stock trends, or healthcare outcomes, regression models provide valuable insights.</p><h4>Acknowledgement</h4><ol><li>Data Science from Scratch by Joel Grus</li><li>Codebasics: Master Machine Learning for Data Science &amp; AI: Beginner to Advanced</li><li>ChatGPT</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b584b47fc889" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing and Predicting Airline Delays: A Comprehensive Data Science Approach]]></title>
            <link>https://rushhabhh.medium.com/analyzing-and-predicting-airline-delays-a-comprehensive-data-science-approach-c08d5de14a10?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/c08d5de14a10</guid>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[airline-delay]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[predictive-modeling]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Mon, 07 Oct 2024 02:27:39 GMT</pubDate>
            <atom:updated>2024-10-07T02:56:40.242Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EUx8ipo1i0TyUo-D" /><figcaption>Photo by <a href="https://unsplash.com/@joshuas?utm_source=medium&amp;utm_medium=referral">Joshua Sukoff</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Introduction</strong></h3><p>Air travel is one of the most time-sensitive forms of transportation, and delays can cause frustration for both airlines and passengers. In this analysis, I used the <a href="https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FGJ&amp;QO_fu146_anzr=b0-gvzr"><strong>Airline On-Time Performance Dataset</strong></a> provided by the U.S. Department of Transportation’s Bureau of Transportation Statistics (BTS) to explore patterns in delays, cancellations, and diversions. Using various machine learning models and visualizations, I aimed to identify trends, uncover the primary causes of delays, and provide predictive insights.</p><h3><strong>Understanding the Datset</strong></h3><p>The dataset spans from <strong>January 2024 to July 2024</strong>, containing over 4 million records and 110 columns detailing flight operations, including departure and arrival times, delays, cancellations, taxi times, and more.</p><p>Key Variables:</p><ul><li>FlightDate: Date of flight</li><li>DepDelay, ArrDelay: Departure and Arrival delays in minutes</li><li>Cancelled, Diverted: Indicators for flight cancellations and diversions</li><li>Distance: Distance between origin and destination airports</li></ul><p>The first step was data preprocessing to clean the dataset, handle missing values, and consolidate the monthly datasets into a comprehensive dataset of <strong>4,095,932 flights</strong>.</p><h3><strong>Data Preprocessing</strong></h3><p>A significant part of the process was ensuring data quality for machine learning models. I performed the following steps:</p><ul><li><strong>Null value handling</strong>: Columns with missing or incorrect values were cleaned. For example, 77% of the flights had missing carrier delay information, likely because these flights were not delayed by airlines.</li><li><strong>Time Conversion</strong>: The flight times were initially in HHMM format. I converted these into a readable <strong>12-hour AM/PM</strong> format using a custom function, enhancing clarity for the analysis.</li></ul><p><a href="https://github.com/rushhabhh/Airline-Delay-Analysis-Prediction">GitHub Link</a> to access the ipynb file</p><h3><strong>Key Visualizations and Analysis</strong></h3><ol><li><strong>Flight Delays Across Airlines</strong></li></ol><p>This bar chart showcases the number of flights operated by each airline from <strong>January to July 2024</strong>. <strong>Southwest Airlines (WN)</strong> operated the highest number of flights, followed by <strong>Delta (DL)</strong> and <strong>American Airlines (AA)</strong>. The visual highlights the operational scale of these airlines, with the largest players contributing significantly to total air traffic. Smaller airlines like <strong>Spirit (NK)</strong> and <strong>Hawaiian (HA)</strong> operate fewer flights in comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KXBrS8XvMQ_bOLHGg-u-UA.png" /></figure><p><strong>2. Determining the Status of the Flight</strong></p><p>In this visualization, the flight statuses are divided into four categories: <strong>on-time, delayed, canceled, and diverted</strong>. A large portion of flights were on time, but a noticeable percentage experienced delays. Cancellations were less frequent, while diversions formed a small segment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BMenjQOy_GYerUYkbcFjjA.png" /></figure><p><strong>3. Average Departure and Arrival Delay Over Time &amp; Cancellation Rate</strong></p><p>The line graph illustrates the average departure and arrival delays from January to July 2024. Notably, <strong>January and July</strong> displayed the highest delays, possibly due to adverse weather conditions and the surge in summer travel, respectively. Cancellation rates also spiked during these months, reflecting the correlation between high delay periods and cancellations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SxuC4tERqMtL1SpARwLYnQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rKHkRW78xMbHE74qM1TiJQ.png" /></figure><p><strong>4. Distribution of Delay Across Different Airlines</strong></p><p>This bar chart breaks down delay percentages across different airlines. While major airlines like <strong>American (AA)</strong> and <strong>Frontier (F9)</strong> report higher numbers of delayed flights (due to their volume of operations), even smaller airlines like <strong>Spirit (NK) </strong>showed moderate delay percentages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SKYIvVImL64q7QX1aZ_JkQ.png" /></figure><p><strong>5. Highest Average Departure and Arrival Delays by Top 20 Airports</strong></p><p>This visualization highlights the top 20 airports with the highest average delays. Airports like <strong>John F. Kennedy (JFK) </strong>has reported highest delay of almost <strong>725+ minutes</strong> in Departure as well as Arrival delays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jpR-u994jpxy3HTpSVORkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Go-jjY_x4FJidDUOQXUJgQ.png" /></figure><p><strong>6. Average Delay by Cause</strong></p><p>In this visualization, we break down delays into different causes: C<strong>arrier, Weather, Security, Late-arriving aircraft</strong>, and <strong>National Aviation System (NAS) delays</strong>. Carrier-related issues and NAS delays were the most frequent causes of delay with almost<strong> 75%</strong>, while security delays were relatively rare.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*l_POrigjBNPQ3qOdqTTGzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*8o8L7VYY2F2Cz8XzUsJJjA.png" /></figure><p><strong>7. Departure and Arrival Delays by Flight Distance</strong></p><p>This visualization explores the relationship between flight distance and delays. <strong>Short and Medium-distance (0–3,000 miles) </strong>were observed to have the highest percentage of delays, while long-haul flights tended to experience fewer delays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wazk-RpXlKkmpKKyZR7pvg.png" /></figure><p><strong>8. Diverted Flights by Airlines and Airports</strong></p><p>This analysis visualizes the diversion rates by airline and airport. <strong>Southwest (WN)</strong> and <strong>American (AA)</strong> had higher diversion rates, particularly at congested airport like <strong>Dallas Fort Worth (DFW)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPyyt3Do3ev_sRj9tllr3A.png" /></figure><p><strong>9. Correlation Heat-Map of Flight Distance</strong></p><p>The heat-map analysis reveals that departure delays (DepDelay) and arrival delays (ArrDelay) are highly correlated with a value of <strong>0.95</strong>. This strong correlation indicates that once a flight is delayed at departure, it is almost certain to experience delays upon arrival. Additionally, departure and arrival delays also have moderate correlation with carrier delay with value of <strong>0.63 and 0.60</strong>. Scatter plot below can be viewed to check the correlation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*1_2QW2nZFnyHMXs3RAfO5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fo7mSbBK9OEGrQz1F7zkww.png" /></figure><p><strong>10. Common Reasons for Flight Cancellations</strong></p><p>This chart breaks down the primary causes of flight cancellations. <strong>Weather (B)</strong> emerges as the top reason for cancellations, followed by <strong>carrier-related (A)</strong> issues. Understanding these cancellation causes is crucial for airlines to implement better contingency plans, such as backup resources or improved communication systems, to handle adverse weather conditions or technical challenges more efficiently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*66U57JWB8JNTXdJXWdHH9g.png" /></figure><h3><strong>Predicting Delays with Machine Learning</strong></h3><p>In the final section of our project, we applied several machine learning models to predict flight delays. These models were chosen to compare their performance on various metrics and identify the best model for accurate delay prediction. The models include Linear Regression, Decision Tree, Random Forest, and XGBoost.</p><blockquote><strong>Linear Regression Model</strong></blockquote><p>Linear regression performed quite well in predicting delays, showing strong correlation between the input features and target variable. The R² score was particularly high, averaging <strong>0.95</strong>, which indicates that about 95% of the variance in the delay data was explained by the model. The average RMSE was <strong>14.20</strong>, indicating a reasonable margin of error for predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*MO8wcLt_fgmZyzHfWdjGTg.png" /></figure><blockquote><strong>Decision Tree Model</strong></blockquote><p>The Decision Tree model, while interpretable, didn’t perform as well as Linear Regression. The average RMSE was <strong>21.24</strong>, and the R² score averaged <strong>0.89</strong>, indicating lower predictive power. The higher error rates suggest that this model might be overfitting or unable to capture complex relationships as well as other models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*OVyG1tD4vKWi_exxY1QtqQ.png" /></figure><blockquote><strong>Random Forest Model</strong></blockquote><p>Random Forest showed similar performance to Linear Regression, with an average R² of <strong>0.94</strong> and RMSE of <strong>14.93</strong>. The model captured more complex patterns compared to the Decision Tree while avoiding overfitting, thanks to ensemble techniques. Random Forest proved to be a robust option for this predictive task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*hmj5TVJio6aaBbNkzp0bfQ.png" /></figure><blockquote><strong>XGBoost Model</strong></blockquote><p>XGBoost, while powerful, didn’t outperform Random Forest in this scenario. With an average R² of <strong>0.92</strong> and RMSE of <strong>18.09</strong>, XGBoost had relatively higher error values compared to Random Forest. This could be due to the complexity of the data or the need for further tuning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*TELN3mCKoj9qK-XM4kRfxQ.png" /></figure><h3><strong>Conclusion</strong></h3><p>The <strong>Airline Delay Analysis and Prediction Project</strong> provided a comprehensive understanding of the factors contributing to flight delays. By analyzing data from <strong>January to July 2024</strong>, we identified key trends and built machine learning models capable of predicting delays with significant accuracy. Insights from these visualizations and models can help airlines and airports optimize their operations, reduce delays, and enhance the overall efficiency of the air transportation system.</p><p>Among the models tested, <strong>Linear Regression</strong> and <strong>Random Forest</strong> emerged as the most accurate predictors for airline delays. Both had high R² scores (~0.95 for Linear Regression and ~0.94 for Random Forest), indicating strong predictive power. <strong>Random Forest</strong>, in particular, stands out due to its ability to generalize well across the data without overfitting, making it the recommended model for deployment in this context.</p><p>In contrast, <strong>Decision Tree</strong> and <strong>XGBoost</strong> showed slightly higher errors and lower R² scores, suggesting they might not capture the complexity of the dataset as well as the other models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c08d5de14a10" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[12 Creative Ways to Visualize Time in Tableau]]></title>
            <link>https://rushhabhh.medium.com/12-creative-ways-to-visualize-time-in-tableau-42536c51e7e6?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/42536c51e7e6</guid>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Fri, 12 Jul 2024 17:16:29 GMT</pubDate>
            <atom:updated>2024-07-12T17:16:29.529Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*42C0WSfMT--OsJJ_" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Visualizing time series data is a fundamental aspect of data analysis, enabling us to identify trends, patterns, and anomalies over time. While line charts are a traditional way to visualize time series data, Tableau offers numerous other options to match your specific requirements. In this guide, you’ll discover 12 distinct methods for visualizing time series data. From bar charts to heat maps and area charts to calendar views, explore a diverse selection of visualizations to suit any analysis.</p><p><strong>Tableau Visualization:</strong> <a href="https://public.tableau.com/app/profile/rushabh.vora/viz/12CreativeWaystoVisualizetimeinTableau/12Ways-Dashboard">12 Ways to Visualize Time in Tableau</a></p><blockquote><strong>1. Line Chart + Moving Average</strong></blockquote><p>The Line Chart with Moving Average is perfect for highlighting overall trends in your data while smoothing out short-term fluctuations. By overlaying a moving average on a line chart, you can better understand the long-term trends.</p><blockquote><strong>2. Stacked Area Chart</strong></blockquote><p>A Stacked Area Chart is ideal for visualizing the cumulative effect of multiple data series. It allows you to see both the total value and the contribution of individual components over time.</p><blockquote><strong>3. Running Total</strong></blockquote><p>Running Total charts are useful for tracking the accumulation of values over time. This type of chart helps in understanding the cumulative effect and identifying trends in cumulative totals.</p><blockquote><strong>4. Lollipop Chart</strong></blockquote><p>Lollipop Charts combine the readability of bar charts with the precision of dot plots. They are useful for emphasizing specific data points while maintaining the context of the entire data series.</p><blockquote><strong>5. Bar in Bar Chart</strong></blockquote><p>Bar in Bar Charts are perfect for comparing two sets of values directly. For instance, you can compare previous year sales to the current year’s sales, providing a clear comparison between two periods.</p><blockquote><strong>6. Dual-Axis Different Measures</strong></blockquote><p>Dual-Axis Charts allow you to plot two different measures on the same chart, providing a comprehensive view of how two variables interact over time. This is particularly useful for comparing related but distinct metrics like profit and sales.</p><blockquote><strong>7. Spark Area Chart</strong></blockquote><p>Spark Area Charts are miniaturized line or area charts, providing a compact visual of trends over time. These are great for dashboards where space is limited, but you still need to convey trend information.</p><blockquote><strong>8. Dot Plot</strong></blockquote><p>Dot Plots are excellent for comparing a large number of data points across multiple categories. They provide a clear visual representation of the distribution of data points over time.</p><blockquote><strong>9. Barcode Chart</strong></blockquote><p>Barcode Charts are similar to dot plots but are even more compact. They are useful for visualizing high-frequency data and spotting dense clusters of data points.</p><blockquote><strong>10. Difference from Previous</strong></blockquote><p>Charts that show the Difference from Previous values are useful for understanding changes over time. This type of visualization helps in identifying periods of growth, decline, or stability.</p><blockquote><strong>11. Calendar Heatmap</strong></blockquote><p>Calendar Heatmaps display data over a calendar view, providing a temporal heatmap of activity. This is particularly useful for visualizing daily or weekly patterns and identifying periods of high or low activity.</p><blockquote><strong>12. Butterfly Chart</strong></blockquote><p>Butterfly Charts, also known as tornado charts, are great for comparing two related variables side by side. They are often used for demographic data but can be applied to any dataset where a side-by-side comparison is useful.</p><h3><strong>Conclusion</strong></h3><p>Incorporating these 12 creative ways to visualize time in Tableau can significantly enhance your data analysis capabilities. By selecting the appropriate chart type, you can better convey insights and make more informed decisions based on your time series data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8v6brC21IAKacsEGgZDlw.png" /><figcaption>Time Series Dashboard</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42536c51e7e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Anomaly Detection in Transaction Data using Machine learning]]></title>
            <link>https://rushhabhh.medium.com/anomaly-detection-in-transaction-data-using-machine-learning-5e4e481383ab?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e4e481383ab</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[isolation-forests]]></category>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[anamoly-detection]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Thu, 11 Jul 2024 21:55:47 GMT</pubDate>
            <atom:updated>2024-07-11T21:57:21.863Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p0D06koIViYX0jid" /><figcaption>Photo by <a href="https://unsplash.com/@homajob?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Introduction</strong></h3><p>In the digital age, financial transactions occur at lightning speed, and detecting anomalies in these transactions is crucial for preventing fraud and ensuring security. This article delves into the world of anomaly detection, showcasing how we can leverage machine learning models like Isolation Forest and Autoencoder to uncover hidden anomalies in transaction data.</p><h3><strong>Understanding Anomaly Detection</strong></h3><p>Anomaly detection is the process of identifying unusual patterns that do not conform to expected behavior. In the context of financial transactions, anomalies could indicate fraudulent activities, errors, or other irregularities that need attention.</p><h3>The Dataset</h3><p><strong>Dataset:</strong> <a href="https://github.com/rushhabhh/Anomaly-Transaction-Detection/blob/main/transaction_anomalies_dataset.csv">Github Link</a></p><p>For this project, we used a dataset containing transaction details, including transaction amounts, account types, and other relevant features. The first step was to load and explore the dataset to understand its structure and contents.</p><pre># Importing Libraries<br>import pandas as pd<br>import plotly.express as px<br>import matplotlib.pyplot as plt<br>import numpy as np<br>import plotly.graph_objects as go<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import classification_report<br>from sklearn.metrics import roc_curve, roc_auc_score<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.ensemble import IsolationForest<br>from tensorflow.keras.models import Model<br>from tensorflow.keras.layers import Input, Dense<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.metrics import roc_auc_score, precision_recall_curve</pre><pre>data = pd.read_csv(&quot;transaction_anomalies_dataset.csv&quot;)<br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PQprqtZnT6rM0mgNWbX2IQ.png" /></figure><p>Looking for null values in dataset</p><pre>data.isnull().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*_c_jemFPwOGfpvjGf0nEHw.png" /></figure><p>Now, let’s look for the column insights</p><pre>data.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*89XQTfY4VxCI4yV7hZs87w.png" /></figure><h3>Data Exploration and Preprocessing</h3><p>Exploratory data analysis (EDA) is vital to understand the distributions and relationships between variables. Visualizations helped in identifying patterns and potential anomalies.</p><pre># Distribution of Transaction (Histogram)<br>dist_transaction = px.histogram(data, x = &#39;Transaction_Amount&#39;, nbins = 20, title = &#39;Distribution of Transaction Amount&#39;)<br>dist_transaction.update_layout(width = 1000, height = 600, xaxis_title = &#39;Transaction Amount&#39;, yaxis_title = &#39;Frequency&#39;)<br>dist_transaction.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h3eTy4Ebf3Nd6O_PWhTkLA.png" /><figcaption>Histogram</figcaption></figure><p>Let’s have a look at the distribution of transactions amount by account type</p><pre># Transaction Amount by Account type (Box plot)<br>transaction_acc_type = px.box(data, x = &#39;Account_Type&#39;, y = &#39;Transaction_Amount&#39;, title = &#39;Transaction Amount by Account Type&#39;)<br>transaction_acc_type.update_layout(width = 1000, height = 600, xaxis_title = &#39;Transaction Amount&#39;, yaxis_title = &#39;Account Type&#39;)<br>transaction_acc_type.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NBJpmp6WhaNlJk4Xdf9Yfw.png" /><figcaption>Box Plot</figcaption></figure><p>Now visualizing Average Transaction Amount vs. Age, and look for a trend line</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UmX2HDsM_WBlA4T_tp0BiA.png" /><figcaption>Scatter Plot</figcaption></figure><p>This code creates a bar chart using Plotly Express to visualize transaction frequencies by the day of the week</p><pre># Count of Transaction by Day of the Week (Bar chart)<br>day_of_week = px.bar(data, x = &#39;Day_of_Week&#39;, title = &#39;Frequency of Transactions by Day of the Week&#39;)<br>day_of_week.update_layout(width = 1000, height = 600, yaxis_title = &#39;Frequency&#39;, xaxis_title = &#39;Day of Week&#39;)<br>day_of_week.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVwBZpC2dTyQyB0-tQAPCA.png" /><figcaption>Bar Chart</figcaption></figure><p>This graphical representation helps identify relationships between variables, crucial for understanding data dynamics in anomaly detection and beyond</p><pre># Heatmap of features<br>correlation_matrix = numeric_data.corr()<br>fig_corr_heatmap = go.Figure(data = go.Heatmap(z = correlation_matrix.values, x = correlation_matrix.columns, y = correlation_matrix.index))<br>fig_corr_heatmap.update_layout(title = &#39;Correlation Heatmap&#39;, height = 600)<br>fig_corr_heatmap.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*oIavEZDFssl4v2L2aTcoyg.png" /><figcaption>Heat Map</figcaption></figure><h3>Statistical Method: Z-Score</h3><p>A simple yet effective method is using the Z-score to detect anomalies. We calculated the mean and standard deviation of transaction amounts to identify outliers.</p><pre># Calculate mean and standard deviation of Transaction Amount<br>mean_amount = data[&#39;Transaction_Amount&#39;].mean()<br>std_amount = data[&#39;Transaction_Amount&#39;].std()<br><br># Define the anomaly threshold (2 standard deviations from the mean)<br>anomaly_threshold = mean_amount + 2 * std_amount<br><br># Flag anomalies<br>data[&#39;Is_Anomaly&#39;] = data[&#39;Transaction_Amount&#39;] &gt; anomaly_threshold<br><br># Scatter plot of Transaction Amount with anomalies highlighted<br>anomalies = px.scatter(data, x = &#39;Transaction_Amount&#39;, y = &#39;Average_Transaction_Amount&#39;,color = &#39;Is_Anomaly&#39;, <br>                           title = &#39;Anomalies in Transaction Amount&#39;)<br>anomalies.update_traces(marker = dict(size = 12), selector = dict(mode = &#39;markers&#39;, marker_size = 1))<br>anomalies.update_layout(height = 600, xaxis_title = &#39;Transaction Amount&#39;, yaxis_title = &#39;Average Transaction Amount&#39;)<br>anomalies.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dO46q423esY0K8_K6my4wg.png" /><figcaption>Scatter Plot</figcaption></figure><h3>Machine Learning Models for Anomaly Detection</h3><blockquote>Isolation Forest</blockquote><p>Isolation Forest is an unsupervised learning algorithm designed for anomaly detection. It works by isolating observations in a tree structure, where anomalies require fewer splits to isolate.</p><pre># Calculate the number of anomalies<br>num_anomalies = data[&#39;Is_Anomaly&#39;].sum()<br><br># Calculate the total number of instances in the dataset<br>total_instances = data.shape[0]<br><br># Calculate the ratio of anomalies<br>anomaly_ratio = num_anomalies / total_instances<br>print(anomaly_ratio)</pre><p>Traning model for Isolation Forest</p><pre># Isolation forest<br>relevant_features = [&#39;Transaction_Amount&#39;, &#39;Average_Transaction_Amount&#39;, &#39;Frequency_of_Transactions&#39;]<br><br># Split data into features (X) and target variable (y)<br>X = data[relevant_features]<br>y = data[&#39;Is_Anomaly&#39;]<br><br># Split data into train and test sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42)<br><br># Train the Isolation Forest model<br>model = IsolationForest(contamination = 0.02, random_state = 42)<br>model.fit(X_train)</pre><pre># Predict anomalies on the test set<br>y_pred = model.predict(X_test)<br><br># Convert predictions to binary values (0: normal, 1: anomaly)<br>y_pred_binary = [1 if pred == -1 else 0 for pred in y_pred]<br><br># Evaluate the model&#39;s performance<br>report = classification_report(y_test, y_pred_binary, target_names=[&#39;Normal&#39;, &#39;Anomaly&#39;])<br>print(report)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*V-12VsyiQmZQC6Zx9V8Wcw.png" /></figure><p>Finally, here’s how we can use our trained model to detect anomalies:</p><pre># Relevant features used during training<br>relevant_features = [&#39;Transaction_Amount&#39;, &#39;Average_Transaction_Amount&#39;, &#39;Frequency_of_Transactions&#39;]<br><br># Get user inputs for features<br>user_inputs = []<br>for feature in relevant_features:<br>    user_input = float(input(f&quot;Enter the value for &#39;{feature}&#39;: &quot;))<br>    user_inputs.append(user_input)<br><br># Create a DataFrame from user inputs<br>user_df = pd.DataFrame([user_inputs], columns=relevant_features)<br><br># Predict anomalies using the model<br>user_anomaly_pred = model.predict(user_df)<br><br># Convert the prediction to binary value (0: normal, 1: anomaly)<br>user_anomaly_pred_binary = 1 if user_anomaly_pred == -1 else 0<br><br>if user_anomaly_pred_binary == 1:<br>    print(&quot;Anomaly detected: This transaction is flagged as an anomaly.&quot;)<br>else:<br>    print(&quot;No anomaly detected: This transaction is normal.&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*Mrr8HiT7bf8r48omCzhEcg.png" /><figcaption>Trained Model Output</figcaption></figure><blockquote>Random Forest Classifier</blockquote><p>Another machine learning models we explored was the Random Forest Classifier. This model is particularly well-suited for classification tasks and offers valuable insights through its feature importance metrics.</p><pre># Assuming &#39;X&#39; is your feature matrix and &#39;y&#39; is your target<br>model = RandomForestClassifier()<br>model.fit(X_train, y_train)<br>feature_importance = pd.DataFrame({&#39;Feature&#39;: X.columns,<br> &#39;Importance&#39;: model.feature_importances_}).sort_values(by = &#39;Importance&#39;, ascending=False)<br>fig = px.bar(feature_importance, x = &#39;Importance&#39;, y = &#39;Feature&#39;, title = &#39;Feature Importance&#39;)<br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Quam1bfDGvNIGfmmrR9GzA.png" /><figcaption>Random Forest Output</figcaption></figure><blockquote>Autoencoder</blockquote><p>An Autoencoder is a type of neural network that learns a compressed representation of the data. It reconstructs the input data, and significant reconstruction errors indicate anomalies.</p><pre>numeric_data = data.select_dtypes(include=[&#39;number&#39;])<br><br>scaler = StandardScaler()<br>numeric_data_scaled = scaler.fit_transform(numeric_data)<br><br># Split data into training and testing sets<br>from sklearn.model_selection import train_test_split<br>X_train, X_test = train_test_split(numeric_data_scaled, test_size=0.2, random_state=42)<br><br># Build the autoencoder model<br>input_dim = X_train.shape[1]<br>input_layer = Input(shape=(input_dim,))<br>encoder = Dense(14, activation = &quot;relu&quot;)(input_layer)<br>encoder = Dense(7, activation = &quot;relu&quot;)(encoder)<br>decoder = Dense(14, activation = &quot;relu&quot;)(encoder)<br>decoder = Dense(input_dim, activation = &quot;sigmoid&quot;)(decoder)<br>autoencoder = Model(inputs=input_layer, outputs=decoder)<br><br>autoencoder.compile(optimizer = &#39;adam&#39;, loss = &#39;mean_squared_error&#39;)<br><br># Train the autoencoder<br>history = autoencoder.fit(X_train, X_train, epochs = 50, batch_size = 32, validation_data = (X_test, X_test), verbose = 1)</pre><p>Anomaly Detection</p><pre>plt.plot(history.history[&#39;loss&#39;], label=&#39;Training Loss&#39;)<br>plt.plot(history.history[&#39;val_loss&#39;], label=&#39;Validation Loss&#39;)<br>plt.legend()<br>plt.show()<br><br># Predict the reconstruction on the test data<br>X_test_pred = autoencoder.predict(X_test)<br>mse = np.mean(np.power(X_test - X_test_pred, 2), axis = 1)<br><br># Calculate the reconstruction error threshold<br>threshold = np.percentile(mse, 95)<br><br># Identify anomalies<br>anomalies = mse &gt; threshold<br><br># Plot the reconstruction error<br>plt.hist(mse, bins = 50)<br>plt.axvline(threshold, color = &#39;r&#39;, linestyle = &#39;--&#39;)<br>plt.xlabel(&#39;Reconstruction Error&#39;)<br>plt.ylabel(&#39;Number of Samples&#39;)<br>plt.show()<br><br># Print number of anomalies detected<br>print(f&#39;Number of anomalies detected: {np.sum(anomalies)}&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*CWyHy6V5UnerG4UcpuL_lg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*Ck9GfJmSA3nPcrtiqzgPTA.png" /></figure><pre>Number of anomalies detected: 10</pre><h3><strong>Model Evaluation</strong></h3><p>Evaluating the performance of anomaly detection models involves metrics like precision, recall, F1-score, and ROC-AUC. These metrics help in understanding how well the model distinguishes between normal and anomalous transactions.</p><pre># Isolation Forest evaluation<br>roc_score_if = roc_auc_score(y_test, y_pred_binary)<br>precision_if, recall_if, _ = precision_recall_curve(y_test, y_pred_binary)<br><br>print(f&quot;Isolation Forest ROC-AUC: {roc_score_if}&quot;)<br><br># Autoencoder evaluation (requires reconstruction error calculation)<br>reconstruction_error = autoencoder.predict(X_test)<br>mse = np.mean(np.power(X_test - reconstruction_error, 2), axis=1)<br>threshold = np.percentile(mse, 95)<br>y_pred_autoencoder = mse &gt; threshold<br><br>roc_score_ae = roc_auc_score(y_test, y_pred_autoencoder)<br>precision_ae, recall_ae, _ = precision_recall_curve(y_test, y_pred_autoencoder)<br><br>print(f&quot;Autoencoder ROC-AUC: {roc_score_ae}&quot;)</pre><pre># Output (Evaluation of ML Models)<br>Isolation Forest ROC-AUC: 1.0<br>7/7 ━━━━━━━━━━━━━━━━━━━━ 0s 461us/step<br>Autoencoder ROC-AUC: 0.9846938775510203</pre><h3>Conclusion</h3><p>In this project, we explored different methods for anomaly detection in transaction data. Both the Isolation Forest and Autoencoder models demonstrated their strengths in identifying anomalies. The Isolation Forest is particularly effective for its simplicity and interpretability, while the Autoencoder shines in capturing complex patterns and relationships in the data.</p><p>Choosing the right model depends on the specific requirements and constraints of your application. For a balance between simplicity and performance, the Isolation Forest is a great choice. If your data has complex patterns, consider using an Autoencoder.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e4e481383ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unveiling the Crime Landscape of Los Angeles: A Data-Driven Exploration]]></title>
            <link>https://rushhabhh.medium.com/unveiling-the-crime-landscape-of-los-angeles-a-data-driven-exploration-199548a1c980?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/199548a1c980</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-exploration]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Mon, 22 Apr 2024 05:49:07 GMT</pubDate>
            <atom:updated>2024-04-22T05:49:07.187Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U0SKHWP55wRTca3P" /><figcaption>Photo by <a href="https://unsplash.com/@james300402?utm_source=medium&amp;utm_medium=referral">James Hartono</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Introduction</strong></p><p>Los Angeles, the City of Angels, is a vibrant metropolis where dreams coexist with harsh realities. Amidst the glitz and glamour, there lies a complex web of criminal activities that demands our attention and understanding. This article takes you on a captivating journey through a comprehensive data analysis project that unravels the complex patterns and trends of crime in Los Angeles. By harnessing the power of data, we aim to shed light on this critical issue, empowering stakeholders with insights that could shape policies and drive meaningful change.</p><p><strong>The Data Foundation</strong></p><p>Our exploration began with a treasure trove of crime-related data provided by the Los Angeles Police Department (LAPD). This raw dataset spanned multiple years and contained a wealth of information, from crime types and locations to victim demographics and weapon usage. However, before we could delve into the analysis, we had to meticulously clean and preprocess the data, ensuring its integrity and readiness for deeper examination.</p><blockquote>Link to Dataset: <a href="https://catalog.data.gov/dataset/crime-data-from-2020-to-present">https://catalog.data.gov/dataset/crime-data-from-2020-to-present</a></blockquote><p><strong>Let’s explore LAPD Crime Dataset</strong></p><p>Gender wise distribution of the victims, clearly from the chart it can be said that there is not much difference between the male and female victims , below is the pie chart which demonstrates the victims by percent</p><pre>crime_by_sex = crime_la[&#39;victim_sex&#39;].value_counts()<br><br>plt.figure(figsize=(10, 6))<br>labels = {&#39;M&#39;: &#39;Male&#39;, &#39;F&#39;: &#39;Female&#39;, &#39;X&#39;: &#39;Unknown&#39;}<br>plt.pie(crime_by_sex, labels = [labels[x] for x in crime_by_sex.index], autopct = &#39;%1.1f%%&#39;, startangle = 90)<br>plt.title(&#39;Proportion of Crimes by Victim Sex&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*10XgRBV7Ac3hEHiVCb7Ajg.png" /></figure><p>Below bar chart represents the area wise crime numbers where it can be identified that the top 5 areas in Los Angeles where crime rates are at peak are Central, Southwest, 77th Street, Pacific and Hollywood. In Central area almost 50,000 crime cases were reported to LAPD from 2020 to till date.</p><pre>crime_by_area = crime_la[&#39;area_name&#39;].value_counts()<br><br>plt.figure(figsize = (10, 6))<br>crime_by_area.plot(kind = &#39;bar&#39;)<br>plt.title(&#39;Number of Crimes by Area&#39;)<br>plt.xlabel(&#39;Area Name&#39;)<br>plt.ylabel(&#39;Number of Crimes&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fa5wDz8vTOKENwZ7tsFlxw.png" /></figure><p>Another main factor which brings to out attention is the age, age also plays important factor in the victims of crime, most number of crimes where occurred to the youngsters of age ranging from 25–35 where the reported crime numbers are roughly around 250K for that particular age group.</p><pre>plt.figure(figsize = (10, 6))<br><br>plt.hist(crime_la[&#39;victim_age&#39;], bins = 20, edgecolor = &#39;black&#39;)<br>plt.title(&#39;Distribution of Crime Victims by Age&#39;)<br>plt.xlabel(&#39;Age&#39;)<br>plt.ylabel(&#39;Number of Victims&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DG8WkkWFg2u1DC9-u-pVLQ.png" /></figure><p>Not only age but there might be the descent factor where one specific community has been targeted, below bar graph shows how race can be the prime factor in the victims life, it is unraveled that the most number of victims where Hispanic/Latinos in the Los Angeles with over 270K+ cases, these number are eye opening where one particular community has to face such issues.</p><pre>crime_by_descent = crime_la[&#39;victim_descent&#39;].value_counts().sort_values(ascending = True)<br><br>descent_meanings = {<br>    &#39;A&#39;: &#39;Other Asian&#39;, &#39;B&#39;: &#39;Black&#39;, &#39;C&#39;: &#39;Chinese&#39;, &#39;D&#39;: &#39;Cambodian&#39;,<br>    &#39;F&#39;: &#39;Filipino&#39;, &#39;G&#39;: &#39;Guamanian&#39;, &#39;H&#39;: &#39;Hispanic/Latin/Mexican&#39;,<br>    &#39;I&#39;: &#39;American Indian/Alaskan Native&#39;, &#39;J&#39;: &#39;Japanese&#39;, &#39;K&#39;: &#39;Korean&#39;,<br>    &#39;L&#39;: &#39;Laotian&#39;, &#39;O&#39;: &#39;Other&#39;, &#39;P&#39;: &#39;Pacific Islander&#39;, &#39;S&#39;: &#39;Samoan&#39;,<br>    &#39;U&#39;: &#39;Hawaiian&#39;, &#39;V&#39;: &#39;Vietnamese&#39;, &#39;W&#39;: &#39;White&#39;, &#39;X&#39;: &#39;Unknown&#39;, &#39;Z&#39;: &#39;Asian Indian&#39;<br>}<br><br>labels_with_descriptions = [f&quot;{idx} - {descent_meanings.get(idx, &#39;N/A&#39;)}&quot; for idx in crime_by_descent.index]<br><br>plt.figure(figsize=(10, 6))<br>bars = plt.barh(crime_by_descent.index, crime_by_descent.values)<br>plt.title(&#39;Number of Crimes by Victim Descent&#39;)<br>plt.xlabel(&#39;Number of Crimes&#39;)<br>plt.ylabel(&#39;Victim Descent&#39;)<br>plt.yticks(ticks=crime_by_descent.index, labels = labels_with_descriptions)<br><br>for bar in bars:<br>    plt.text(bar.get_width(), bar.get_y() + bar.get_height()/2, f&#39; {int(bar.get_width())}&#39;, va=&#39;center&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GRQRpJvbxNT27Z9-nfNXwg.png" /></figure><p>There were over 695K+ crime cases reported to LAPD since 2020 to till date and also there has been significant increase of 10.93% in the crime rate from 2020 to 2023. With that being said the most frequent reported crimes were Battery — Simple Assault, Theft of Identity, and Burglary — From Vehicles.</p><pre>crime_la[&#39;year&#39;] = crime_la[&#39;date_of_occurence&#39;].dt.year<br><br>total_crimes = crime_la.shape[0]<br><br>crimes_per_year = crime_la[&#39;year&#39;].value_counts().sort_index()<br><br>print(&quot;Total number of crimes:&quot;, total_crimes)<br>print(&quot;Crimes per year:\n&quot;, crimes_per_year)<br><br>crime_la[&#39;year&#39;] = crime_la[&#39;date_of_occurence&#39;].dt.year<br>crimes_per_year = crime_la[&#39;year&#39;].value_counts().sort_index()<br><br>if 2020 in crimes_per_year.index and 2023 in crimes_per_year.index:<br>    crimes_2020 = crimes_per_year[2020]<br>    crimes_2023 = crimes_per_year[2023]<br>    percent_increase = ((crimes_2023 - crimes_2020) / crimes_2020) * 100<br>    print(f&quot;Percent increase from 2020 to 2023: {percent_increase:.2f}%&quot;)<br>    # Percent increase from 2020 to 2023: 10.93%<br>else:<br>    print(&quot;Data for 2020 or 2023 is not available to calculate the increase.&quot;)<br><br>plt.figure(figsize=(10, 6))<br>crimes_per_year.plot(kind = &#39;bar&#39;)<br>plt.title(&#39;Number of Crimes Per Year&#39;)<br>plt.xlabel(&#39;Year&#39;)<br>plt.ylabel(&#39;Number of Crimes&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Hyo6vUAT0-_ZuLNr33_5A.png" /></figure><pre>crime_frequencies = crime_la[&#39;crime_commited_description&#39;].value_counts()<br><br>top_crimes = crime_frequencies.head(10)<br><br>plt.figure(figsize=(10, 6))<br>top_crimes.plot(kind = &#39;bar&#39;)<br>plt.title(&#39;Top 10 Most Frequent Crimes&#39;)<br>plt.xlabel(&#39;Crime Type&#39;)<br>plt.ylabel(&#39;Frequency&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y2ja2cKcshsR0gRq0Ehddw.png" /></figure><p>After considering all the factors we found some interesting fact about the weapon used for crime, the most dangerous among all could be the Strong-Arm (Hands, Fist, Feet or Bodily Force), citizens should be aware of the weapons which can be used against them and they are shown below in the bar chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KzdsEgm9EwnHfcDKErKvrw.png" /></figure><p>At last it is the citizens who are the ultimate victims of these crimes and this type of act has to be reported at LAPD as soon as possible so that law enforcement and police department could act on the situation and make sure that no other citizen has to face such type of crimes again, reporting crimes will help the police department to identify the criminals and ensure that they are punished as per the law and order. This is every citizens responsibility to report the crime but still by the data it can be said that some of the crimes were never reported and most of the other crimes where reported after avg. of 12 days in Los Angeles.</p><pre>crime_la[&#39;delay_reporting&#39;] = (crime_la[&#39;date_reported&#39;] - crime_la[&#39;date_of_occurence&#39;]).dt.days<br><br>avg_reporting_delay = crime_la[&#39;delay_reporting&#39;].mean()<br>print(&#39;Average repoting delay: &#39;,avg_reporting_delay)<br><br># Average reporting delay:  12.775649932395249</pre><p><strong>Bringing Data to Life</strong></p><p>After performing exploratory data analysis in python, to enhance the accessibility and comprehension of our findings, we crafted a visually captivating and interactive dashboard using Tableau. This dashboard serves as a powerful tool for law enforcement agencies, policymakers, and the general public, allowing them to explore the data dynamically and gain a deeper understanding of the crime landscape in Los Angeles. With just a few clicks, users can delve into the complex details and uncover insights that resonate with their specific areas of interest or concern.</p><blockquote>Github Link: <a href="https://github.com/rushhabhh/Los-Angeles-Crime-Analysis">https://github.com/rushhabhh/Los-Angeles-Crime-Analysis</a></blockquote><blockquote>Link to Dashboard: <a href="https://public.tableau.com/app/profile/rushabh.vora/viz/LosAngelesCrimeRateAnalysis/CrimeRateAnalysisDashboard?publish=yes">https://public.tableau.com/app/profile/rushabh.vora/viz/LosAngelesCrimeRateAnalysis/CrimeRateAnalysisDashboard?publish=yes</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KnSPrVha1tI99VP6JCry8w.png" /></figure><p><strong>The Path Forward</strong></p><p>This data-driven exploration of crime in Los Angeles represents a significant stride towards informed decision-making and effective crime prevention strategies. By leveraging the power of data analysis and visualization, we have shed light on critical patterns and trends, empowering stakeholders with actionable insights. However, our journey does not end here. As Los Angeles continues to evolve, ongoing data collection and analysis will be crucial in addressing emerging challenges and fostering a safer environment for all residents.</p><p><strong>Conclusion</strong></p><p>In a city as dynamic and diverse as Los Angeles, understanding the complex web of criminal activities is a pressing challenge. Through this project, we have demonstrated the transformative power of data in unmasking the crime landscape, revealing insights that could shape policies and drive meaningful change. By fostering collaboration between law enforcement, policymakers, and the community, we can leverage these findings to develop targeted interventions, enhance public safety measures, and ultimately create a safer and more secure Los Angeles for all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=199548a1c980" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deciphering the Housing Market Through Data-Driven Insights]]></title>
            <link>https://rushhabhh.medium.com/deciphering-the-housing-market-through-data-driven-insights-95477b0f439e?source=rss-401aabbf88fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/95477b0f439e</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[housing-market]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Rushabh Vora]]></dc:creator>
            <pubDate>Tue, 05 Mar 2024 04:33:08 GMT</pubDate>
            <atom:updated>2024-03-05T04:33:52.352Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*neEjsxNPHfpWdFpR" /><figcaption>Photo by <a href="https://unsplash.com/@joshsorenson?utm_source=medium&amp;utm_medium=referral">Josh Sorenson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The housing market, a reflection of both economic vitality and consumer confidence, is a sector of critical importance. Its ebbs and flows can signal shifts in the broader economy, making the analysis of this market both necessary and challenging. This is where the power of data analytics and visualization becomes indispensable. By converting complex datasets into understandable narratives, we can extract insights that drive strategic decision-making.</p><p>My recent project involved an in-depth exploration of Zillow’s housing dataset, a collection of information that encapsulates housing price dynamics across various regions. The journey from raw data to polished insights entailed transforming a wide dataset into a long format, performing rigorous SQL analysis, and ultimately visualizing the findings through an intuitive Tableau dashboard and Python. This article records that journey, revealing the story of Wisconsin’s housing market over nearly a decade.</p><p><strong>Data Transformation: Laying the Analytical Foundation</strong></p><p>The starting point for any data analysis project is ensuring the data is in a usable format. Zillow’s dataset initially presented a wide array of information that was intricate and extensive. To make this dataset conducive for analysis, it was crucial to convert it into a long format, which was achieved through meticulous data manipulation techniques in Python using pandas. This transformation was not merely a procedural step but a foundational one, ensuring the subsequent analyses could be conducted with precision and clarity.</p><pre>import pandas as pd<br><br># Load the dataset<br>file_path = &#39;/Users/rush/Documents/Data/Wisconsin-House-Price/Original_Dataset_wide.csv&#39;<br>data_wide = pd.read_csv(file_path)<br><br># Convert the wide format data to long format<br>data_long = pd.melt(data_wide, id_vars=[&#39;RegionID&#39;, &#39;SizeRank&#39;, &#39;RegionName&#39;, &#39;RegionType&#39;, &#39;StateName&#39;, &#39;State&#39;, &#39;City&#39;, &#39;Metro&#39;, &#39;CountyName&#39;],<br>                    var_name=&#39;Date&#39;, value_name=&#39;Price&#39;)<br><br># Convert the Date column to a datetime format to ensure consistency<br>data_long[&#39;Date&#39;] = pd.to_datetime(data_long[&#39;Date&#39;], format=&#39;%m/%d/%y&#39;)<br>data_long[&#39;Year&#39;] = data_long[&#39;Date&#39;].dt.year<br><br># Saving output file<br>output_file_path = &#39;/Users/rush/Documents/Data/Wisconsin-House-Price/Converted_Dataset_long.csv&#39;<br>data_long.to_csv(output_file_path, index=False)<br><br>print(&quot;Dataset has been converted and saved as Converted_Dataset_long.csv&quot;)<br><br># Printing the head of the dataset<br>print(data_long.head())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X2yUErcixcbz-UrCoWREdg.png" /></figure><h4><strong>Excel Analysis: Data Cleaning and Exploration</strong></h4><p>Transferring the downloaded data into excel to find the average price of the houses by years, using Averageifs function to get the insights on how the prices increased over the years in the state of Wisconsin</p><pre>=AVERAGEIFS(K2:K8149, L2:L8149, 2014)<br>=AVERAGEIFS(K8150:K16297, L8150:L16297, 2015)<br>=AVERAGEIFS(K16298:K24445, L16298:L24445, 2016)<br>=AVERAGEIFS(K24446:K32593, L24446:L32593, 2017)<br>=AVERAGEIFS(K32594:K40741, L32594:L40741, 2018)<br>=AVERAGEIFS(K40742:K48889, L40742:L48889, 2019)<br>=AVERAGEIFS(K48890:K57037, L48890:L57037, 2020)<br>=AVERAGEIFS(K57038:K65185, L57038:L65185, 2021)<br>=AVERAGEIFS(K65186:K73333, L65186:L73333, 2022)<br>=AVERAGEIFS(K73334:K81481, L73334:L81481, 2023)<br>=AVERAGEIFS(K81482:K82160, L81482:L82160, 2024)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*AyCLPf25s9IWAqVBhvw1jQ.png" /></figure><p><strong>SQL Analysis: The Quest for Deeper Understanding</strong></p><p>With the data transformed, the next step was to delve into SQL analysis, a process akin to an archaeological dig — careful, methodical, and aimed at uncovering hidden treasures. The SQL queries ranged from basic commands to extract the total number of entries to more complex operations that isolated specific insights such as average prices and price trends over time. Each query peeled back a layer of the data, revealing the underlying patterns and shifts within the Wisconsin housing market.</p><p>For instance, the SQL query to retrieve the average housing price from 2014 to 2023 was more than just a numerical result; it was a reflection of the market’s trajectory, influenced by economic policies, demographic changes, and consumer behavior. Similarly, identifying the top and bottom counties by average price wasn’t simply about rankings; it was about understanding the micro-economies within the state and the factors contributing to their market positions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Au4p5AsRkbXIYoT9jiTLSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YQKgwHrXtka-v0O9SsK_HQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qhvbRfOhpARQmTCAdZA_-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8EGWK_Xml4dsyX5Ru6vzA.png" /></figure><p><strong>Visualization with Tableau: Crafting a Narrative Through Data</strong></p><p>Data visualization is the art of storytelling with a purpose. Through the use of Tableau, the previously opaque data was transformed into a clear, engaging narrative. The dashboard created is not just a collection of charts and graphs; it is an interactive platform that allows users to witness the growth trends, compare counties, and observe the striking increase in housing prices over the last decade. By utilizing Tableau’s capabilities, the dashboard serves as a visual representation of the market’s history, current state, and potential future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CnxEAFECBbU4ZNAADPQ1FQ.png" /></figure><p><strong>Insights and Observations: Unearthing Market Dynamics</strong></p><p>The data analysis and visualization revealed several key insights into Wisconsin’s housing market. There was a clear upward trend in housing prices across the state, with some counties experiencing growth that outpaced others. This divergence in county-level performance suggests varying degrees of economic development, investment attractiveness, and policy impact. The percent change in average housing prices, an impressive 76.77% over a decade, spoke volumes about the market’s buoyancy and the underlying demand driving it.</p><p><strong>Conclusion: The Strategic Edge of Data-Driven Decisions</strong></p><p>The journey through the Zillow housing dataset has been illuminating. It underscores the importance of data analysis and visualization in interpreting complex market data and transforming it into actionable intelligence. The insights gained provide a strategic edge to various market participants, allowing for more informed and confident decision-making.</p><p><strong>Call to Action: A Dialogue for Growth</strong></p><p>I encourage readers to engage with the content, explore the Tableau dashboard, and contribute to the ongoing dialogue. Whether you are a seasoned real estate professional, a potential homebuyer, or a data science aficionado, your engagement will enrich the conversation. Together, we can demystify the housing market and harness the collective wisdom to foresee and capitalize on emerging trends.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95477b0f439e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>