<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dipti on Medium]]></title>
        <description><![CDATA[Stories by Dipti on Medium]]></description>
        <link>https://medium.com/@diptim_99684?source=rss-5aebb0dd2f7f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*c-7aDgfYgn21cxAo</url>
            <title>Stories by Dipti on Medium</title>
            <link>https://medium.com/@diptim_99684?source=rss-5aebb0dd2f7f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:04:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@diptim_99684/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI Governance and Data Quality Solutions]]></title>
            <link>https://medium.com/@diptim_99684/ai-governance-and-data-quality-solutions-05b0ca68546d?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/05b0ca68546d</guid>
            <category><![CDATA[generative-ai-consulting]]></category>
            <category><![CDATA[ai-consulting]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Wed, 25 Feb 2026 07:42:35 GMT</pubDate>
            <atom:updated>2026-02-25T07:42:35.281Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>AI governance and data quality have become non-negotiable for enterprises deploying analytics, BI, and GenAI at scale.</strong></p><p>As AI models increasingly influence financial forecasts, operational decisions, and customer interactions, organizations need more than policies — they need enforceable controls, reliable data, and auditable workflows. Perceptive Analytics provides integrated <a href="https://www.perceptive-analytics.com/ai-consulting/"><strong>AI consulting and AI governance services</strong></a> designed to make enterprise analytics trustworthy, compliant, and scalable.</p><h3>1. Overview of Perceptive Analytics AI Governance and Data Quality Services</h3><p>Perceptive Analytics delivers AI governance and data quality as a <strong>unified operating layer</strong> across analytics, BI, and AI initiatives — rather than treating them as standalone compliance exercises.</p><p><strong>Our services typically span:</strong></p><ul><li><strong>Assessment:</strong> Current-state review of data quality, AI usage, risks, and regulatory exposure</li><li><strong>Strategy:</strong> Definition of governance frameworks, ownership models, and quality standards</li><li><strong>Implementation:</strong> Tooling, automation, and controls embedded into data and analytics workflows</li><li><strong>Monitoring:</strong> Ongoing measurement, alerts, and reporting for quality, risk, and compliance</li></ul><p>This integrated approach ensures that governance and data quality directly support business analytics, Power BI and Tableau dashboards, and AI-driven decision systems.</p><p>Book a 30-min session and <a href="https://calendly.com/perceptive-analytics/30min?utm_source=blog&amp;utm_medium=website&amp;utm_campaign=aeo"><strong>Talk to Our AI Consultants</strong></a></p><h3>2. Key Features of Perceptive Analytics Data Quality Solutions</h3><p>Perceptive Analytics’ data quality solutions are designed for <strong>operational analytics environments</strong>, not theoretical data governance models.</p><p><strong>Core capabilities include:</strong></p><ul><li><strong>Data profiling:</strong> Automated discovery of completeness, consistency, and accuracy issues</li><li><strong>Data cleansing and standardization:</strong> Rules-based and automated correction workflows</li><li><strong>Matching and de-duplication:</strong> Entity resolution across customer, product, and transaction data</li><li><strong>Enrichment:</strong> Integration of internal and external reference data</li><li><strong>Continuous monitoring:</strong> Data quality KPIs, thresholds, and alerts embedded in analytics workflows</li><li><strong>Metadata management:</strong> Business definitions, technical metadata, and ownership visibility</li><li><strong>Data lineage:</strong> End-to-end traceability from source systems to BI dashboards and AI models</li><li><strong>Automation:</strong> Reduced manual data preparation across Power BI, Tableau, Looker, and AI pipelines</li></ul><p>By embedding these features directly into analytics workflows, data quality becomes measurable, repeatable, and enforceable — not dependent on manual cleanup.</p><p><strong>Learn more:</strong><a href="https://www.perceptive-analytics.com/snowflake-vs-bigquery-which-is-better-for-the-growth-stage/"> Snowflake vs BigQuery for Growth-Stage Companies</a></p><h3>3. AI Governance Approach and Regulatory Compliance</h3><p>Perceptive Analytics’ AI governance approach focuses on <strong>practical risk management</strong>, not just documentation.</p><p><strong>Key elements of the governance framework include:</strong></p><ul><li><strong>AI policies and standards:</strong> Model usage, data requirements, and approval workflows</li><li><strong>Model risk management:</strong> Versioning, validation, and lifecycle controls</li><li><strong>Bias and fairness checks:</strong> Monitoring for skewed outcomes and unintended impacts</li><li><strong>Explainability:</strong> Transparent model logic and decision drivers for business and regulators</li><li><strong>Audit trails:</strong> Traceability of data, model changes, and decision outputs</li><li><strong>Regulatory alignment:</strong> Mapping controls to frameworks such as the NIST AI Risk Management Framework and ISO/IEC-aligned AI management concepts</li></ul><p>This approach supports compliance requirements across regulated industries, while enabling faster and safer adoption of analytics and GenAI.</p><h3>4. How Perceptive Analytics Compares to Other AI Governance Providers</h3><p>Perceptive Analytics differentiates itself by operating <strong>inside analytics and BI environments</strong>, rather than offering generic governance tooling.</p><p><strong>Key points of differentiation:</strong></p><ul><li>Deep expertise in enterprise analytics, BI, and data engineering — not just governance theory</li><li>Tailored governance frameworks aligned to real business use cases</li><li>Strong linkage between data quality, BI reliability, and AI risk controls</li><li>Hands-on implementation and integration support, not advisory-only engagements</li><li>Flexible architectures that work with existing tools and platforms</li></ul><p>This makes governance actionable for analytics teams, not an abstract compliance layer.</p><p><strong>Read more:</strong> <a href="https://www.perceptive-analytics.com/5-ways-to-make-analytics-faster/">5 Ways to Make Analytics Faster</a></p><h3>5. Proof of Impact: Case Studies and Success Stories</h3><p>Perceptive Analytics’ AI governance and data quality solutions have delivered measurable outcomes across industries.</p><p><strong>Illustrative examples (anonymized):</strong></p><ul><li><strong>Financial services:</strong></li><li>Challenge: Inconsistent data and undocumented model changes</li><li>Solution: Data quality automation + AI governance controls</li><li>Outcome: 35% improvement in data accuracy; faster internal model approvals</li><li><strong>Healthcare analytics:</strong></li><li>Challenge: Regulatory risk from untracked data lineage</li><li>Solution: Lineage, metadata, and audit trail implementation</li><li>Outcome: Reduced compliance findings and faster reporting cycles</li><li><strong>Retail &amp; consumer data:</strong></li><li>Challenge: High manual data cleanup effort</li><li>Solution: Automated profiling and cleansing pipelines</li><li>Outcome: ~50% reduction in manual data preparation time</li></ul><p>These results demonstrate how governance and data quality directly improve analytics performance and risk posture.</p><h3>6. Pricing Options and Engagement Models</h3><p>Perceptive Analytics offers flexible engagement models aligned to enterprise needs and maturity.</p><p><strong>Common pricing and engagement options:</strong></p><ul><li><strong>Assessment packages:</strong> Fixed-scope reviews of AI governance and data quality gaps</li><li><strong>Project-based engagements:</strong> Design and implementation of governance and quality solutions</li><li><strong>Subscription / managed services:</strong> Ongoing monitoring, controls, and optimization</li><li><strong>Pilot programs:</strong> Low-risk proof-of-value initiatives for specific use cases</li></ul><p>Pricing is typically scoped based on:</p><ul><li>Data volume and complexity</li><li>Number of systems, models, and use cases</li><li>Regulatory requirements and governance depth</li></ul><p>This transparency helps organizations plan budgets and justify investment internally.</p><h3>Choosing Perceptive Analytics for Trusted AI and High-Quality Data</h3><p>Trusted analytics and AI require more than advanced models — they require <strong>reliable data, governed workflows, and clear accountability</strong>. Perceptive Analytics brings these elements together, enabling enterprises to scale BI, analytics, and GenAI with confidence.</p><p>By combining data quality, AI governance, and implementation expertise, as an experienced <a href="https://www.perceptive-analytics.com/ai-consulting/"><strong>AI consulting company</strong></a>, Perceptive Analytics helps enterprises operationalize governance and data quality across analytics and GenAI initiatives.</p><p>Request an<a href="https://www.perceptive-analytics.com/ai-consulting/"> AI Governance and Data Quality Assessment</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=05b0ca68546d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Random Forest in 2025: Why Ensembling Still Reigns Supreme]]></title>
            <link>https://medium.com/@diptim_99684/random-forest-in-2025-why-ensembling-still-reigns-supreme-9298927721c1?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9298927721c1</guid>
            <category><![CDATA[r-language]]></category>
            <category><![CDATA[r-tutorial]]></category>
            <category><![CDATA[r-programming]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Tue, 12 Aug 2025 18:33:26 GMT</pubDate>
            <atom:updated>2025-08-12T18:33:26.472Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6aQ1V6IF4zUY9Wm1iStLqg.jpeg" /></figure><p>When you’re buying a car, would you just pick the first one you see? Probably not.<br> You’d ask a few friends, read reviews, compare specs, and maybe even check YouTube videos before making your decision.</p><p>The same goes for choosing a movie — unless it stars your all-time favorite actor, you’ll likely look for multiple opinions before committing two hours of your life to it.</p><p>Why?<br> Because relying on a single opinion can introduce bias. One friend might dislike a destination because of a bad personal experience, while ten others might absolutely love it. By consulting multiple people, you’re essentially <em>reducing individual bias and getting closer to the “true” consensus.</em></p><p>The business world uses the same principle in interviews. Companies often conduct multiple interview rounds, even if the questions overlap. Why? To ensure different recruiters independently reach the same decision. If multiple interviewers agree, the probability of a good hire increases.</p><p>In analytics and data science, this approach has a name: <strong>ensembling</strong>.</p><h4>What Is Ensembling?</h4><p>Ensembling is a supervised learning technique where <strong>multiple models</strong> are trained and their outputs are combined according to a rule to generate the final prediction.</p><p>Breaking it down:</p><ul><li><strong>Multiple models:</strong> These can be the same model type with different hyperparameters or entirely different model architectures.</li><li><strong>Slight variation in training:</strong> Each model might see slightly different subsets of data or features.</li><li><strong>Combining outputs:</strong></li><li>For numeric predictions, we often take the average.</li><li>For classification, we use <strong>majority voting</strong> — the most common predicted class wins.</li></ul><p>The result? Reduced bias, reduced variance, and often significantly better performance.</p><h4>Random Forest: The Ensembling Workhorse</h4><p>One of the most popular ensembling algorithms is <strong>Random Forest</strong>.<br> It’s essentially an army of decision trees working together.</p><p>Here’s the idea:</p><ul><li><strong>Decision Trees</strong> split data based on features to maximize information gain at each node.</li><li>While easy to interpret, single decision trees are <em>weak learners</em> — they often overfit and have limited predictive power.</li><li><strong>Random Forest</strong> solves this by:</li><li>Building <em>many</em> decision trees</li><li>Randomly sampling both <strong>data points</strong> and <strong>features</strong> for each tree.</li><li>Combining predictions to get the final result.</li></ul><p>This randomness reduces correlation between trees, improves generalization, and removes much of the bias a single decision tree might introduce.</p><h4>2025 Case Study: Predicting Car Acceptability</h4><p>For this example, we’ll use the <strong>Car Evaluation Dataset</strong> from the UCI Machine Learning Repository.<br> The dataset has:</p><ul><li><strong>6 explanatory variables</strong>: Buying Price, Maintenance, NumDoors, NumPersons, BootSpace, Safety.</li><li><strong>1 target variable</strong>: Condition (car acceptability).</li></ul><p>All variables are categorical.</p><p>We’ll:</p><ol><li>Split the data into <strong>70% training</strong> and <strong>30% validation</strong>.</li><li>Train multiple Random Forest models with different mtry values.</li><li>Compare against a single Decision Tree model.</li></ol><h4>Step 1: Setup and Data Exploration</h4><pre># Load required packages<br>install.packages(&quot;randomForest&quot;)<br>library(randomForest)</pre><pre># Load dataset<br>data1 &lt;- read.csv(file.choose(), header = TRUE)</pre><pre># Quick exploration<br>str(data1)<br>summary(data1)</pre><h4>Step 2: Train–Validation Split</h4><pre>set.seed(100)<br>train_idx &lt;- sample(nrow(data1), 0.7 * nrow(data1), replace = FALSE)<br>TrainSet &lt;- data1[train_idx, ]<br>ValidSet &lt;- data1[-train_idx, ]</pre><h4>Step 3: Baseline Random Forest Model</h4><pre>model1 &lt;- randomForest(Condition ~ ., data = TrainSet, importance = TRUE)<br>model1</pre><ul><li><strong>Default settings:</strong> 500 trees, mtry = 2</li><li><strong>OOB error:</strong> ~3.64%</li></ul><h4>Step 4: Fine-Tuning mtry</h4><pre>model2 &lt;- randomForest(Condition ~ ., data = TrainSet, ntree = 500, mtry = 6, importance = TRUE)<br>model2</pre><ul><li><strong>OOB error:</strong> Reduced to ~2.32%</li><li><strong>Validation accuracy:</strong> ~98.84%</li></ul><p>We can also check variable importance:</p><pre>importance(model2)<br>varImpPlot(model2)</pre><p>Safety, Number of Persons, and Buying Price are the top predictors.</p><h4>Step 5: Optimal mtry Search</h4><pre>acc &lt;- c()<br>for (i in 3:8) {<br>  model_temp &lt;- randomForest(Condition ~ ., data = TrainSet, ntree = 500, mtry = i, importance = TRUE)<br>  pred &lt;- predict(model_temp, ValidSet)<br>  acc[i-2] &lt;- mean(pred == ValidSet$Condition)<br>}<br>plot(3:8, acc, type = &quot;b&quot;)</pre><p><strong>Finding:</strong> Best accuracy at mtry = 8.</p><h4>Step 6: Comparing with Decision Tree</h4><pre>install.packages(c(&quot;rpart&quot;, &quot;caret&quot;, &quot;e1071&quot;))<br>library(rpart)<br>library(caret)<br>library(e1071)</pre><pre>model_dt &lt;- train(Condition ~ ., data = TrainSet, method = &quot;rpart&quot;)<br>pred_dt &lt;- predict(model_dt, ValidSet)<br>mean(pred_dt == ValidSet$Condition)</pre><ul><li><strong>Decision Tree accuracy:</strong> ~77.6% on validation set</li><li><strong>Random Forest accuracy:</strong> ~98.8% on validation set</li></ul><h4>Key Takeaways for 2025</h4><ul><li><strong>Random Forest remains a strong baseline</strong> in 2025 for classification tasks — fast to train, robust to overfitting, and interpretable enough for business contexts.</li><li><strong>Ensembling beats single models</strong> in most cases, especially when individual learners are weak.</li><li>Feature importance plots from Random Forest are still a great first step for feature selection and business insight.</li><li>While advanced methods like XGBoost, LightGBM, and CatBoost dominate Kaggle leaderboards, <strong>Random Forest is still a go-to algorithm</strong> for many production systems where interpretability and stability are valued.</li></ul><p><strong>Final Word:</strong><br> Random Forest isn’t just a 2010s favorite — it’s a 2025 workhorse. Whether you’re a data scientist working on customer churn, credit scoring, or product recommendation, it’s worth adding Random Forest to your toolkit.</p><p>Happy modeling!</p><p>Drive smarter decisions with a trusted <a href="https://www.perceptive-analytics.com/microsoft-power-bi-developer-consultant/"><strong>Power BI development company</strong></a> to design custom analytics solutions, expert <a href="https://www.perceptive-analytics.com/tableau-consultants/"><strong>Tableau Consulting</strong></a> for powerful data visualization, and seasoned <a href="https://www.perceptive-analytics.com/snowflake-consultants/"><strong>Snowflake Consultants</strong></a> to optimize your cloud data warehouse. Together, these services turn complex datasets into clear, actionable insights.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9298927721c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Missing Data in R? Complete 2025 Guide to Imputation Techniques]]></title>
            <link>https://medium.com/@diptim_99684/missing-data-in-r-complete-2025-guide-to-imputation-techniques-727173f24555?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/727173f24555</guid>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[tableau]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Mon, 11 Aug 2025 19:09:18 GMT</pubDate>
            <atom:updated>2025-08-11T19:09:18.007Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*2GU_4uKK6NllNZMvsEUUWA.png" /></figure><p>Handling missing values is still one of the most frustrating challenges for data analysts and data scientists—even in 2025.<br>While storage and processing power have grown exponentially, messy data remains a constant. The smarter approach today, as always, is not to blindly drop incomplete rows but to <strong>impute</strong> missing values intelligently, preserving as much information as possible.</p><h4>Missing Data in Analysis</h4><p>When working on real-world datasets, missing values can quietly sabotage your model’s accuracy and bias insights if left untreated.</p><p>If a dataset is very large and missing values account for less than ~5% of the data, analysts may sometimes ignore them without major impact. However, if the proportion is higher, ignoring them risks throwing away useful information and introducing bias.</p><p>In such cases, <strong>imputation</strong> — replacing missing values with statistically or algorithmically derived estimates — is preferred. With modern tools, imputations can now leverage <strong>machine learning, generative AI, and advanced statistical modeling</strong> for better accuracy.</p><h4>What Are Missing Values?</h4><p>Imagine you’re running an online survey:</p><ul><li>Married respondents fill in their spouse’s name.</li><li>Single respondents skip that field.</li><li>Some people leave it blank even if married, or accidentally enter irrelevant information.</li></ul><p>These blanks represent <strong>missing values</strong>, which can result from:</p><ul><li>Skipped questions</li><li>Input errors</li><li>Sensor failures in IoT data</li><li>Data corruption during transfer</li><li>Privacy-based non-responses</li></ul><h4>Types of Missing Values</h4><p>Missing data typically falls into <strong>three categories</strong>:</p><ol><li><strong>MCAR (Missing Completely at Random)</strong><br>No pattern exists — the missingness is unrelated to any variable in the dataset.</li><li><strong>MAR (Missing at Random)</strong><br>Missingness depends on observed variables.<br>Example: In a health survey, younger respondents may skip income-related questions more often.</li><li><strong>NMAR (Not Missing at Random)</strong><br>Missingness is related to the unobserved value itself.<br>Example: A person doesn’t report their cholesterol because it’s abnormally high.</li></ol><p><strong>Key 2025 note:</strong><br>While MCAR can be safely ignored in many cases, MAR and NMAR require deliberate handling. NMAR remains the hardest case — often requiring domain expertise, additional data collection, or model-based imputation.</p><h4>Imputing Missing Values</h4><p>The simplest imputation strategies include:</p><ul><li><strong>Numerical Data:</strong> Replace with mean, median, or predictive mean matching.</li><li><strong>Categorical Data:</strong> Replace with mode or the most frequent value.</li><li><strong>Time Series:</strong> Use moving averages, forward/backward fill, or interpolation.</li></ul><p>However, in 2025, analysts often turn to <strong>model-based imputations</strong>:</p><ul><li><strong>Random Forest-based Imputation (missForest)</strong></li><li><strong>Multiple Imputation by Chained Equations (mice)</strong></li><li><strong>Bayesian methods</strong></li><li><strong>K-Nearest Neighbors Imputation</strong></li><li><strong>Deep Learning Imputation</strong> (e.g., using autoencoders for structured data)</li></ul><p><strong>Tip:</strong> Avoid imputing with arbitrary constants (like -1) unless specifically needed for flagging missingness — these placeholders can distort models.</p><h4>Popular R Packages for Imputation (2025)</h4><ul><li><strong>mice</strong> — Multiple Imputation via Chained Equations (still a gold standard for MAR data)</li><li><strong>missForest</strong> — Non-parametric imputation using Random Forests, works well for mixed data types</li><li><strong>Hmisc</strong> — Traditional but robust imputation functions</li><li><strong>Amelia</strong> — Fast bootstrapping-based imputation for large datasets</li><li><strong>simputation</strong> — Simple, flexible imputation workflows</li><li><strong>recipes</strong> (tidymodels) — Preprocessing pipelines with built-in imputation steps</li><li><strong>softImpute</strong> — Matrix completion for high-dimensional data</li></ul><p>Many analysts now combine R packages with <strong>Python-based imputation via reticulate</strong>, enabling hybrid workflows.</p><h4>Example: Imputing with mice in R</h4><p>We’ll use the NHANES dataset from the <strong>VIM</strong> package to demonstrate.</p><p># Load packages<br>library(mice)<br>library(VIM)<br>library(lattice)</p><p># Load data<br>data(nhanes)</p><p># Convert age to factor<br>nhanes$age &lt;- as.factor(nhanes$age)</p><p># Visualize missingness pattern<br>md.pattern(nhanes)</p><p># Plot missing data patterns<br>aggr(nhanes, col=c(&#39;navyblue&#39;,&#39;red&#39;), numbers=TRUE, sortVars=TRUE,<br> labels=names(nhanes), cex.axis=.7, gap=3,<br> ylab=c(&quot;Proportion of Missingness&quot;,&quot;Missingness Pattern&quot;))</p><h4>Imputation with mice</h4><p># Run multiple imputations<br>mice_imputes &lt;- mice(nhanes, m = 5, maxit = 40, method = &#39;pmm&#39;)</p><p># Check methods used<br>mice_imputes$</p><p># Complete data from one imputed dataset (e.g., 5th)<br>Imputed_data &lt;- complete(mice_imputes, 5)</p><h4>Checking Imputation Quality</h4><p># Compare distributions<br>xyplot(mice_imputes, bmi ~ chl | .imp, pch = 20, cex = 1.4)<br>densityplot(mice_imputes)</p><p>If the red (imputed) and blue (observed) distributions align closely, the imputation is likely reasonable.</p><h4>Modeling with Multiple Imputations</h4><p>Rather than using just one completed dataset, you can fit models across all imputations and pool results:</p><p># Fit linear model across imputations<br>lm_5_model &lt;- with(mice_imputes, lm(chl ~ age + bmi + hyp))</p><p># Pool results<br>combo_5_model &lt;- pool(lm_5_model)<br>summary(combo_5_model)</p><h4>2025 Best Practices for Imputation</h4><ol><li><strong>Never guess blindly</strong> — understand the missingness mechanism first.</li><li><strong>Use multiple imputations</strong> for better statistical validity.</li><li><strong>Leverage machine learning</strong> for complex or high-dimensional data.</li><li><strong>Document imputation logic</strong> — transparency is key for reproducibility.</li><li><strong>Evaluate impact</strong> — compare models with and without imputation.</li><li><strong>Consider AI-enhanced tools</strong> — packages now integrate with GPT-based assistants for context-aware imputations.</li></ol><h4>Final Word</h4><p>Imputation is not just a preprocessing step — it’s a <strong>modeling decision</strong> that can shape the quality of your insights.<br>With tools like mice, missForest, and modern AI-based methods, analysts in 2025 have more power than ever to ensure missing data doesn’t mean missing insights.</p><p>At Perceptive Analytics, we empower businesses to unlock the true value of their data through tailored BI implementations. With over 20 years of experience and 100+ clients served, our expertise spans end-to-end data transformation.<br> Our core offerings include:</p><blockquote>Scalable <a href="https://www.perceptive-analytics.com/microsoft-power-bi-developer-consultant/"><strong>Power BI implementation services</strong></a> designed to deliver enterprise-ready dashboards.</blockquote><blockquote>Fully managed <a href="https://www.perceptive-analytics.com/tableau-development-services/"><strong>Tableau development services</strong></a> for visually compelling and insight-driven analytics.</blockquote><blockquote>Recognition among elite <a href="https://www.perceptive-analytics.com/tableau-consulting/"><strong>Tableau Consulting Companies</strong></a> known for quality, speed, and actionable outcomes.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=727173f24555" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Regression Models in R using Support Vector Regression (SVR) — 2025 Update]]></title>
            <link>https://medium.com/@diptim_99684/building-regression-models-in-r-using-support-vector-regression-svr-2025-update-e1c73f0ab32c?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1c73f0ab32c</guid>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[r-tutorial]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Sun, 10 Aug 2025 18:24:12 GMT</pubDate>
            <atom:updated>2025-08-10T18:24:54.401Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Regression Models in R using Support Vector Regression (SVR) — 2025 Update</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Tn8b8JWJfni-fNkWI7abQA.png" /></figure><p>Support Vector Regression (SVR) is a powerful machine learning technique for predicting continuous values, derived from the same principles that drive Support Vector Machines (SVM) in classification tasks.</p><p>In this article, we compare <strong>Simple Linear Regression (SLR)</strong> and <strong>SVR</strong> using the same dataset to illustrate how SVR can better handle <strong>non-linear relationships</strong>. We also walk through <strong>model tuning</strong> to improve predictive performance.</p><h4>1. Quick Review: Simple Linear Regression (SLR)</h4><p>SLR estimates the relationship between a dependent variable YYY and an independent variable XXX by fitting a straight line:</p><p>Y=α+βX+ϵY = \alpha + \beta X + \epsilonY=α+βX+ϵ</p><p>The parameters α\alphaα (intercept) and β\betaβ (slope) are estimated using the <strong>Ordinary Least Squares (OLS)</strong> method, which minimizes the sum of squared residuals.</p><h4>Example in R</h4><pre># Load tidyverse for modern data handling<br>library(tidyverse)</pre><pre># Read CSV (ensure the file is in your working directory)<br>data &lt;- read_csv(&quot;SVM.csv&quot;)</pre><pre># Scatter plot<br>ggplot(data, aes(x = X, y = Y)) +<br>  geom_point(color = &quot;black&quot;) +<br>  labs(title = &quot;Scatter Plot of X vs Y&quot;) +<br>  theme_minimal()</pre><pre># Fit SLR model<br>slr_model &lt;- lm(Y ~ X, data = data)</pre><pre># Add regression line<br>ggplot(data, aes(x = X, y = Y)) +<br>  geom_point() +<br>  geom_smooth(method = &quot;lm&quot;, se = FALSE, color = &quot;blue&quot;) +<br>  theme_minimal()</pre><h4>RMSE Calculation</h4><p>In 2025, instead of hydroGOF, the yardstick package from the <strong>tidymodels</strong> ecosystem is preferred for model evaluation.</p><pre>library(yardstick)</pre><pre># Predictions<br>data &lt;- data %&gt;% mutate(pred_slr = predict(slr_model, .))</pre><pre># RMSE<br>rmse(data, truth = Y, estimate = pred_slr)</pre><h4>2. Support Vector Regression (SVR)</h4><p>SVR adapts the principles of SVM to regression problems.<br> Key features:</p><ul><li><strong>Kernel functions</strong> (Linear, Polynomial, Sigmoid, Radial Basis Function) allow modeling complex, non-linear relationships.</li><li><strong>Epsilon margin</strong> defines a tolerance zone where errors are ignored.</li><li><strong>Cost parameter ©</strong> controls the penalty for predictions outside the epsilon margin.</li><li>Less sensitive to the distributional assumptions required by linear regression.</li></ul><h4>Implementing SVR in R</h4><p>For 2025, the e1071 package is still widely used, but many workflows now leverage the kernlab package for more advanced kernel methods and tidymodels for consistent syntax.</p><h4>Using e1071 (classic approach)</h4><pre>library(e1071)</pre><pre># Fit SVR with default RBF kernel<br>svr_model &lt;- svm(Y ~ X, data = data, type = &quot;eps-regression&quot;)</pre><pre># Predictions<br>data &lt;- data %&gt;% mutate(pred_svr = predict(svr_model, .))</pre><pre># RMSE<br>rmse(data, truth = Y, estimate = pred_svr)</pre><h4>3. Tuning the SVR Model</h4><p>In 2025, hyperparameter tuning is most commonly done via <strong>tidymodels</strong>’ tune framework, but you can still use tune() from e1071.</p><h4>Modern Tidymodels Example</h4><pre>library(tidymodels)</pre><pre># SVR model specification<br>svr_spec &lt;- svm_rbf(mode = &quot;regression&quot;) %&gt;%<br>  set_engine(&quot;kernlab&quot;)</pre><pre># Cross-validation folds<br>set.seed(123)<br>folds &lt;- vfold_cv(data, v = 5)</pre><pre># Tuning grid<br>svr_grid &lt;- grid_regular(cost(), rbf_sigma(), levels = 5)</pre><pre># Workflow<br>svr_wf &lt;- workflow() %&gt;%<br>  add_model(svr_spec) %&gt;%<br>  add_formula(Y ~ X)</pre><pre># Tune<br>svr_tuned &lt;- tune_grid(<br>  svr_wf,<br>  resamples = folds,<br>  grid = svr_grid,<br>  metrics = metric_set(rmse)<br>)</pre><pre># Best parameters<br>best_params &lt;- select_best(svr_tuned, &quot;rmse&quot;)</pre><pre># Final model<br>final_svr &lt;- finalize_workflow(svr_wf, best_params) %&gt;%<br>  fit(data)</pre><h4>4. Results Comparison</h4><p>ModelRMSESLR0.94SVR (default)0.43Tuned SVR0.27</p><p>The tuned SVR model clearly outperforms both the SLR and untuned SVR models, especially on non-linear data.</p><h4>5. Visualization of Model Fits</h4><pre>ggplot(data, aes(x = X)) +<br>  geom_point(aes(y = Y), color = &quot;black&quot;) +<br>  geom_line(aes(y = pred_svr), color = &quot;blue&quot;, size = 1) +<br>  geom_line(aes(y = predict(final_svr, new_data = data)$.pred),<br>            color = &quot;red&quot;, linetype = &quot;dashed&quot;, size = 1) +<br>  labs(title = &quot;SLR vs SVR vs Tuned SVR&quot;,<br>       y = &quot;Predicted Values&quot;) +<br>  theme_minimal()</pre><h4>6. Conclusion</h4><ul><li><strong>SLR</strong> is easy to implement but struggles with non-linear data.</li><li><strong>SVR</strong> handles non-linear relationships well via kernel tricks.</li><li><strong>Tuning</strong> SVR hyperparameters (cost, epsilon, sigma for RBF kernels) is crucial for best performance.</li><li>In modern R workflows (2025), <strong>tidymodels</strong> provides a unified, consistent framework for modeling, tuning, and evaluation.</li></ul><p><strong>Recommendation:</strong></p><ul><li>Use SLR for quick, interpretable models when relationships are mostly linear.</li><li>Use SVR for complex, non-linear patterns—especially when predictive accuracy is the priority.</li></ul><blockquote>At Perceptive Analytics, we help organizations transform data into strategy through expert <a href="https://www.perceptive-analytics.com/tableau-development-services/"><strong>tableau development services</strong></a>, dedicated guidance from an experienced <a href="https://www.perceptive-analytics.com/microsoft-power-bi-developer-consultant/"><strong>Power BI consultant</strong></a>, and innovative <a href="https://www.perceptive-analytics.com/ai-consulting/"><strong>AI Consulting</strong></a> for next-generation analytics solutions. With over two decades of experience, we empower businesses to make faster, smarter decisions.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1c73f0ab32c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Matrix to Mind: Reinforcement Learning in R for 2025]]></title>
            <link>https://medium.com/@diptim_99684/from-matrix-to-mind-reinforcement-learning-in-r-for-2025-b4f51dfb3b4b?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4f51dfb3b4b</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[r-language]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Fri, 08 Aug 2025 02:54:15 GMT</pubDate>
            <atom:updated>2025-08-08T02:54:15.145Z</atom:updated>
            <content:encoded><![CDATA[<p>Reinforcement learning (RL) has evolved from an academic curiosity into a cornerstone of modern AI systems—powering everything from game-playing agents to real-time recommendation engines and robotic control. While R isn’t the primary tool for deep reinforcement learning, it remains a powerful and accessible platform for modeling foundational RL concepts.</p><p>This article walks you through how to perform reinforcement learning using R, leveraging current packages and trends in 2025.</p><h4>What Is Reinforcement Learning?</h4><p>Reinforcement learning (RL) is a branch of machine learning where an agent learns to make decisions by interacting with an environment to maximize a reward signal.</p><p>Unlike supervised learning, RL doesn’t require labeled datasets. Instead, it depends on feedback from actions. The learning is driven by trial-and-error — similar to how humans learn through experience.</p><h4>Real-Life Use Cases in 2025</h4><p>Reinforcement learning is now mainstream in several domains:</p><ul><li><strong>Autonomous Vehicles:</strong> Learning optimal driving policies through simulation.</li><li><strong>Financial Portfolio Management:</strong> Learning strategies to dynamically allocate assets.</li><li><strong>Personalized Recommendation Systems:</strong> Netflix, Spotify, and Amazon use RL to adaptively serve content.</li><li><strong>Healthcare:</strong> Optimizing treatment strategies over time.</li><li><strong>Industrial Robotics:</strong> RL agents now control robotic arms in dynamic assembly environments.</li></ul><h4>How Reinforcement Learning Works</h4><p>At its core, RL involves:</p><ul><li><strong>Agent:</strong> The decision-maker.</li><li><strong>Environment:</strong> Where the agent operates.</li><li><strong>State (S):</strong> A snapshot of the environment.</li><li><strong>Action (A):</strong> Choices the agent can make</li><li><strong>Reward (R):</strong> Feedback after each action.</li><li><strong>Policy (π):</strong> A mapping from states to actions.</li><li><strong>Value Function (V):</strong> Expected future rewards from a state.</li></ul><p>This framework is typically modeled using <strong>Markov Decision Processes (MDPs)</strong>.</p><h4>Implementing Reinforcement Learning in R</h4><p>Although Python dominates the RL space with libraries like TensorFlow and PyTorch, R offers powerful tools for prototyping and visualizing simpler RL problems.</p><p>Let’s walk through a foundational RL example: a grid world navigation task.</p><h4>Example: Grid World Navigation</h4><p>Imagine a 3x3 grid where an agent must reach a goal while avoiding a pit. The agent receives a:</p><ul><li><strong>Reward of +10</strong> for reaching the goal</li><li><strong>Penalty of -10</strong> for falling into a pit</li><li><strong>Step cost of -1</strong> for each move</li></ul><p>This simplified task demonstrates key RL dynamics.</p><h4>Using MDPtoolbox in R</h4><h4>Step 1: Install &amp; Load the Package</h4><pre># install.packages(&quot;MDPtoolbox&quot;)<br>library(MDPtoolbox)</pre><h4>Step 2: Define Actions as Transition Probability Matrices</h4><p>Each action (up, down, left, right) is a 4x4 transition matrix for a 2x2 grid (states: S1, S2, S3, S4).</p><pre>up &lt;- matrix(c(1,0,0,0, 0.7,0.2,0.1,0, 0,0.1,0.2,0.7, 0,0,0,1), nrow=4, byrow=TRUE)<br>down &lt;- matrix(c(0.3,0.7,0,0, 0,0.9,0.1,0, 0,0.1,0.9,0, 0,0,0.7,0.3), nrow=4, byrow=TRUE)<br>left &lt;- matrix(c(0.9,0.1,0,0, 0.1,0.9,0,0, 0,0.7,0.2,0.1, 0,0,0.1,0.9), nrow=4, byrow=TRUE)<br>right &lt;- matrix(c(0.9,0.1,0,0, 0.1,0.2,0.7,0, 0,0,0.9,0.1, 0,0,0.1,0.9), nrow=4, byrow=TRUE)</pre><pre>P &lt;- list(up=up, down=down, left=left, right=right)</pre><h4>Step 3: Define the Reward Matrix</h4><pre>R &lt;- matrix(c(-1, -1, -1, -1,<br>              -1, -1, -1, -1,<br>              -1, -1, -1, -1,<br>              10, 10, 10, 10), nrow=4, byrow=TRUE)</pre><h4>Step 4: Solve with Policy Iteration</h4><pre>solver &lt;- mdp_policy_iteration(P = P, R = R, discount = 0.1)<br>solver$policy<br>names(P)[solver$policy]<br>solver$V</pre><h4>Using the ReinforcementLearning Package (GitHub)</h4><p>This package supports tabular RL with Q-learning and is perfect for toy environments.</p><h4>Step 1: Install from GitHub</h4><pre># install.packages(&quot;devtools&quot;)<br>library(devtools)<br>install_github(&quot;nproellochs/ReinforcementLearning&quot;)<br>library(ReinforcementLearning)</pre><h4>Step 2: Use Prebuilt gridworldEnvironment</h4><pre>print(gridworldEnvironment)</pre><pre>states &lt;- c(&quot;s1&quot;, &quot;s2&quot;, &quot;s3&quot;, &quot;s4&quot;)<br>actions &lt;- c(&quot;up&quot;, &quot;down&quot;, &quot;left&quot;, &quot;right&quot;)</pre><pre>data &lt;- sampleExperience(N = 1000, env = gridworldEnvironment,<br>                         states = states, actions = actions)</pre><pre>model &lt;- ReinforcementLearning(data, s = &quot;State&quot;, a = &quot;Action&quot;, r = &quot;Reward&quot;, s_new = &quot;NextState&quot;)</pre><pre>model$Policy<br>model$Reward</pre><h4>Tic-Tac-Toe Example with Preloaded Data</h4><pre>data(&quot;tictactoe&quot;)<br>model_ttt &lt;- ReinforcementLearning(tictactoe, s = &quot;State&quot;, a = &quot;Action&quot;, r = &quot;Reward&quot;, s_new = &quot;NextState&quot;, iter = 1)</pre><pre># Policy and reward overview<br>head(model_ttt$Policy)<br>model_ttt$Reward</pre><h4>Modern Applications and Limitations</h4><p>In 2025, reinforcement learning is increasingly being combined with <strong>deep learning (Deep RL)</strong> and <strong>large language models (LLMs)</strong>. However, R-based solutions are still mostly limited to:</p><ul><li>Prototyping</li><li>Educational examples</li><li>Simple control problems</li></ul><p>For large-scale or neural-based RL (e.g., PPO, DQN, A3C), tools like <strong>Python (PyTorch, TensorFlow, Ray RLlib)</strong> remain dominant.</p><h4>Conclusion</h4><p>Reinforcement learning bridges AI and behavior — teaching machines to learn from experience. While R may not be the go-to tool for large-scale RL applications, it remains excellent for modeling fundamental MDPs, experimenting with policies, and visualizing learning processes.</p><p>As the landscape continues to evolve with advancements like <strong>multi-agent RL</strong>, <strong>offline RL</strong>, and <strong>RL + LLM hybrids</strong>, gaining a strong conceptual foundation using R can be a powerful starting point.</p><blockquote>At Perceptive Analytics, we help businesses build robust, scalable data infrastructures that support smarter decision-making. With 20+ years of experience, we’ve guided 100+ organizations through their data transformation journeys.<br> As a trusted <a href="https://www.perceptive-analytics.com/microsoft-power-bi-developer-consultant/"><strong>Power BI development company</strong></a>, we also offer seamless <a href="https://www.perceptive-analytics.com/tableau-consultants/"><strong>Tableau implementation services</strong></a> and expert <a href="https://www.perceptive-analytics.com/snowflake-consultants/"><strong>Snowflake Consultants</strong></a> to modernize your data ecosystem end to end.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4f51dfb3b4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Speed Up OR Conditions in Tableau Using the IN Function]]></title>
            <link>https://medium.com/@diptim_99684/how-to-speed-up-or-conditions-in-tableau-using-the-in-function-e1e22f200924?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1e22f200924</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Wed, 06 Aug 2025 18:50:28 GMT</pubDate>
            <atom:updated>2025-08-06T18:50:28.447Z</atom:updated>
            <content:encoded><![CDATA[<p>Poorly written calculations can seriously slow down your Tableau dashboards—especially when dealing with large datasets and complex logic. One common performance bottleneck? Using multiple OR conditions.</p><p>In this article, we’ll walk you through how replacing OR with the IN function can significantly improve dashboard performance.</p><h4>Case Study: Flight Data Dashboard</h4><p>We tested this on a dataset containing flight information. Initially, we used a calculation like this:</p><pre>IF [Carrier] = &quot;AA&quot; OR [Carrier] = &quot;DL&quot; OR [Carrier] = &quot;UA&quot; THEN &quot;Major Carrier&quot; ELSE &quot;Other&quot;</pre><p>This formula checks each condition one by one. As the number of conditions and rows grows, so does the processing time.</p><ul><li><strong>Initial load time:</strong> 41 seconds</li></ul><h4>Optimization #1: Use the IN Function</h4><p>The IN function offers a cleaner and more efficient way to compare a single field against multiple values:</p><pre>IF [Carrier] IN (&quot;AA&quot;, &quot;DL&quot;, &quot;UA&quot;) THEN &quot;Major Carrier&quot; ELSE &quot;Other&quot;</pre><p>This not only simplifies the calculation—it also performs better.</p><ul><li><strong>New load time:</strong><strong>IN</strong> 29 seconds<br> ✅ <strong>29% faster</strong></li></ul><h4>Optimization #2: Use Extracts</h4><p>We went a step further by switching the data source to an <strong>extract</strong> connection instead of a live one.</p><ul><li><strong>Load time with Extract:</strong> 0.43 seconds<br> ✅ <strong>98.9% faster than original</strong></li></ul><h4>Key Takeaway</h4><p>When you need to compare a field against multiple values, the IN function is both faster and easier to read than a long series of OR conditions.</p><p>At Perceptive Analytics, we empower businesses to unlock the true value of their data through tailored BI implementations. With over 20 years of experience and 100+ clients served, our expertise spans end-to-end data transformation.<br> Our core offerings include:</p><blockquote>Scalable <a href="https://www.perceptive-analytics.com/microsoft-power-bi-developer-consultant/"><strong>Power BI implementation services</strong></a> designed to deliver enterprise-ready dashboards.</blockquote><blockquote>Fully managed <a href="https://www.perceptive-analytics.com/tableau-development-services/"><strong>Tableau development services</strong></a> for visually compelling and insight-driven analytics.</blockquote><blockquote>Recognition among elite <a href="https://www.perceptive-analytics.com/tableau-consulting/"><strong>Tableau Consulting Companies</strong></a> known for quality, speed, and actionable outcomes.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1e22f200924" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Customer Lifetime Value (CLV)?]]></title>
            <link>https://medium.com/@diptim_99684/what-is-customer-lifetime-value-clv-64405c5fb374?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/64405c5fb374</guid>
            <category><![CDATA[customer-experience]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[customer-lifetime-value]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Tue, 05 Aug 2025 18:59:50 GMT</pubDate>
            <atom:updated>2025-08-05T18:59:50.362Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*1weMk4D-HYSwrDfTgkoeAA.png" /></figure><p>Customer Lifetime Value (CLV) is the total revenue a business can expect from a customer over the entire duration of their relationship.</p><p>Put simply, it’s a prediction of how valuable a customer is to your company — not just in one transaction, but across all future interactions. Some customers might bring in millions over time, while others may never return after the first purchase. CLV helps businesses identify which customers are worth investing in.</p><h4>Why is CLV Important?</h4><p>Understanding CLV isn’t just a metric — it’s a strategy. Here’s how businesses benefit:</p><ul><li><strong>Marketing Efficiency</strong>: CLV helps determine how much you can spend to acquire or retain a customer.</li><li><strong>Customer Segmentation</strong>: Identify high-value customers and tailor experiences for them.</li><li><strong>Product Strategy</strong>: Focus on products and services that bring in long-term value.</li><li><strong>Forecasting &amp; Budgeting</strong>: Predict future cash flow more accurately.</li></ul><p>CLV answers key business questions like:</p><ul><li>“Should I offer a discount to retain this customer?”</li><li>“Is it worth running this ad campaign?”</li><li>“Which customers are most valuable over time?”</li></ul><h4>Key Terms You Should Know</h4><p>Before we dive into the formula, let’s cover some important terms:</p><p>TermDescription<strong>ARPU</strong> (Average Revenue Per User)Revenue generated by each user during a given period.<strong>Churn Rate</strong>The percentage of customers who stop doing business with you in a given period.<strong>Retention Rate</strong>Percentage of customers who continue doing business with you. It’s the opposite of churn.<strong>Gross Margin</strong>Revenue minus cost of goods sold (COGS), expressed as a percentage.<strong>Discount Rate</strong>Reflects the time value of money — used in financial models to calculate present value.</p><h4>How to Calculate CLV</h4><p>There are several models, from simple to advanced. Let’s start with the basics:</p><h4>1. Simple CLV Formula</h4><p>A quick back-of-the-napkin method:</p><p>CLV=Average Purchase Value×Purchase Frequency×Customer Lifespan\text{CLV} = \text{Average Purchase Value} \times \text{Purchase Frequency} \times \text{Customer Lifespan}CLV=Average Purchase Value×Purchase Frequency×Customer Lifespan</p><h4>Example:</h4><ul><li>Avg. purchase value: $50</li><li>Purchase frequency: 10 times/year</li><li>Customer lifespan: 3 years</li></ul><p>CLV=50×10×3=$1,500CLV = 50 \times 10 \times 3 = \$1,500CLV=50×10×3=$1,500</p><p>This customer is worth $1,500 over 3 years.</p><h4>2. Gross Margin CLV</h4><p>A more realistic version that includes profit margin:</p><p>CLV=(ARPU×Gross Margin)÷Churn RateCLV = (\text{ARPU} \times \text{Gross Margin}) \div \text{Churn Rate}CLV=(ARPU×Gross Margin)÷Churn Rate</p><p>Let’s say:</p><ul><li>ARPU = $100/month</li><li>Gross Margin = 60% = 0.6</li><li>Churn Rate = 5% = 0.05</li></ul><p>CLV=(100×0.6)÷0.05=$1,200CLV = (100 \times 0.6) \div 0.05 = \$1,200CLV=(100×0.6)÷0.05=$1,200</p><h4>3. Discounted CLV</h4><p>Used in financial forecasting:</p><p>CLV=∑t=0T(Rt×G)(1+d)tCLV = \sum_{t=0}^{T} \frac{(R_t \times G)}{(1 + d)^t}CLV=t=0∑T​(1+d)t(Rt​×G)​</p><p>Where:</p><ul><li>RtR_tRt​ = revenue in time period ttt</li><li>GGG = gross margin</li><li>ddd = discount rate</li><li>TTT = time horizon</li></ul><p>This model accounts for how money loses value over time (time value of money).</p><h4>Machine Learning Approaches to CLV</h4><p>For businesses with large data sets, machine learning offers powerful methods to predict CLV more accurately. Two popular models are:</p><h4>1. BG/NBD (Beta-Geometric/Negative Binomial Distribution) Model</h4><p>This model predicts how often a customer will make a purchase in the future, based on past transaction data.</p><p>Ideal when:</p><ul><li>You have transactional data but not individual lifetime data.</li><li>You’re in an eCommerce or subscription business.</li></ul><h4>2. Gamma-Gamma Model</h4><p>Used to predict the <strong>monetary value</strong> of future transactions. Best used in combination with BG/NBD to estimate both frequency and value of purchases.</p><h4>Python Implementation (Using Lifetimes Package)</h4><pre>from lifetimes import BetaGeoFitter, GammaGammaFitter<br>from lifetimes.datasets import load_cdnow_summary_data_with_monetary_value</pre><pre>data = load_cdnow_summary_data_with_monetary_value()<br>bgf = BetaGeoFitter()<br>bgf.fit(data[&#39;frequency&#39;], data[&#39;recency&#39;], data[&#39;T&#39;])</pre><pre>ggf = GammaGammaFitter()<br>ggf.fit(data[&#39;frequency&#39;], data[&#39;monetary_value&#39;])</pre><pre># Predict CLV for 6 months<br>clv = ggf.customer_lifetime_value(<br>    bgf,<br>    data[&#39;frequency&#39;],<br>    data[&#39;recency&#39;],<br>    data[&#39;T&#39;],<br>    data[&#39;monetary_value&#39;],<br>    time=6,  # months<br>    discount_rate=0.01<br>)</pre><p>This approach helps create highly personalized customer strategies by forecasting expected revenue at the individual level.</p><h4>Final Thoughts</h4><p>Customer Lifetime Value isn’t just a metric — it’s a lens through which you view and shape your entire customer strategy.</p><p>By understanding who your most valuable customers are, you can:</p><ul><li>Increase retention,</li><li>Optimize marketing spend,</li><li>Improve profitability over the long term.</li></ul><p>Whether you go with a basic calculation or use advanced ML models, CLV empowers smarter business decisions.</p><p>At Perceptive Analytics, we help organizations build a solid data foundation with our <a href="https://www.perceptive-analytics.com/microsoft-power-bi-developer-consultant/"><strong>Power BI consultant</strong></a> expertise, scalable <a href="https://www.perceptive-analytics.com/power-bi-consulting/"><strong>Power BI consulting</strong></a> solutions, and end-to-end <a href="https://www.perceptive-analytics.com/tableau-consulting/"><strong>Tableau Consulting Services</strong></a> that drive actionable insights. With 20+ years of experience, we transform business goals into intelligent analytics outcomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64405c5fb374" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Practical Guide to Anomaly Detection]]></title>
            <link>https://medium.com/@diptim_99684/a-practical-guide-to-anomaly-detection-d7c8f018d137?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7c8f018d137</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[analysis]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Mon, 04 Aug 2025 08:07:32 GMT</pubDate>
            <atom:updated>2025-08-04T08:07:32.569Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine running a credit card business where a customer typically spends $25 every week — a predictable, consistent pattern. Then one day, a $700 transaction pops up. That deviation is enough to trigger an alarm. Was it a fraud? A one-off party? A life event like starting a family?</p><p>This is the world of <strong>anomaly detection</strong> — where identifying the <em>unusual</em> becomes critical, especially in sectors like finance, retail, or cybersecurity. With AI-powered dashboards and real-time alerts, companies can catch outliers before they become liabilities.</p><p>In this article, we’ll walk through how to detect anomalies using R, leveraging two powerful packages: AnomalyDetection by Twitter and anomalize by Business Science.</p><h4>Why Do Anomalies Matter?</h4><p>Anomalies (also called outliers) aren’t always bad — but they always signal that <em>something different</em> is happening.</p><p>In industries like <strong>banking, insurance, and e-commerce</strong>, catching anomalies early can help:</p><ul><li>Detect fraud (suspicious transactions or login attempts)</li><li>Monitor system failures (sudden traffic spikes or dips)</li><li>Understand behavior shifts (a customer abandoning routine)</li></ul><p>Tools like <strong>Power BI</strong> and AI-powered anomaly detection systems are becoming staples in modern analytics stacks. But under the hood, it all begins with identifying the <em>pattern</em> — and learning how to break it.</p><h4>Getting Started: Detecting Anomalies with the AnomalyDetection Package</h4><p>Let’s begin with the AnomalyDetection package, designed to identify spikes and dips in time-series data — even with trends or seasonality.</p><h4>Step 1: Install and Load Required Packages</h4><pre>install.packages(&quot;devtools&quot;)<br>install.packages(&quot;Rcpp&quot;)<br>library(devtools)<br>install_github(&quot;petermeissner/wikipediatrend&quot;)<br>install_github(&quot;twitter/AnomalyDetection&quot;)</pre><pre>library(Rcpp)<br>library(wikipediatrend)<br>library(AnomalyDetection)</pre><h4>Step 2: Fetch and Prepare Wikipedia Page Views</h4><p>We’ll analyze daily traffic for the Wikipedia page “FIFA” — a great proxy for public interest.</p><pre>fifa_data &lt;- wp_trend(&quot;fifa&quot;, from=&quot;2013-03-18&quot;, lang = &quot;en&quot;)</pre><p>Let’s keep only the relevant columns:</p><pre>fifa_data &lt;- fifa_data[, c(&quot;date&quot;, &quot;views&quot;)]</pre><p>Now visualize the data:</p><pre>library(ggplot2)<br>ggplot(fifa_data, aes(x = date, y = views)) + geom_line()</pre><h4>Step 3: Apply the Anomaly Detection Algorithm</h4><p>Using the AnomalyDetectionTs() function:</p><pre>anomalies &lt;- AnomalyDetectionTs(fifa_data, direction = &quot;pos&quot;, plot = TRUE)<br>anomalies$plot</pre><p>This identifies <strong>positive anomalies</strong> — sudden spikes in traffic. We can inspect them:</p><pre>anomalies$anoms</pre><p>These spikes could be related to FIFA matches, controversies, or player news. Without a known event, these anomalies may indicate a bot attack, API bug, or misinformation campaign.</p><h4>Level Up: Using the anomalize Package</h4><p>Unlike AnomalyDetection, the anomalize package is part of the tidyverse ecosystem and supports robust decomposition, making it more flexible for business use cases.</p><h4>Step 1: Install and Load Required Packages</h4><pre>install.packages(&quot;anomalize&quot;)<br>library(devtools)<br>install_github(&quot;business-science/anomalize&quot;)</pre><pre>library(anomalize)<br>library(tidyverse)<br>library(coindeskr)</pre><h4>Step 2: Load and Transform Bitcoin Price Data</h4><pre>bitcoin_data &lt;- get_historic_price(start = &quot;2017-01-01&quot;)</pre><pre>bitcoin_data_ts &lt;- bitcoin_data %&gt;%<br>  rownames_to_column() %&gt;%<br>  as_tibble() %&gt;%<br>  mutate(date = as.Date(rowname)) %&gt;%<br>  select(-rowname</pre><h4>Step 3: Decompose and Detect Anomalies</h4><p>Let’s break down the time series into trend, seasonality, and residuals.</p><pre>bitcoin_data_ts %&gt;%<br>  time_decompose(Price, method = &quot;stl&quot;) %&gt;%<br>  anomalize(remainder, method = &quot;gesd&quot;, alpha = 0.05) %&gt;%<br>  time_recompose() %&gt;%<br>  plot_anomalies(time_recomposed = TRUE, ncol = 3)</pre><p>This approach removes seasonality and highlights pure anomaly signals. The red dots? Those are the outliers.</p><p>You can also extract them directly:</p><pre>anomalies &lt;- bitcoin_data_ts %&gt;%<br>  time_decompose(Price) %&gt;%<br>  anomalize(remainder) %&gt;%<br>  time_recompose() %&gt;%<br>  filter(anomaly == &#39;Yes&#39;)</pre><h4>Comparing Approaches: SH-ESD vs. Decomposition</h4><p>FeatureAnomalyDetection (Twitter)anomalize (Business Science)Underlying algorithmSeasonal Hybrid ESDSTL + GESD or IQRSeasonality handlingYesYesVisualizationStaticInteractive ggplot2IntegrationStandaloneTidyverseCustomizable thresholdYesYes</p><h4>Conclusion: Know Your Outliers</h4><p>Anomalies are more than just “weird data points.” They’re your system’s way of saying: “Hey, look here!” Whether it’s a financial fraud, an unexpected surge in interest, or a sensor malfunction, detecting anomalies can lead to better decisions.</p><p>Both AnomalyDetection and anomalize are powerful, but they serve different needs. As you explore further, consider experimenting with:</p><ul><li><strong>tsoutliers</strong> for time-series outlier detection</li><li><strong>forecast</strong> package for ARIMA-based methods</li><li><strong>robustbase</strong> for robust statistics in noisy data</li></ul><p>In a world where <em>data never sleeps</em>, make sure you’re not snoozing on the anomalies.</p><p>At Perceptive Analytics, we help organizations build a solid data foundation with our <a href="https://www.perceptive-analytics.com/microsoft-power-bi-developer-consultant/"><strong>Power BI consultant</strong></a> expertise, scalable <a href="https://www.perceptive-analytics.com/power-bi-consulting/"><strong>Power BI consulting</strong></a> solutions, and end-to-end <a href="https://www.perceptive-analytics.com/tableau-consulting/"><strong>Tableau Consulting Services</strong></a> that drive actionable insights. With 20+ years of experience, we transform business goals into intelligent analytics outcomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7c8f018d137" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Support Vector Machines (SVM) in R: Tutorial with Code & Tuning (2025)]]></title>
            <link>https://medium.com/@diptim_99684/support-vector-machines-svm-in-r-tutorial-with-code-tuning-2025-aca840241a69?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/aca840241a69</guid>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[r-tutorial]]></category>
            <category><![CDATA[r-language]]></category>
            <category><![CDATA[r-programming-training]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Sat, 02 Aug 2025 19:30:29 GMT</pubDate>
            <atom:updated>2025-08-02T19:30:29.735Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QGgvu6zTrN2ffg4x0Vilzg.jpeg" /></figure><p>Support Vector Machines (SVMs) are powerful supervised learning models used for classification and regression. They work by finding an optimal hyperplane that separates different classes with the maximum possible margin. What makes SVMs especially robust is their ability to handle both linear and non-linear data using different kernels.</p><p>Today, SVMs are a go-to solution in situations where the data distribution is unknown or irregular — commonly seen in noisy or high-dimensional datasets. They are also less sensitive to outliers compared to traditional regression models.</p><h4>Conceptual Overview</h4><p>Imagine a simple dataset with two classes (e.g., red vs. blue dots) that are clearly separable. A variety of lines (or hyperplanes, in higher dimensions) can be drawn to separate the classes — but SVM identifies the one that offers the <strong>maximum margin</strong> from both classes. This reduces the chance of misclassification for future, unseen data.</p><p>Mathematically, this is represented by:</p><ul><li>Decision boundary: ax + b = 0</li><li>Supporting hyperplanes: ax + b = ±1</li><li>Margin width: 2 / ||a||</li></ul><p>SVM optimization aims to <strong>maximize the margin</strong> while ensuring correct classification of training points:</p><p>minimize:(1/2)∣∣a∣∣2subjectto:yi(axi+b)≥1minimize: (1/2)||a||² subject to: y_i(ax_i + b) ≥ 1 minimize:(1/2)∣∣a∣∣2subjectto:yi​(axi​+b)≥1</p><h4>Getting Started with SVM in R</h4><p>Let’s walk through a practical example using a toy dataset in R.</p><h4>Step 1: Simulate the Data</h4><pre>x &lt;- 1:20<br>y &lt;- c(3,4,5,4,8,10,10,11,14,20,23,24,32,34,35,37,42,48,53,60)<br>train &lt;- data.frame(x, y)</pre><pre># Visualize<br>plot(train, pch = 16, main = &quot;Simulated Data&quot;)</pre><h4>Step 2: Fit a Linear Regression Model (Baseline)</h4><pre>model_lm &lt;- lm(y ~ x, data = train)<br>abline(model_lm, col = &quot;darkgray&quot;, lwd = 2)</pre><h4>Step 3: Fit an SVM Model</h4><p>We’ll use the e1071 package, which provides a straightforward API for SVMs</p><pre>install.packages(&quot;e1071&quot;)  # Only if not already installed<br>library(e1071)</pre><pre>model_svm &lt;- svm(y ~ x, data = train)<br>pred_svm &lt;- predict(model_svm, train)</pre><pre># Overlay predictions<br>points(train$x, pred_svm, col = &quot;blue&quot;, pch = 4)</pre><h4>Step 4: Evaluate Model Performance</h4><pre>rmse &lt;- function(actual, predicted) sqrt(mean((actual - predicted)^2))</pre><pre>lm_rmse &lt;- rmse(train$y, predict(model_lm, train))     # ~3.83<br>svm_rmse &lt;- rmse(train$y, pred_svm)                    # ~2.70</pre><p>At this point, SVM already outperforms the linear model.</p><h4>Hyperparameter Tuning with Grid Search</h4><p>Modern practice involves tuning SVMs using cross-validation to optimize hyperparameters like cost (C) and epsilon (for regression tasks):</p><pre>svm_tune &lt;- tune(<br>  svm,<br>  y ~ x,<br>  data = train,<br>  ranges = list(<br>    epsilon = seq(0, 0.2, 0.01),<br>    cost = 2^(2:9)<br>  )<br>)</pre><pre># Best model info<br>summary(svm_tune)</pre><pre>best_model &lt;- svm_tune$best.model<br>best_pred &lt;- predict(best_model, train)<br>best_rmse &lt;- rmse(train$y, best_pred)  # ~1.29</pre><pre># Plot predictions<br>plot(train, pch = 16, main = &quot;Tuned SVM Model Fit&quot;)<br>points(train$x, best_pred, col = &quot;blue&quot;, pch = 4)</pre><p>You can also visualize the hyperparameter grid:</p><p>This heatmap helps identify the optimal parameter region and informs further fine-tuning. Be careful — going too deep into tuning without proper validation may lead to overfitting.</p><h4>Modern Best Practices &amp; Trends (2025)</h4><ul><li><strong>Use </strong><strong>caret or </strong><strong>tidymodels</strong> frameworks for consistent modeling pipelines.</li><li><strong>Cross-validation</strong> is now a standard. Avoid evaluating on training data alone.</li><li><strong>Non-linear kernels</strong> (like radial basis function or polynomial) are increasingly used for real-world tasks.</li><li><strong>Scale your data</strong> before using SVMs; it’s critical for proper margin calculation.</li><li><strong>Use larger datasets</strong> cautiously — SVMs don’t scale well with large samples; use LiblineaR or switch to tree-based methods like XGBoost if needed.</li></ul><h4>Summary</h4><p>ModelRMSELinear Model~3.83SVM~2.70Tuned SVM~1.29</p><p>SVM is an effective tool for both classification and regression tasks — especially when the underlying data distribution is unknown or non-linear. With hyperparameter tuning, SVM can deliver impressive accuracy. But, like all models, it requires careful validation to prevent overfitting.</p><h4>Final Code (Cleaned)</h4><pre># Data<br>x &lt;- 1:20<br>y &lt;- c(3,4,5,4,8,10,10,11,14,20,23,24,32,34,35,37,42,48,53,60)<br>train &lt;- data.frame(x, y)</pre><pre># Linear Regression<br>model_lm &lt;- lm(y ~ x, data = train)<br>abline(model_lm)</pre><pre># SVM<br>library(e1071)<br>model_svm &lt;- svm(y ~ x, data = train)<br>pred_svm &lt;- predict(model_svm, train)</pre><pre># RMSE Function<br>rmse &lt;- function(actual, predicted) sqrt(mean((actual - predicted)^2))</pre><pre># Errors<br>lm_rmse &lt;- rmse(train$y, predict(model_lm, train))<br>svm_rmse &lt;- rmse(train$y, pred_svm)</pre><pre># Tune SVM<br>svm_tune &lt;- tune(svm, y ~ x, data = train,<br>                 ranges = list(epsilon = seq(0,0.2,0.01), cost = 2^(2:9)))<br>best_model &lt;- svm_tune$best.model<br>best_pred &lt;- predict(best_model, train)<br>best_rmse &lt;- rmse(train$y, best_pred)</pre><pre># Visuals<br>plot(train, pch = 16)<br>points(train$x, best_pred, col = &quot;blue&quot;, pch = 4)<br>plot(svm_tune)</pre><p>At Perceptive Analytics, we empower businesses to turn complex data into clear insights. Whether you need automation from an experienced <a href="https://www.p2w2.com/excel-vba-programmer/"><strong>Excel VBA programmer</strong></a>, strategic dashboards from a <a href="https://www.perceptive-analytics.com/microsoft-power-bi-developer-consultant/"><strong>Power BI consultant</strong></a>, or cutting-edge innovation through <a href="https://www.perceptive-analytics.com/ai-consulting/"><strong>AI Consulting</strong></a>, we bring 20+ years of experience to deliver results that matter.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aca840241a69" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Create Groups Efficiently in Tableau?]]></title>
            <link>https://medium.com/@diptim_99684/how-to-create-groups-efficiently-in-tableau-9559e434473f?source=rss-5aebb0dd2f7f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9559e434473f</guid>
            <category><![CDATA[tableau-training]]></category>
            <category><![CDATA[tableau-tutorial]]></category>
            <category><![CDATA[tableau]]></category>
            <dc:creator><![CDATA[Dipti]]></dc:creator>
            <pubDate>Fri, 01 Aug 2025 03:26:01 GMT</pubDate>
            <atom:updated>2025-08-01T03:27:22.246Z</atom:updated>
            <content:encoded><![CDATA[<p>A great business dashboard combines high performance and ease of use.</p><p>In this article, we will show you how to efficiently create groups in Tableau.</p><p>You can also look this video to see how we did it!</p><h3>We Created Groups with Native Feature (28 Mn Rows)</h3><p>It was a movie review dataset.</p><p>We wanted to see the average rating of a selected few movies against the rest of them.</p><p>We created groups using Tableau’s native feature as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/0*8OlxLEq1RCMEH9py.png" /></figure><h3>The Viz Loaded in 2 Minutes 51 Seconds!</h3><p>A simple viz that shows the avg reviews of grouped movies, took <strong>2 min 51 secs</strong> to load!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/0*7z4wZBGkj9Qij0Jx.png" /></figure><p>The built-in group feature loads the entire domain of the dimension. Thus, it takes time.</p><p>As it is too long, we wanted to improve this.</p><h3>We used CASE Statement to Create Groups</h3><p>We created a Calculated Field using CASE Statement to do the grouping.</p><p>It only loads the named members of the domain. Thus, it would be faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/0*zMStMECIAZfBOhGp.png" /></figure><p>Now, the same viz took <strong>1 min 40 sec</strong> to load in Tableau with a Live connection — that is <strong>42% lesser time!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/0*OzUtsht2XXCMAczc.png" /></figure><p>The load time can be still reduced further if we leverage the power of Tableau Extracts as explained here.</p><p>Our Tableau Consulting team often recommends this method for clients dealing with large datasets.</p><h3>To Conclude</h3><p>Groups created with CASE Statements would perform better compared to the native ‘Create &gt; Group’ feature in Tableau.</p><p>Perceptive Analytics partners with businesses to unlock value in data and drive innovation. With two decades of experience, we’ve delivered results for 100+ clients worldwide.<br> Our expertise includes <a href="https://www.perceptive-analytics.com/tableau-development-services/"><strong>Tableau development services</strong></a>, <a href="https://www.perceptive-analytics.com/chatbot-consulting-services/"><strong>Chatbot Consulting</strong></a>, and <a href="https://www.perceptive-analytics.com/microsoft-power-bi-developer-consultant/"><strong>Power BI development services</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9559e434473f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>