<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lucas Nogueira on Medium]]></title>
        <description><![CDATA[Stories by Lucas Nogueira on Medium]]></description>
        <link>https://medium.com/@lucasnogsousa?source=rss-a304f6ccdc2b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CgqPpMw6T7-br4YYd1Ye7w.jpeg</url>
            <title>Stories by Lucas Nogueira on Medium</title>
            <link>https://medium.com/@lucasnogsousa?source=rss-a304f6ccdc2b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:57:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lucasnogsousa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why A/B Testing Isn’t Enough: Using Difference-in-Differences to Evaluate AI Rollouts]]></title>
            <link>https://medium.com/@lucasnogsousa/why-a-b-testing-isnt-enough-using-difference-in-differences-to-evaluate-ai-rollouts-62500878c5d8?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/62500878c5d8</guid>
            <category><![CDATA[zendesk]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Fri, 09 Oct 2026 19:50:41 GMT</pubDate>
            <atom:updated>2026-10-09T19:53:05.296Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3rM1YjeGoajZkqTeBlLtA.png" /></figure><p>When operational leadership deploys a new generative AI routing bot — <strong>an automated dispatcher analyzing incoming messages to route customers to the right specialist</strong> — or a ticket summarizer — <strong>a tool condensing long, messy interaction logs into crisp bullet points</strong> — the evaluation strategy is almost always a naive before-and-after comparison. If Average Handling Time (AHT) drops from 18 minutes in Q1 to 14 minutes in Q2, leadership immediately declares the AI ​​project a triumph.</p><p>However, this simple comparison ignores confounding variables — <strong>unobserved external factors, such as seasonal ticket surges, queue restructuring, or agent hiring sprees</strong> — and falls victim to selection bias — <strong>the systematic error that occurs when software is handed to an already-fast team rather than assigned randomly</strong> . Comparing a Tier 1 password-reset team (equipped with AI) against a Tier 2 technical bug team (without AI) is like racing a speedboat against a cargo ship and thanking the engine oil for the speedboat’s victory; the two groups were fundamentally different long before the software was ever installed.</p><p>To prove true causality in non-experimental settings — <strong>observational operational environments where software is rolled out organically without the luxury of a randomized A/B test</strong> — data scientists must build a valid counterfactual baseline — <strong>a parallel, untreated control group operating during the exact same timeframe to estimate what would have happened without the intervention</strong> . By comparing how the untreated group moves over time, Difference-in-Differences (DiD) acts like a statistical detective, filtering out background noise — <strong>company-wide shocks like system outages or holiday traffic spikes that hit every queue simultaneously</strong> — to isolate the pure causal impact of the AI ​​deployment.</p><h3><strong>1. The 2x2 Matrix: How the Control Group Acts as Your “Time Machine”</strong></h3><p>To understand how DiD strips away external noise, consider a real-world scenario where a company deploys a GenAI bot to Tier 1 Support in Q2, while Tier 2 Support continues operating manually without AI. During Q2, a massive platform outage hits the company, spiking ticket complexity and resolution times across all departments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*CxCyxydrlqGS0kOaoBB2ig.png" /></figure><p>If you only evaluated Tier 1 before and after the rollout, you would measure a -<strong>4 minute </strong>reduction and assume that was the total value of the AI.</p><p>However, look at the Control Group: without any AI intervention, Tier 2’s resolution time increased by<strong>+3 minutes </strong>due to the company-wide outage. That +3 minutes represents the <strong>unobserved background trend </strong>that Tier 1 was also fighting against.</p><p>Without the AI ​​bot, Tier 1’s resolution time would have increased from 22 to 25 minutes (the unobservable <strong>counterfactual</strong> ). By subtracting the Control Group’s change from the Treatment Group’s change, DiD calculates the true, isolated impact:</p><p><em>NetCausalEffect = (-4 mins) - (+3 mins) = -7 mins</em></p><p>The AI ​​bot didn’t just save 4 minutes — it actually saved <strong>7 minutes per ticket </strong>by absorbing the impact of the platform outage.</p><h3>2. Translating Intuition into Math: The OLS Regression Framework</h3><p>Subtracting four numbers in a 2x2 matrix gives us the raw point estimate, but it leaves a critical operational question unanswered: <strong>Is this -7 minute drop statistically significant, or did we just get lucky with random noise?</strong></p><p>To calculate confidence intervals, standard errors, and p-values ​​for executive decision-making, we translate the 2x2 table into an <strong>Ordinary Least Squares (OLS) regression model</strong> — a statistical method that estimates the relationship between a target variable and one or more predictor variables by minimizing the sum of squared differences between observed and predicted values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*o7R-qtnropwL7asHPxqNHg.png" /></figure><p>Where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*lWJw_gDWeyUDyzNP_HJeVw.png" /></figure><h3>Decoding the Coefficients (): The Anatomy of the Model</h3><p>Each coefficient () in this regression equation corresponds directly to a specific cell in our 2x2 matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yWigc9Tbi6_YVzpj4V2u4Q.png" /></figure><h3>3. The Core Assumption: Parallel Trends</h3><p>Before trusting any DiD regression output, we must pass one non-negotiable test: the <strong>parallel trends assumption</strong> — <strong>the fundamental requirement stating that, in the absence of the AI ​​intervention, the treatment and control groups would have evolved along identical trajectories over time</strong>.</p><p>While we can never directly observe what would have happened to Tier 1 without AI during Q2 (the unobservable counterfactual), we can validate this assumption by testing historical behavior. If Tier 1 and Tier 2 moved in visual and statistical lockstep across multiple weeks in Q1 before the AI ​​bot was introduced, we gain confidence that Tier 2 serves as a reliable proxy for global operational noise.</p><p>If pre-intervention trends diverged — <strong>meaning Tier 1 was already accelerating or decelerating independently of Tier 2 during Q1</strong> — our DiD estimate will be biased, and we must explore matching techniques like Synthetic Control or Propensity Score Matching.</p><h3>4. Full Python Implementation</h3><p>The following complete script simulates weekly support queue log data, fits an OLS Difference-in-Differences interaction model using statsmodels, tests for parallel trends during the pre-treatment period, and generates a publication-ready visualization.</p><pre>&quot;&quot;&quot;<br>Difference-in-Differences (DiD) Causal Inference Pipeline for AI Deployments<br><br>Author: Lucas Nogueira<br>Description: Simulates weekly ticket handling times, estimates the DiD interaction effect <br>             via statsmodels OLS, verifies parallel trends, and plots queue trajectories.<br>&quot;&quot;&quot;<br><br>import os<br>import numpy as np<br>import pandas as pd<br>import statsmodels.api as sm<br>import statsmodels.formula.api as smf<br>import matplotlib.pyplot as plt<br><br><br>def generate_did_data(n_weeks: int = 16, intervention_week: int = 9, seed: int = 42) -&gt; pd.DataFrame:<br>    &quot;&quot;&quot;Simulates weekly support ticket logs for Treatment and Control groups.&quot;&quot;&quot;<br>    np.random.seed(seed)<br>    records = []<br>    <br>    # Baseline resolution times in minutes (Tier 1 starts higher due to queue backlog)<br>    base_treat = 22.0<br>    base_control = 17.0<br>    <br>    # The true unobserved causal effect of the AI bot (-7.0 minutes)<br>    true_ai_effect = -7.0<br>    <br>    for week in range(1, n_weeks + 1):<br>        # Global operational noise (e.g., platform outage pushing times up over time)<br>        global_trend = 0.35 * week <br>        <br>        is_post = 1 if week &gt;= intervention_week else 0<br>        <br>        # 200 tickets generated per queue each week<br>        for _ in range(200):<br>            # Treatment Group (Tier 1 - Receives GenAI Routing Bot in Week 9)<br>            err_treat = np.random.normal(0, 3.0)<br>            y_treat = base_treat + global_trend + (true_ai_effect * is_post) + err_treat<br>            records.append({<br>                &#39;week&#39;: week,<br>                &#39;group&#39;: &#39;Treatment (Tier 1)&#39;,<br>                &#39;treat&#39;: 1,<br>                &#39;post&#39;: is_post,<br>                &#39;resolution_time&#39;: y_treat<br>            })<br>            <br>            # Control Group (Tier 2 - No AI Intervention throughout)<br>            err_control = np.random.normal(0, 3.0)<br>            y_control = base_control + global_trend + err_control<br>            records.append({<br>                &#39;week&#39;: week,<br>                &#39;group&#39;: &#39;Control (Tier 2)&#39;,<br>                &#39;treat&#39;: 0,<br>                &#39;post&#39;: is_post,<br>                &#39;resolution_time&#39;: y_control<br>            })<br>            <br>    return pd.DataFrame(records)<br><br><br>def fit_did_model(df: pd.DataFrame) -&gt; sm.regression.linear_model.RegressionResultsWrapper:<br>    &quot;&quot;&quot;Fits the Ordinary Least Squares (OLS) DiD model with interaction terms.&quot;&quot;&quot;<br>    # Statsmodels formula notation automatically handles main effects and the interaction (treat:post)<br>    formula = &quot;resolution_time ~ treat + post + treat:post&quot;<br>    model = smf.ols(formula=formula, data=df).fit()<br>    return model<br><br><br>def plot_did_trajectories(df: pd.DataFrame, intervention_week: int, save_path: str = None) -&gt; None:<br>    &quot;&quot;&quot;Plots weekly resolution time trajectories with pre/post intervention markers.&quot;&quot;&quot;<br>    weekly_avg = df.groupby([&#39;week&#39;, &#39;group&#39;])[&#39;resolution_time&#39;].mean().reset_index()<br>    <br>    plt.figure(figsize=(11, 6), dpi=300)<br>    <br>    colors = {&#39;Treatment (Tier 1)&#39;: &#39;#d62728&#39;, &#39;Control (Tier 2)&#39;: &#39;#1f77b4&#39;}<br>    <br>    for grp in weekly_avg[&#39;group&#39;].unique():<br>        subset = weekly_avg[weekly_avg[&#39;group&#39;] == grp]<br>        plt.plot(subset[&#39;week&#39;], subset[&#39;resolution_time&#39;], marker=&#39;o&#39;, <br>                 linewidth=2.2, label=grp, color=colors[grp])<br>        <br>    # Vertical marker for AI rollout<br>    plt.axvline(x=intervention_week - 0.5, color=&#39;#333333&#39;, linestyle=&#39;--&#39;, <br>                linewidth=1.8, label=&#39;GenAI Deployment&#39;)<br>    <br>    # Highlight pre-treatment window for parallel trend visual inspection<br>    plt.axvspan(1, intervention_week - 0.5, color=&#39;#e0e0e0&#39;, alpha=0.35, label=&#39;Pre-Rollout Window&#39;)<br>    <br>    plt.title(&#39;Difference-in-Differences: Ticket Resolution Time Trajectory&#39;, fontsize=13, fontweight=&#39;bold&#39;, pad=15)<br>    plt.xlabel(&#39;Week of Year&#39;, fontsize=11)<br>    plt.ylabel(&#39;Mean Resolution Time (Minutes)&#39;, fontsize=11)<br>    plt.xticks(range(1, df[&#39;week&#39;].max() + 1))<br>    plt.grid(alpha=0.3)<br>    plt.legend(loc=&#39;upper left&#39;, frameon=True)<br>    plt.tight_layout()<br>    <br>    if save_path:<br>        os.makedirs(os.path.dirname(save_path), exist_ok=True)<br>        plt.savefig(save_path)<br>        print(f&quot;Plot saved successfully to {save_path}&quot;)<br>        <br>    plt.show()<br><br><br>if __name__ == &quot;__main__&quot;:<br>    print(&quot;Generating synthetic support logs...&quot;)<br>    df_tickets = generate_did_data(n_weeks=16, intervention_week=9)<br>    <br>    print(&quot;Fitting OLS Difference-in-Differences Model...&quot;)<br>    model_fit = fit_did_model(df_tickets)<br>    <br>    print(&quot;\n&quot; + &quot;=&quot;*65)<br>    print(&quot;REGRESSION MODEL SUMMARY&quot;)<br>    print(&quot;=&quot;*65)<br>    print(model_fit.summary())<br>    <br>    plot_did_trajectories(df_tickets, intervention_week=9, save_path=&quot;assets/did_trajectories.png&quot;)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*om4_DKH0Vp_hhF_jB-qrGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5YGmT5wI8CfNmWoKyuHNQ.png" /></figure><p>Before looking at regression output, we must validate the core identification assumption of Difference-in-Differences: the parallel trends assumption.</p><p>Looking at the pre-rollout window (Weeks 1 through 8 in the trajectory plot below), Tier 1 (Treatment) and Tier 2 (Control) move in visual and structural lockstep. Even though Tier 1 handles longer baseline resolution times, both queues respond identically to week-to-week operational fluctuations prior to Week 9. This lockstep movement gives us strong statistical confidence that Tier 2 serves as a valid proxy for global background noise.</p><p>Once the GenAI bot goes live at Week 8.5 (the dashed line), Tier 1’s resolution time sharply decouples from Tier 2, dropping from over 24 minutes down to under 19 minutes, while Tier 2 continues its gradual upward background trend.</p><p>By running this regression in Python, we do not just get a point estimate of treat:post = -6.7550; we also get a <em>p_value </em>&lt; 0.000 and a <strong>95%</strong> Confidence Interval ([<strong>-7.058, -6.452</strong>]), allowing us to state with statistical certainty that the generative AI bot significantly improved queue performance.</p><h3>Key Executive Takeaways:</h3><ol><li><strong>Selection Bias Absorbed <em>treat </em>= +4.8803: </strong>The regression acknowledges that Tier 1 started <strong>4.8803 </strong>minutes slower than Tier 2 long before AI entered the picture, preventing baseline team differences from corrupting our causal estimate.</li><li><strong>Global Trend Isolated <em>post</em> = +2.5392):</strong> Company-wide friction pushed average handling times up by <strong>2.5392 </strong>minutes across all queues during Q2.</li><li><strong>Net Causal Impact Confirmed <em>treat:post </em>= -6.7550:</strong> After stripping away external noise and baseline gaps, the Generative AI routing bot drove a net reduction of <strong>6.7550 minutes per ticket</strong>.</li></ol><h3>Conclusion &amp; Operational Best Practices</h3><p>Evaluating technological investments in non-experimental settings requires moving beyond simple before-and-after averages. By leveraging Difference-in-Differences (DiD), data science teams can establish robust causal estimates even when randomized A/B testing is operationally impossible.</p><p>When applying DiD in your own support infrastructure, keep three core principles in mind:</p><ol><li>Validate Parallel Trends First: Always inspect pre-intervention trajectories to verify visual and statistical alignment between treatment and control queues.</li><li>Account for Group Heterogeneity: Ensure control groups experience similar macroeconomic, seasonal, or system-wide shocks as your treatment group.</li><li>Translate Metrics into Business Impact: Convert time reductions (<strong><em>treat:post </em>= -6.7550 mins</strong>) into direct capacity gains and cost savings per ticket for executive buy-in.</li></ol><blockquote><em>📂 Code &amp; Reproducibility:<br>The complete, fully reproducible Python pipeline, synthetic dataset generator, and visualization scripts are available on GitHub: </em><a href="https://github.com/lucas-nog-sousa/causal-inference-did-operations"><em>lucasnogsousa/causal-inference-did-operations</em></a><em>.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62500878c5d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handling Heavily Skewed Operational Metrics: Why Non-Parametric Tests and Bootstrapping Beat…]]></title>
            <link>https://medium.com/@lucasnogsousa/handling-heavily-skewed-operational-metrics-why-non-parametric-tests-and-bootstrapping-beat-4f63fb83fc5d?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f63fb83fc5d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[customer-experience]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[customer-success]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 22:19:00 GMT</pubDate>
            <atom:updated>2026-09-15T22:20:34.793Z</atom:updated>
            <content:encoded><![CDATA[<h3>Handling Heavily Skewed Operational Metrics: Why Non-Parametric Tests and Bootstrapping Beat Standard t-Tests</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t0e3tfHiM4Ivj_SpJYXdrQ.jpeg" /></figure><h3>The Trap of Operational Averages</h3><p>The operational analysis of customer support event logs contains persistent illusions. When executive leadership asks whether longer support interactions drive customer churn, the default response from many data teams is to pull call duration records, calculate the mean, and — when basic statistical knowledge is present — run a Student’s <em>t</em>-test. In practice, this approach fails immediately.</p><p>Support metrics — such as Handling Time (HT), First Response Time (FRT), and ticket transfer counts — rarely follow a Gaussian distribution where the mean is fully reliable. Instead, they exhibit heavy right-skewness: the vast majority of tickets are resolved within a few minutes, while a small fraction of complex, high-friction issues drag on for hours or days.</p><p>When you feed these long-tailed distributions into parametric tests that require data to follow a specific distribution (such as the normal curve), extreme values inflate the sample variance, distort the mean, and produce skewed <em>p</em>-values that mask whether a result is a mere coincidence or a real operational effect. You risk declaring a statistically significant difference where none exists or, worse, failing to detect a critical bottleneck that is quietly driving customers away — including high-value ones.</p><h3>Why the Student’s <em>t</em>-Test Fails on Operational Event Logs</h3><p>To understand why common parametric tests fail in this context, we must evaluate the core assumptions of the Student’s <em>t</em>-test against the structural reality of customer support data.</p><p>The <em>t</em>-test evaluates whether two population means are equal (μ_churn = μ_active) based on three primary assumptions:</p><ul><li><strong>Normality:</strong> Sample data are normally distributed.</li><li><strong>Homoscedasticity:</strong> Variances between the two groups are roughly equal.</li><li><strong>Independence:</strong> Observations are independent of one another.</li></ul><p>A common misconception among analysts is that the Central Limit Theorem (CLT) acts as a universal green light: <em>“If our sample size is large enough (e.g., N &gt; 1000), the distribution of the sample mean becomes normal; therefore, the t-test is valid.”</em> However, while the CLT asserts that the distribution of the <em>sample mean</em> approaches normality, it does not protect the test from extreme sensitivity to outliers.</p><p>The distribution of support ticket handling time is typically log-normal, characterized by sharp right-skewness. Although logarithmic transformations are often used to contract extreme values and expand smaller, denser ones — bringing them closer to normality — analyzing raw operational data in its original scale suffers a severe impact from anomalies. A single support ticket lasting 48 hours exerts a disproportionate influence on the sample variance. This inflates the standard error (which measures uncertainty in the mean estimate), reduces the statistical power of hypothesis tests, and elevates the Type II error rate (failing to detect a real effect).</p><p>Consequently, in business operations, the median reflects the typical experience of most customers far more reliably than the mean, which is severely distorted by outliers.</p><h3>Robust Alternatives in Python: Rank-Based Tests and Computational Bootstrapping</h3><p>To handle non-normal and skewed operational data correctly, we shift our framework from parametric hypothesis testing to non-parametric evaluations and computational bootstrap resampling — a technique that simulates collecting multiple new samples from the existing data to calculate error margins and complex statistics (like the median) with higher reliability.</p><p>Below is a complete implementation in Python, generating a realistic synthetic dataset of support handling times with right-skewness and evaluating the difference between active and churned customers using the <strong>Mann-Whitney U Test</strong> and <strong>Non-Parametric Bootstrapping</strong>.</p><pre>import numpy as np<br>import pandas as pd<br>from scipy import stats<br><br># 1. Simulate Synthetic Suport Operational Data (Log-Normal Distribution)<br>np.random.seed(42)<br>n_active, n_churn = 1500, 300<br><br># Handling time in minutes (Active vs. Churn)<br>active_ht = np.random.lognormal(mean=2.1, sigma=0.75, size=n_active) # Median ~8 min<br>churn_ht  = np.random.lognormal(mean=2.4, sigma=0.85, size=n_churn)  # Median ~11 min<br><br># 2. Exploratory Normality &amp; Skewness Diagnosis<br>print(f&quot;Active Group Skewness: {stats.skew(active_ht):.2f}&quot;)<br>print(f&quot;Churn Group Skewness:  {stats.skew(churn_ht):.2f}&quot;)<br><br># 3. Parametric vs. Non-Parametric Hypothesis Testing<br>t_stat, p_val_t = stats.ttest_ind(churn_ht, active_ht, equal_var=False)<br>u_stat, p_val_u = stats.mannwhitneyu(churn_ht, active_ht, alternative=&#39;two-sided&#39;)<br><br># Rank-Biserial Correlation (Effect Size for Mann-Whitney). In other words, <br># how much higher or lower the values ​​of one group tend to be in relation <br># to another on a scale of -1 to 1<br>n1, n2 = len(churn_ht), len(active_ht)<br>rank_biserial = 1 - (2 * u_stat) / (n1 * n2)<br><br>print(&quot;\n--- Test Comparison ---&quot;)<br>print(f&quot;Welch&#39;s t-test:     p-value = {p_val_t:.5f}&quot;)<br>print(f&quot;Mann-Whitney U:     p-value = {p_val_u:.5f}&quot;)<br>print(f&quot;Rank-Biserial Effect Size: {rank_biserial:.3f}&quot;)<br><br># 4. Non-Parametric Bootstrapping for Median Difference (95% CI)<br>def bootstrap_median_diff(data1, data2, n_boot=10000, ci=95):<br>    boot_diffs = np.empty(n_boot)<br>    for i in range(n_boot):<br>        sample1 = np.random.choice(data1, size=len(data1), replace=True)<br>        sample2 = np.random.choice(data2, size=len(data2), replace=True)<br>        boot_diffs[i] = np.median(sample1) - np.median(sample2)<br>    <br>    alpha = (100 - ci) / 2<br>    lower = np.percentile(boot_diffs, alpha)<br>    upper = np.percentile(boot_diffs, 100 - alpha)<br>    return np.median(data1) - np.median(data2), lower, upper<br><br>diff, ci_lower, ci_upper = bootstrap_median_diff(churn_ht, active_ht)<br>print(&quot;\n--- Bootstrap Results ---&quot;)<br>print(f&quot;Observed Median Difference: {diff:.2f} minutes&quot;)<br>print(f&quot;95% Confidence Interval:    [{ci_lower:.2f}, {ci_upper:.2f}] minutes&quot;)</pre><p>Execution Output:</p><pre>Active Group Skewness: 3.59<br>Churn Group Skewness:  3.89<br><br>--- Test Comparison ---<br>Welch&#39;s t-test:     p-value = 0.00000<br>Mann-Whitney U:     p-value = 0.00000<br>Rank-Biserial Effect Size: -0.217<br><br>--- Bootstrap Results ---<br>Observed Median Difference: 2.62 minutes<br>95% Confidence Interval:    [1.38, 3.91] minutes</pre><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br># Configuração estética do gráfico<br>sns.set_theme(style=&quot;whitegrid&quot;)<br>plt.figure(figsize=(10, 5), dpi=300)<br><br># 1. Plot do Histograma com curva KDE<br>ax = sns.histplot(active_ht, kde=True, bins=50, color=&quot;#2b5c8f&quot;, alpha=0.5, edgecolor=&quot;none&quot;)<br><br># 2. Cálculo da Média e Mediana<br>mean_val = np.mean(active_ht)<br>median_val = np.median(active_ht)<br><br># 3. Destaque das linhas de Média e Mediana<br>plt.axvline(median_val, color=&#39;#2ca02c&#39;, linestyle=&#39;--&#39;, linewidth=2.5, label=f&#39;Median: {median_val:.2f} min (Typical Customer)&#39;)<br>plt.axvline(mean_val, color=&#39;#d62728&#39;, linestyle=&#39;--&#39;, linewidth=2.5, label=f&#39;Mean: {mean_val:.2f} min (Distorted by Outliers)&#39;)<br><br># 4. Anotação explicativa da Cauda Longa (Skewness)<br>plt.annotate(&#39;Long Right Tail\n(Outliers / Extreme Tickets)&#39;, <br>             xy=(mean_val + 8, 30), <br>             xytext=(mean_val + 15, 60),<br>             arrowprops=dict(facecolor=&#39;black&#39;, shrink=0.05, width=1, headwidth=6),<br>             fontsize=10, fontweight=&#39;bold&#39;, color=&#39;#333333&#39;)<br><br># 5. Títulos e rótulos<br>plt.title(&#39;Right-Skewed Distribution of Support Handling Time&#39;, fontsize=14, fontweight=&#39;bold&#39;, pad=15)<br>plt.xlabel(&#39;Handling Time (Minutes)&#39;, fontsize=12)<br>plt.ylabel(&#39;Ticket Frequency&#39;, fontsize=12)<br>plt.xlim(0, 50)  # Limite para focar na densidade principal<br>plt.legend(fontsize=11, loc=&#39;upper right&#39;)<br><br>plt.tight_layout()<br>plt.savefig(&#39;right_skewed_handling_time.png&#39;, dpi=300)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AF5vQBPfSvBmQQGpSDBm9g.png" /><figcaption>Figure 1: Distribution of support handling time. Notice how the long right tail pulls the mean to the right of the median, demonstrating why parametric assumptions fail.</figcaption></figure><p>The complete, fully reproducible Python pipeline and visualization scripts are available on GitHub: <a href="https://github.com/lucas-nog-sousa/data-science-ai"><strong>data-science-ai Repository</strong>.</a></p><h3>Translating Statistical Hygiene into Business Impact</h3><p>By replacing the standard <em>t</em>-test with a non-parametric pipeline, we transform how operational findings are communicated to executive stakeholders.</p><p>Instead of telling leadership that <em>“the mean handling time differed with p &lt; 0.05,”</em> we present an actionable, business-focused diagnostic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jEwC4Lc3dWdfumw3Gbro4Q.png" /></figure><h3>Repository &amp; Reproducibility</h3><p>All synthetic datasets, diagnostic functions, and bootstrap simulation scripts used in this article are open-source and structured for production use.</p><p>You can clone, fork, or run the code directly from GitHub: 👉 <a href="https://github.com/lucas-nog-sousa/data-science-ai">GitHub: data-science-ai / skewed-operational-metrics-bootstrap</a></p><h3>Conclusion</h3><p>When analyzing support metrics, mathematical hygiene is not an academic luxury — it is a prerequisite for sound decision-making. Relying on standard parametric tests for heavily skewed operational logs introduces structural noise that distorts reality and risks misallocating engineering and support resources.</p><p>Before running your next hypothesis test on event logs, inspect the skewness of your variables. If the distribution displays a long right tail, step away from the <em>t</em>-test and utilize rank-based tests alongside computational bootstrapping to deliver robust, audit-proof insights to your business.</p><p>Whether you are auditing customer support event logs directly from platforms like Zendesk or optimizing internal IT ticketing workflows, replacing standard parametric assumptions with rank-based evaluation ensures your operational decisions are built on mathematical reality.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f63fb83fc5d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ultimate Cheat Sheet for ML Engineering on GCP]]></title>
            <link>https://medium.com/@lucasnogsousa/the-ultimate-cheat-sheet-for-ml-engineering-on-gcp-52c8c2d29ca4?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/52c8c2d29ca4</guid>
            <category><![CDATA[ml-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gcp-certification]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Thu, 29 Feb 2024 18:28:49 GMT</pubDate>
            <atom:updated>2024-02-29T18:30:06.683Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction</strong></p><p>Hello! ML Engineering is the discipline that focuses on developing and implementing Machine Learning (ML) systems at scale. It encompasses the creation of robust, efficient, and scalable ML pipelines that can be used to generate business value from data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OATqPLHnZQJtV-cEAR6iYg.png" /></figure><p>In this article, we will explore the importance of ML Engineering and how to implement it using the Google Cloud Platform (GCP) platform. We will cover the following topics:</p><ul><li>Why is ML Engineering important?</li><li>Common challenges in ML Engineering</li><li>GCP as a platform for ML Engineering</li><li>Implementing an ML project on GCP</li><li>Considerations and best practices</li><li>Conclusion</li></ul><p><strong>Why is ML Engineering important?</strong></p><p>ML Engineering is crucial for the success of any ML project. It ensures that ML models are:</p><ul><li><strong>Scalable:</strong> able to handle large volumes of data as needed. This can be achieved through horizontal scaling (adding more resources) and/or vertical scaling (upgrading existing resources).</li><li><strong>Efficient:</strong> optimized for resource utilization (compute and storage) for performance and cost reduction, which can be achieved through techniques such as caching, query optimization, and parallelization.</li><li><strong>Robust:</strong> tolerant to errors and failures, ensuring continuity of system actions. In this sense, reliability is sought through redundancy, failover, and monitoring mechanisms.</li><li><strong>Reproducible:</strong> capable of being re-executed with the same results.</li><li><strong>Manageable:</strong> easy to monitor and manage. Data management tools, such as BigQuery, which is used for analytical query enhancement, and Looker Studio, which transforms data into reports and dashboards, provide the ability to apply data governance solutions.</li><li><strong>Secure:</strong> protected against unauthorized access and cyberattacks with techniques such as authentication, authorization, and encryption.</li></ul><p><strong>Common challenges in ML Engineering</strong></p><p>Implementing ML projects can be challenging. Some very common challenges include:</p><ul><li><strong>Data management:</strong> collecting, preparing, and storing data for ML. This is the stage where Exploratory Data Analysis (EDA) takes place, which is responsible for discovering essential information and a deeper understanding of the data provided.</li><li><strong>Model selection:</strong> choosing the right ML model for the specific problem. This is done by understanding the problem to be solved and by testing the application of models for the subsequent choice of the one that presents the best results.</li><li><strong>Model training:</strong> optimizing the model to improve performance and achieve the desired results.</li><li><strong>Model deployment:</strong> deploying the model in production.</li><li><strong>Model monitoring:</strong> tracking model performance over time.</li></ul><p><strong>GCP as a platform for ML Engineering</strong></p><p>GCP offers a wide range of services that can be used to implement end-to-end ML projects. Some of the most popular services include:</p><ul><li><strong>Cloud Storage:</strong> data storage at scale for persisting data from the source that will be used as well as data produced by ML models, acting as a Data Lake.</li><li><strong>BigQuery:</strong> building a data warehouse or data lakehouse, which is more recent and powerful and encompasses the DW in its structure, and are cloud services for large-scale data analysis.</li><li><strong>Cloud Dataflow:</strong> batch and stream data processing service for ML pipelines. I note that Airflow presents itself as a more viable solution in the face of possible migrations between cloud services, since it is supported on other platforms such as Azure and AWS, which is not the case with Dataflow, which belongs to GCP.</li><li><strong>Cloud ML Engine:</strong> ML model training and prediction service. Python and Tensorflow libraries can be used in this service.</li><li><strong>AI Platform:</strong> unified platform for ML lifecycle management. It was developed to facilitate the development of ML models.</li></ul><p><strong>Implementing an ML project on GCP</strong></p><p>Implementing an ML project on GCP can be divided into several steps:</p><ul><li><strong>Define the problem:</strong> identify the problem you want to solve with ML. Many projects in this area fail because they do not bring specific solutions to the problems faced at the time, often being postulates of generic, but ineffective solutions.</li><li><strong>Collect and prepare the data:</strong> gather the data needed to train the model. In EDA, tasks such as attribute engineering facilitate the selection of more useful variables, which makes it easier to understand the information generated.</li><li><strong>Select and train the model:</strong> choose the right ML model and train it on the data.</li><li><strong>Evaluate the model:</strong> evaluate the model’s performance on a test data set. The selection of the appropriate metrics will depend on the type of problem that the models are intended to solve, with classification, regression, and clustering being the most common types.</li><li><strong>Deploy the model:</strong> deploy the model to production on GCP.</li><li><strong>Monitor the model:</strong> track model performance over time.</li><li><strong>CI/CD:</strong> Continuous Integration and Continuous Delivery can be used in the implementation of ML models, as it is an application development practice that aims to automate and streamline the delivery process by continuous cooperation between development teams in building, testing, and deploying ML applications.</li></ul><p><strong>Considerations and best practices</strong></p><p>When implementing ML projects on GCP, it is important to consider the following best practices:</p><ul><li><strong>Use the right tool for the job:</strong> choose the GCP services that best meet your needs, aiming to logically reduce costs and increase benefits.</li><li><strong>Optimize for performance and cost reduction:</strong> configure your ML pipelines to be efficient and economical. Creating custom and well-compiled codes will contribute to this goal, although they require some expertise.</li><li><strong>Ensure reproducibility:</strong> document your ML process so that it can be reproduced. Reusing code saves time.</li><li><strong>Monitor your models:</strong> track the performance of your models over time, as there may be a need to adjust parameters given the stochastic behavior common in real data.</li><li><strong>Be agile:</strong> be prepared to adjust your ML process as you learn more about your data and your model.</li></ul><p><strong>Conclusion</strong></p><p>ML Engineering is an essential discipline for the success of any ML project. GCP offers a powerful and flexible platform for implementing end-to-end ML projects. By following best practices and using the right tools, you can create ML systems that generate business value from data.</p><p><strong>And there you have it, what did you think of the presentation? What is your opinion on the topic covered? Be sure to leave your opinion in the comments. Follow me for more content like this. See you next time! Hugs!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52c8c2d29ca4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ChatGPT: The Game-Changing Automation Tool for AI and Machine Learning Professionals]]></title>
            <link>https://medium.com/@lucasnogsousa/chatgpt-the-game-changing-automation-tool-for-ai-and-machine-learning-professionals-d90c921d2906?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d90c921d2906</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[professional]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Mon, 17 Apr 2023 17:36:33 GMT</pubDate>
            <atom:updated>2023-04-17T17:44:04.055Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0J35cuXNhV5rQ4ouvox82A.jpeg" /><figcaption>Source: Pexels</figcaption></figure><p>Automation is the process of using technology to automate recurring tasks, increasing the accuracy and efficiency of activity execution. An example of this is the use of machine learning models to predict a company’s sales volume in the coming months.</p><p>One of the tools that has gained prominence and can be used in process automation is ChatGPT. ChatGPT is an automation tool that can assists us in creating applications and machine learning models, saving time and effort in writing code. It makes it more accessible to create complex activities, even for users with little advanced technical knowledge in programming. ChatGPT is an important ally in the automation of machine learning processes, contributing to increased efficiency and accuracy in activity execution.</p><p>However, as automated processes involving machine learning become more common, there may be a need to redefine the skills required for artificial intelligence and machine learning professionals, including managing automation tools.</p><h3>The Role of ChatGPT as an Automation Tool</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZrV-jU_l05bYIAwXuHUHHg.jpeg" /><figcaption>Source: Pexels</figcaption></figure><p>One of the most notable automation tools is ChatGPT, which has made the knowledge necessary to develop complex activities more accessible and democratic. ChatGPT can assist developers in accelerating the creation of applications and saving time and effort in writing code for machine learning models. Although coding is a relevant skill, ChatGPT has also made it easier for non-technical users with little coding knowledge to create software.</p><h3>Risks for IT Professionals</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CdLhop5ViKRBzYRbqPYPhg.jpeg" /><figcaption>Source: Pexels</figcaption></figure><p>There are discussions about the risks that automated processes present to professionals in various fields, including the IT area. While automated processes can offer more time to improve skills and acquire new abilities, it is necessary to consider that in some cases, IT professionals are already tasked with performing tasks complementary to those developed by automation. However, the supervision of automated processes is something that only humans can perform.</p><h3>Conclusion</h3><p>The scenario of process automation demands that professionals in artificial intelligence and machine learning continue to refine skills such as machine learning, data analysis, Python, R, SQL, and statistics, as well as develop new skills such as creativity, innovation, and curiosity, in order to fully leverage the potential of automation tools in problem-solving. Therefore, IT professionals need to adapt to the new automation landscape to stay ahead and find ways to channel their efforts towards generating ideas rather than focusing solely on code.</p><p><strong><em>That’s it! I created this article just to encourage you to seek the expansion of the skills you already have, as well as the acquisition of others that are much needed in this scenario of process automation. Don’t forget to share, comment, suggest, and follow me </em></strong><a href="https://medium.com/@lucasnogsousa"><strong><em>here</em></strong></a><strong><em> and on my social networks </em></strong><a href="https://twitter.com/lucasnogsousa"><strong><em>twitter</em></strong></a><strong><em> and </em></strong><a href="https://gettr.com/user/lucasnogsousa"><strong><em>gettr</em></strong></a><strong><em>! Cheers!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d90c921d2906" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing Production Line Errors: Hypothesis Testing and Data Visualization]]></title>
            <link>https://medium.com/@lucasnogsousa/analyzing-production-line-errors-hypothesis-testing-and-data-visualization-d4c7637ea255?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4c7637ea255</guid>
            <category><![CDATA[normal-distribution]]></category>
            <category><![CDATA[production-line]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[nonparametric-test]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Mon, 10 Apr 2023 14:27:05 GMT</pubDate>
            <atom:updated>2023-04-10T14:27:05.226Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>A production line is a system used in the manufacturing of products in sequential stages so that only after the completion of the current activity in one stage will there be a migration to the next and so on until the completion of the work. For example, in the production process of a car, at a station or stage within this context, the bodywork is developed, followed by the steps of placing the wheels and windows, and finally the bodywork is painted.</p><p>As this process is complex and involves several stages, errors may occur. These errors can be more or less prolonged in a given station. In this scenario, it is necessary to analyze the errors to identify if the stop time at a given station is significantly longer than in others and to take measures to correct them in less time.</p><p>One way to identify if the time elapsed in the stations due to errors is significantly different between them is by using a hypothesis test.</p><p>The hypothesis test is a method widely used in data science to confirm a hypothesis. In the circumstance presented above, it would be used to confirm whether the time spent at a given station is superior to the time elapsed in another and whether this difference in time is significant, not being an occasional oscillation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*VaOaNQoo8G675mcGDv9tKg.jpeg" /><figcaption>Source:Pixabay</figcaption></figure><h3>Learning Objective:</h3><p>In this article, we will learn:</p><ol><li>How to sum the time in minutes in a database with a month of records grouping it by error date and station description.</li><li>How to perform a hypothesis test to verify if the median time in which cars in production pass being adjusted due to errors varies significantly from one station to another.</li><li>How to identify which method to use, whether parametric or non-parametric, in the hypothesis test.</li><li>How to visualize data to draw conclusions and make decisions based on test results.</li></ol><h3>Table of Content</h3><ol><li>Importing the Necessary Libraries and Loading the Data</li><li>Data Preprocessing</li><li>Testing Normality and Homoscedasticity</li><li>Non-parametric Hypothesis Testing using Friedman Test</li><li>Post-hoc Analysis</li><li>Data Visualization</li><li>Conclusion</li></ol><h4>Importing the Necessary Libraries and Loading the Data</h4><p>We will start by importing the necessary libraries, followed by the creation of a dataset of errors in car production.</p><p>We will create our dataset with dummy data, using the numpy library and pandas. The dataset will have five fields, namely, the field “error date”, “car body”, “wheels”, “window,” and “painting”, these last four represent the description of the stations through which the cars follow in the line of production, and they count the time in minutes in which the vehicles were stopped due to errors.</p><pre># Importing libraries<br>import numpy as np<br>import statistics<br>from scipy import stats<br>import pandas as pd<br>import scikit_posthocs as sp<br><br># Define the date range<br>start_dt = &#39;2023-01-01&#39;<br>end_dt = &#39;2023-01-31&#39;<br><br># Define the station categories<br>station_categories = [&#39;car body&#39;, &#39;wheels&#39;, &#39;window&#39;, &#39;painting&#39;]<br><br># Define the number of rows<br>n_rows = 60<br><br># Generate the dates<br>dates = pd.date_range(start_dt, end_dt, periods=n_rows)<br><br># set random seed for reproducibility<br>np.random.seed(23)<br><br># create four columns with 60 values each<br>col1 = np.random.normal(loc=5, scale=1, size=n_rows)<br>col2 = np.random.normal(loc=5, scale=1, size=n_rows)<br>col3 = np.random.exponential(scale=7, size=n_rows)<br>col4 = np.random.exponential(scale=15, size=n_rows)<br>#col4 = np.random.gamma(shape=7, scale=2, size=num_rows)<br><br># Create the DataFrame<br>df = pd.DataFrame(np.column_stack((col1, col2, col3, col4)), columns=station_categories)<br><br># Create &quot;error date&quot; column<br>df[&#39;error date&#39;] = dates</pre><p>Clarifying a little more about the columns of the stations, the goal is to investigate if the time spent at a given station is significantly higher compared to others.</p><p>The “car body” and “wheels” columns will be created with the same mean and standard deviation, which are represented by the loc and scale parameters of the “numpy.random.normal()” function.</p><p>The “window” column will be created using the “np.random.exponential()” function, which is used to create a distribution of average error time of 7 minutes each day, making it relatively close in mean time to the first two columns.</p><p>Finally, the “painting” column will have a mean error time of 15 minutes, which will distance it from the other means of the other distributions.</p><pre># Estimating the probability density function (PDF) of the variables distribution.<br>df.plot.kde(figsize=(10,5))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*4pRT0Zb70PSRlH3GQFyLpw.jpeg" /><figcaption>Group Distribution</figcaption></figure><h4>Data Preprocessing</h4><p>We will need to convert the “error date” column from datetime to date. As our dataset will be grouped by date, so we’ll need to sum the time in minutes for each step.</p><p>Thus, we will have a view of the amount of time in minutes per day that the cars spent at the stations. In addition, we will remove columns or rows if they have null values, as we intend to work with paired data, which is when the samples under study have the same size.</p><pre># Convert &quot;error date column&quot; from datetime to date<br>df[&#39;error date&#39;] = pd.to_datetime(df[&#39;error date&#39;],&#39;%Y-%m-%d&#39;).dt.date<br><br># Set &quot;error date&quot; column as index<br>df = df.groupby(&#39;error date&#39;).sum()<br><br># Remove columns that have more than 3 null values. The idea is to leave the data paired.<br>df = df.dropna(axis=1,thresh=len(df)-3)<br><br># Remove all rows with null values.<br>df = df.dropna(axis=0)<br><br># Show the first 5 columns<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Y2vN92J6PR16h0wmB1xJSQ.jpeg" /><figcaption>Head</figcaption></figure><h4>Testing Normality and Homoscedasticity</h4><p>Now, we will carry out two tests to know whether we should apply the parametric or non-parametric method, namely, the normality test and the homoscedasticity test of the data groups, which are the fields of the table.</p><p>To apply the parametric method, both hypotheses need to be true. Preliminarily, we will test whether the groups follow a normal distribution using the Shapiro-Wilk test. It is important to note that all groups must follow a normal distribution for parametric methods to be applied.</p><p>The normal distribution is characterized by symmetry around the mean and by the graphical representation of a bell-shaped curve. Then, we will perform Levene’s test to verify homoscedasticity in the variance between groups. Homoscedasticity is used to verify whether the variability between the compared groups is the same.</p><pre># normality assumption using the Shapiro-Wilk test<br>for d in df.columns:<br>    test_stat_normality, p_value_normality=stats.shapiro(df[d].values)<br>    print(f&quot;p value {d}:%.4f&quot; % p_value_normality)<br>    if p_value_normality &lt;0.05:<br>        print(f&quot;Reject null hypothesis for {d} &gt;&gt; The data is not normally distributed&quot;)<br>    else:<br>        print(f&quot;Fail to reject null hypothesis {d} &gt;&gt; The data is normally distributed&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*c5VbZpcy_pCbBSblUWpWUg.jpeg" /><figcaption>Normality Test</figcaption></figure><pre># I&#39;ll check the variance between the data (homoscedasticity), but I already know that they are not normally distributed. Therefore, it is non-parametric.<br>stat, pvalue_levene = stats.levene(col1,col2,col3,col4)<br>print(&quot;Levene test p-value: %.4f&quot; % pvalue_levene)<br>if pvalue_levene &lt; 0.05:<br>    print(&quot;Reject Null Hypothesis &gt;&gt; The variance of the columns is different.&quot;)<br>else:<br>    print(&quot;Do not reject Null Hypothesis &gt;&gt; The variance of the columns is the same.&quot;)<br>    <br># The normality test and homoscedasticity test was violated.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*yqtTTEnapIAGoYENlf0nAQ.jpeg" /><figcaption>Levene Test</figcaption></figure><h4>Non-parametric Hypothesis Testing using Friedman Test</h4><p>As we can see above, the assumption of normality and homoscedasticity was violated. Therefore, we will perform the Friedman test. It is a non-parametric test that we will use to investigate if there are significant differences between the medians of the samples in the dataset. This is equivalent to saying that we will analyze the dependence between them.</p><p>Since the p-value, which is used in this case to check whether there is or is not a difference in the median between at least two groups, presented a value lower than the significance level, which is 0.05, we will reject the null hypothesis, which indicates that at least one of the groups has a median greater than the other samples.</p><pre># I will use the Friedman test, which is a non-parametric version of the anova test.<br>test_stat,p_value = stats.friedmanchisquare(col1,col2,col3,col4)<br>print(&quot;p value:%.4f&quot; % p_value)<br>if p_value &lt; 0.05:<br>    print(&quot;The null hypothesis is rejected, indicating that at least one of the medians is significantly greater than the others.&quot;)<br>else:<br>    print(&quot;The null hypothesis is not rejected. The medians are the same.&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*nzwCf6KsWYJ-gArDahppyQ.jpeg" /><figcaption>Friedman Test</figcaption></figure><p>But since we have more than two samples, what do we do to identify which ones are the largest? To know this, we will use the post-hoc analysis.</p><h4>Post-hoc Analysis</h4><p>The post-hoc analysis will be used to identify which groups have significant differences in the median. It is important to note that it does not indicate which sample is larger or smaller, but only if the difference between them is relevant.</p><pre># I will use Post-hoc analysis to find out which group(s) caused the difference in the median.<br>posthoc_df = sp.posthoc_nemenyi_friedman(df.values) <br>group_names= df.columns<br>posthoc_df.columns= group_names<br>posthoc_df.index= group_names<br>posthoc_df.style.applymap(lambda x: &quot;background-color:violet&quot; if x&lt;0.05 else &quot;background-color: white&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*GNkow6NVH7GS6XQKXOvvOg.jpeg" /><figcaption>Post-hoc Analysis</figcaption></figure><h4>Data Visualization</h4><p>As can be seen from the Post-hoc graph above, the time at the “painting” station due to error is significantly different than at the others. However, to know if it is significantly higher or lower, we need to plot the median of the steps to verify this.</p><pre># Add the line plot for the stage columns<br>ax = df.median().round(2).plot(marker=&#39;.&#39;, ylabel=&#39;Stage&#39;)<br><br># Annotate the line by iterating through each row<br>for i,v in enumerate(df.median().round(2)):<br>    ax.annotate(text=v, xy=(i, v))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*Jdwrzv9cNVYU47O5I1eROA.jpeg" /><figcaption>Visualization</figcaption></figure><p>As we can see, stage “painting” is significantly larger than the others. Thus, we can conclude that some action should be taken to try to reduce errors at station “painting”.</p><h3>Conclusion</h3><p>The elapsed time that a product, such as a car, spends in different stages on a production line, can be longer than acceptable when compared by stages. Hypothesis testing can be a useful tool in diagnosing the excess time that errors take at each station, enabling corrective actions to be taken.</p><p>In the example in this article, we saw how to import the necessary libraries in Python and create a fictitious dataset. In addition, we performed the normality and homoscedasticity tests to verify if we should apply parametric tests in identifying differences between the median of the data groups, but we concluded that these assumptions were violated, leading us to use the non-parametric Friedman test. We performed the post-hoc test that showed which groups have different medians, and we plotted the median of the data to visualize which groups had higher medians.</p><p>Three key takeaways from this article are:</p><ol><li>Hypothesis testing can help us identify if products are spending more time in one stage than in others, which will aid in corrective decision-making.</li><li>Non-parametric hypothesis testing can be used in situations where the assumption of normality and/or homoscedasticity are violated.</li><li>Post-hoc analysis can help us identify groups with differences in their medians, but we will need a visual analysis to identify which groups have higher, significantly speaking, medians.</li></ol><p>That’s all! Don’t forget to follow me <a href="https://medium.com/@lucasnogsousa">here</a> and on my social network <a href="https://twitter.com/lucasnogsousa">Twitter</a> and <a href="https://gettr.com/user/lucasnogsousa">Gettr</a> so I’ll know content like this is relevant for you! Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4c7637ea255" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing the Power of Digital Marketing Metrics with Python and Power BI]]></title>
            <link>https://medium.com/@lucasnogsousa/unleashing-the-power-of-digital-marketing-metrics-with-python-and-power-bi-423fb4ed93b6?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/423fb4ed93b6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[business-strategy]]></category>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[power-bi]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Sat, 01 Apr 2023 04:03:04 GMT</pubDate>
            <atom:updated>2023-04-01T04:03:04.573Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6YAcMj_bsjGAW4JWjXtjOw.jpeg" /><figcaption>Source: Pexels</figcaption></figure><p>Digital marketing is a set of strategies used to promote companies or brands online, to attract potential customers, and engage them. Nowadays, digital marketing is indispensable for any business that aims for success. This success can and should be measured. However, measuring the effectiveness of digital marketing campaigns is not an easy task. And this is where we need to use metrics, which are measures of values or quantities, and indicators, which are calculated measures, that can be presented in percentage form, for digital marketing.</p><p>In this article, we will explore some very commonly used metrics and indicators of digital marketing in the optimization of campaigns, using a fictional dataset created with python code from the pandas and numpy libraries. We will also perform a visual analysis of the metrics and indicators using Power BI and discuss the metrics and indicators created, demonstrating how they can be useful for companies in their decision-making processes, aiming to achieve success.</p><h3>Importing Libraries</h3><p>Firstly, we need to import the necessary libraries for the task.</p><pre># Importing the necessary libraries<br>import pandas as pd<br>import numpy as np</pre><h3>Creating a Fictional Dataset with Python</h3><p>Next, we create a fictional dataset using python code. The dataset will be composed of 1,000 records, containing four columns, two of which are metrics, “website_traffic” and “social_media_engagement”, and two are indicators, “click_through_rate” and “conversion_rate”. These columns are commonly used in measuring the effectiveness of digital marketing campaigns. As we will use Power BI for visual data analysis, we will need to save the file so that we can read it in Power BI Desktop. To do this, we will use the pandas function to_csv().</p><pre># Define the size of the dataset<br>n_records = 12<br><br># Define the start of date range<br>start_date = &#39;2022-01-01&#39;<br><br># Generate random dates<br>dates = pd.date_range(start=start_date, periods=n_records, freq=&#39;M&#39;)<br><br># Generate website traffic<br>website_traffic = np.random.randint(low=1000, high=5000, size=n_records)<br><br># Generate click-through rates<br>click_through_rate = np.random.uniform(low=0.05, high=0.1, size=n_records)<br><br># Generate conversion rates<br>conversion_rate = np.random.uniform(low=0.01, high=0.05, size=n_records)<br><br># Generate social media engagement<br>social_media_engagement = np.random.randint(low=10, high=500, size=n_records)<br><br># Create the dataframe<br>df = pd.DataFrame({<br>    &#39;date&#39;: dates,<br>    &#39;website_traffic&#39;: website_traffic,<br>    &#39;click_through_rate&#39;: click_through_rate,<br>    &#39;conversion_rate&#39;: conversion_rate,<br>    &#39;social_media_engagement&#39;: social_media_engagement<br>})<br><br># Save into .csv file<br>df.to_csv(&quot;Metrics_Indicators_Digital_Marketing.csv&quot;)<br><br># First few rows of the dataframe<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*yM-lGPZ8HLK9LRSlad53Ew.png" /></figure><h3>Exploring Metrics and Indicators of Digital Marketing</h3><p>After creating the dataset, we will analyze the metrics and indicators using the Power BI desktop, which is a powerful free data visualization tool in the Desktop version. To do this, we will need to open the program and import the file we created.</p><h3>Data Preprocessing</h3><p>Once the file has been imported into Power BI, we will need to go to the “Transform Data” option, convert the “click_through_rate” and “conversion_rate” columns to text, replace the dots in them with commas, and then change their type to percentage. This is necessary because Python inserts periods in percentage numbers, and Power BI uses commas in this scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*KUTKYezqAnM7SWE5RQgdjw.png" /></figure><p>After that, we will perform the analysis of the metrics and indicators contained in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*5ZP-IlBOYSKEtlekjIaXeQ.png" /></figure><p>By viewing the dashboard above, we can obtain important insights that will be described below by metric and/or indicator:</p><ol><li><strong>Website Traffic:</strong> Website Traffic describes the number of people who visited a website. Through it, companies can identify which marketing channel is most efficient in directing more traffic to the website, enabling them to optimize campaigns accordingly. We can also identify the months, for example, when website traffic is higher. As we could see in the dashboard, the website traffic was higher in January, April, July, and November.</li><li><strong>Click-through Rate (CTR):</strong> CTR is the percentage of people who click on links or ads on the website. It measures how effective the advertising campaigns were in attracting the attention of customers, giving us the tools to create ads more suited to the interests of the target audience. The analysis of CTR over a period can be useful in analyzing the reach of a given campaign, since after sending one of them, we can contemplate both a significant increase in CTR and an extension of the periods in which there were high click rates, which is a strong indication of the effectiveness of this ad. The CTR of our dataset, as we can see from the dashboard, was higher in March, October, and November.</li><li><strong>Conversion Rate:</strong> Conversion rate is the percentage of the target audience that responds to the stimulus of campaigns and generates results for the business by making purchases, for example. By analyzing the conversion rate, companies can identify the most profitable campaigns, as well as the most productive periods, which will allow them to better direct their marketing campaign actions. According to our dashboard, the best conversion rates occurred in April, August, November, and December.</li><li><strong>Social Media Engagement:</strong> Social Media Engagement is the desired number of likes, comments, and shares that a post has received. This metric is useful in indicating customer sentiment toward marketing campaigns, and guiding businesses on the need for maintenance or changes in strategies. For example, a given period in which a post received more likes indicates a positive sentiment from the post’s audience. According to the dashboard, we had the highest engagement in February, May, July, and August.</li></ol><h3>Conclusion</h3><p>As can be seen, digital marketing has become indispensable to stimulate business growth. Growth must be measured through metrics and indicators that inform us of the path to be followed to achieve success. By analyzing metrics and indicators such as website traffic, click-through rates, conversion rates, and social media engagement, companies can identify which marketing campaigns are more effective in increasing traffic, clicks, and conversions, as well as in satisfying the target audience, enabling the optimization of campaigns. We can use data analysis and visual analysis tools such as Python and Power BI to obtain valuable insights that will drive increased effectiveness in digital marketing campaigns and improvement in decision-making.</p><p>That’s all! Please consider giving me a follow <a href="https://medium.com/@lucasnogsousa">here</a> and on my social networks <a href="https://twitter.com/lucasnogsousa">Twitter</a> and <a href="https://gettr.com/user/lucasnogsousa">Gettr</a>, leaving a comment, and subscribing to my content. Your support is greatly appreciated!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=423fb4ed93b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Classification into Topics with Naive Bayes Classifier: A Powerful Approach]]></title>
            <link>https://medium.com/@lucasnogsousa/text-classification-into-topics-with-naive-bayes-classifier-a-powerful-approach-f74851b6b0a1?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f74851b6b0a1</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[text-classification]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Mon, 27 Mar 2023 19:51:31 GMT</pubDate>
            <atom:updated>2023-03-27T19:51:31.430Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Text classification is a task that involves labeling texts from, for example, news articles or blogs. It is widely used by professionals in the journalistic field to compile texts on the same topic. It is a very promising activity, given that there is a constant growth in the volume of data produced that needs to be organized for analysis. It is done using classification algorithms such as the Naive Bayes Classifier.</p><p>The Naive Bayes Classifier is a supervised learning algorithm, in which the data has a labeled response variable. It has a high capacity for making good predictions even in datasets with few or many profiles. Another good characteristic of this algorithm is that it assumes that each predictor variable is conditionally independent of the others, hence the name Naive, which means, among other things, that it does not have problems with multicollinearity, which is when independent variables are highly correlated, making it difficult to separate the effects produced by each independent variable. This situation is quite common in regression problems. However, the assumption of independence of the predictor variables may not be true for all types of texts, which could lead to losses of effectiveness in text classifications.</p><p>In this article, we will use Naive Bayes Classifier to perform a task of text classification into topics, which is when we categorize them into topics such as health, religion, politics, and technology, among others. Although there are other more modern and robust approaches for performing this task, using this algorithm combined with TF-IDF, which is a form of text vectorization, is still a common practice, even because good results with low temporal latency and the use of hardware resources can be obtained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8S3sq3QB44dBb_PJQEOnTg.jpeg" /><figcaption>Source: Pexels</figcaption></figure><h3>Learning Objectives:</h3><p>In this article, we will</p><ol><li>Use the Naive Bayes Classifier, making use of the TF-IDF library to classify text into topics.</li><li>Create a model using the TfidfVectorizer and GaussianNB library.</li><li>Analyze the results of the accuracy metrics and f1 score to evaluate the performance of our classification model and determine if any adjustments need to be made.</li><li>Predict the categories of some texts after training the model.</li></ol><h3>Table of Contents</h3><ol><li>Text Classification into Topics</li><li>How does Naive Bayes work?</li><li>Importing Libraries</li><li>Loading the Dataset</li><li>Data Preprocessing</li><li>Model Creation and Evaluation Metrics</li><li>Prediction of Text Categories</li><li>Conclusion</li></ol><h3>Text Classification into Topics</h3><p>The objective of text classification is to classify documents or texts into topics. A well-known example is the classification of email texts as spam or not spam. In our project, we will classify texts according to topics. We will use 4 topics related to health, technology, religion, and politics.</p><p>The Naive Bayes Classifier algorithm does not deal with textual values, but with numerical values. Therefore, we will create a text vectorization process, which is the assignment of weights to the words of sentences. We will do this by dividing the texts into lists of words, by counting the words in the documents, which will generate a matrix of terms from the documents, and by assigning weights to them.</p><p>For this, we will use the TF-IDF (term frequency-inverse document frequency) method, which assigns weights to words based on the ratio of the number of times a word appears in a given document to the number of words in that document multiplied by the logarithmic ratio of the total number of documents to the documents in which that word appears at least once. I note that the more documents a given word appear in, the less relevant it becomes, as it will not be very useful in discriminating between the documents since it is present in many of them. In Python, we have the TfidfVectorizer function, through which we can apply the TF-IDF method mentioned above.</p><h3>How does Naive Bayes work?</h3><p>Naive Bayes relies heavily on conditional probability, for example, it can answer the following question: what is the probability of event A occurring given that event B occurred? For instance, let’s say we have 21 points in a sample space, with 13 blue, 7 red, and 1 whose color is unknown. We know this unknown ball is close to 3 blue balls and 1 red ball. In this context, the probability of it being blue is P(x is blue) = P(blue) * P(x | blue) = 13/20 * 3/13 = 0.65, and the probability of it being red is P(x is red) = P(red) * P(x | red) = 7/20 * 1/7 = 0.05. Therefore, we can conclude that it is more likely that the unknown point is blue. The great advantage of using Naive Bayes is that it is easy to understand, not sensitive to irrelevant profiles, and its predictor profiles are independent of each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*gXE84m1B4XoBqSZyebiXkg.png" /></figure><h3>Importing Libraries</h3><p>Firstly, we will import the necessary libraries for the proposed activity, as presented below:</p><pre># Importing Libraries<br>from sklearn.datasets import fetch_20newsgroups<br>from pprint import pprint<br>import pandas as pd<br>from sklearn.feature_extraction.text import  TfidfVectorizer<br>import numpy as np<br>from sklearn.naive_bayes import GaussianNB<br>from sklearn import metrics</pre><h3>Loading the Dataset</h3><p>We will use the sklearn.datasets library to load the fetch_20newsgroups dataset, which is the 20 newsgroups text dataset. This dataset contains about 18,000 newsgroup documents divided into 20 different topics with two subsets: one for training and one for testing. Before loading the data, we will select the categories we will work within this project to fetch only the related data: “sci.med”, “comp.graphics”, “soc.religion.christian”, and “talk.politics.misc”. This way, we will avoid loading all records.</p><p>Next, we will retrieve the data and save it into two datasets: the training set and the test set. Our dataset will have two fields: the text field, where we locate the sentences related to each topic, and the category field, which represents the topic.</p><pre># Selected categories <br>categories = [&#39;sci.med&#39;, &#39;comp.graphics&#39;, &#39;soc.religion.christian&#39;, &#39;talk.politics.misc&#39;]<br><br># Loading training dataset<br>df_train = fetch_20newsgroups(subset=&#39;train&#39;,categories=categories, shuffle=True, random_state=3)<br><br># Loading testing dataset<br>df_test = fetch_20newsgroups(subset=&#39;test&#39;,categories=categories)</pre><h3>Data Preprocessing</h3><p>In this stage, we will create text vectorization, using the TfidfVectorizer function, which will assign weights to the words, representing the relevance of each of them.</p><pre># Text vectorization<br>tfidf_vector = TfidfVectorizer()<br>train_tfidf = tfidf_vector.fit_transform(df_train.data)<br>test_tfidf = tfidf_vector.transform(df_test.data)</pre><h3>Model Creation and Evaluation Metrics</h3><p>Now that we have the processed data, we can create the model, train it, test it, and analyze it using some metrics such as accuracy and f1 score. Below, we provide the code we will use for that:</p><pre># Model creation<br>model = GaussianNB().fit(train_tfidf.toarray(),df_train.target)<br><br># Evaluation metrics<br>pred = model.predict(test_tfidf.toarray())<br>print(&quot;F1 Score&quot;,metrics.f1_score(df_test.target, pred, average=&#39;macro&#39;))<br>print(&quot;Accuracy&quot;,metrics.accuracy_score(df_test.target, pred))<br><br># Metrics show that the model&#39;s predictive ability is not bad<br>F1 Score 0.8616575143527554<br>Accuracy 0.8626925653047556</pre><h3>Prediction of Text Categories</h3><p>Once the model has been created, trained, and tested, we can use it to make predictions on texts as needed. We will create a dictionary with simulated texts related to the selected topics in this project to classify them. For this purpose, in addition to the dictionary, we will perform the necessary transformation/vectorization via TF-IDF.</p><pre># Creating texts for prediction<br>doc_new = doc_new = [&quot;Computer designers create amazing animations.&quot;,<br>&quot;Researchers are constantly finding new treatments to improve people&#39;s health.&quot;,<br>&quot;Having faith is fundamental as it provides comfort and hope.&quot;,<br>&quot;People with different political thoughts should come together to discuss solutions for the common good of all.&quot;,<br>&quot;Computer designers develop amazing animations.&quot;,<br>&quot;Our health must be taken care of in a preventive way too.&quot;,<br>&quot;Religion, for us Christians, is an essential part of our lives and must be respected.&quot;,<br>&quot;Political talking has been intense in recent days.&quot;]<br><br># Category of the document above<br>categories_doc_new = [&quot;sci.med&quot;, &quot;comp.graphics&quot;, &quot;soc.religion.christian&quot;, &quot;talk.politics.misc&quot;, &quot;sci.med&quot;, &quot;comp.graphics&quot;, &quot;soc.religion.christian&quot;, &quot;talk.politics.misc&quot;]<br><br><br># Performing text vectorization<br>doc_new_tfidf = tfidf_vector.transform(doc_new)<br>pred_doc_new = model.predict(doc_new_tfidf.toarray())<br><br># Compare the observed and predicted model<br>for dn, categories_dn inzip(categories_doc_new,pred_doc_new):<br>    print(f&quot;Observed category {dn} and Predicted category {df_train.target_names[categories_dn]}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*xGtMuqvg1Fb3WV1l072DNA.png" /></figure><p>As we can see above, we only had 1 error in 8, which represents a success rate of 87.5%. Not bad!</p><h3>Conclusion</h3><p>Text classification into topics is a very important task in labeling texts and is widely used by professionals such as journalists who need to compile information by topics. This task can be performed using the Naive Bayes Classifier algorithm, which is commonly used in text classification into topics. It has the advantage of assuming independence between predictor variables, which means it is not significantly impacted by multicollinearity, although this can be a disadvantage in some texts where variables may not be independent.</p><p>Texts need to be vectorized for which the TF-IDF method was used.</p><p>We imported the necessary libraries, loaded the dataset, and pre-processed it. Then we created and trained the model, analyzing it with accuracy metrics and F1 score.</p><p>Finally, we predicted the categories of simulated texts related to the topics of health, religion, politics, and technology. Overall, the Naive Bayes Classifier proved to be very useful in the text classification into topics process.</p><p>In this article, some takeaways can be highlighted:</p><ol><li>Text classification into topics is an important task in organizing and analyzing large volumes of data, and the Naive Bayes Classifier is a popular algorithm used for this purpose.</li><li>The Naive Bayes Classifier assumes independence between predictor variables, which can be a disadvantage for some types of texts, but it still offers advantages such as being less impacted by multicollinearity.</li><li>To perform text classification into topics, it’s necessary to preprocess the data, vectorize the texts using methods like TF-IDF, train the model, and evaluate it using metrics such as accuracy and f1 score.</li></ol><p>That’s all! Don’t forget to follow me here and on my social network <a href="https://twitter.com/lucasnogsousa">twitter</a> e <a href="https://gettr.com/user/lucasnogsousa">gettr</a> so I’ll know content like this is relevant for you! Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f74851b6b0a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a Customer Segmentation and Analyzing by Period with Python]]></title>
            <link>https://medium.com/@lucasnogsousa/creating-a-customer-segmentation-and-analyzing-by-period-with-python-e1061c20771f?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1061c20771f</guid>
            <category><![CDATA[marketing-analytics]]></category>
            <category><![CDATA[customer-behavior]]></category>
            <category><![CDATA[customer-segmentation]]></category>
            <category><![CDATA[rfm-analysis]]></category>
            <category><![CDATA[k-means-algorithm]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 12:02:13 GMT</pubDate>
            <atom:updated>2023-03-13T12:02:13.011Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Customer segmentation is the process of dividing consumers into groups with similar purchasing characteristics. By doing so, businesses are able to gain a better understanding of their customers’ needs and consumption preferences, ultimately becoming more effective in meeting their demands. Furthermore, grouping customers based on similar characteristics empower businesses to develop more targeted marketing strategies, which can increase customer loyalty and ultimately drive profitability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*DNtclBiH13TeGnYTM1e2Ug.jpeg" /><figcaption>Source: Pexels</figcaption></figure><p>In this article, we will explore the importance of customer segmentation, specifically using the RFM analysis and K-means clustering techniques with Python coding, and we will analyze customer behavior by period.</p><h3>Table of Contents</h3><ol><li>Introduction</li><li>Preparing for Analysis: Importing Packages, Loading Datasets, and Data Preprocessing</li><li>Understanding RFM Analysis</li><li>Implementing Customer Segmentation with K-means</li><li>Conclusion</li></ol><h3>Preparing for Analysis: Importing Packages, Loading Datasets, and Data Preprocessing</h3><p>To begin our analysis of customer purchasing behavior, we will need to take the following steps:</p><ol><li>First, we will import the necessary packages.</li><li>Next, we will download the dataset from Kaggle using the following link: <a href="https://www.kaggle.com/datasets/gabrielramos87/an-online-shop-business">https://www.kaggle.com/datasets/gabrielramos87/an-online-shop-business</a>.</li><li>Finally, we will select only the necessary fields for analysis and convert the Date column type from string to datetime.</li></ol><p>This dataset contains a set of e-commerce transactions with a periodicity of more than one year. By focusing on the customers’ purchasing behavior, we will be able to better understand their needs and desires, and use this information to create more effective marketing campaigns.</p><pre>#Importing packages<br>import pandas as pd<br>import numpy as np<br>import math<br>from sklearn.cluster import KMeans<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import datetime<br><br>#Data loading<br>df = pd.read_csv(&#39;Sales_Transaction.csv&#39;)<br><br>#We will only select the fields necessary for the action<br>df = df[[&#39;CustomerNo&#39;,&#39;TransactionNo&#39;,&#39;Date&#39;,&#39;Price&#39;,&#39;Quantity&#39;]].drop_duplicates()<br><br>#Creating a Total Value variable<br>df[&quot;Value&quot;] = df[&quot;Price&quot;]*df[&quot;Quantity&quot;]<br><br>#convert the Date column to datetime<br>df.Date = pd.to_datetime(df.Date, format=&#39;%m-%d-%Y&#39;, infer_datetime_format=True)</pre><h3>Understanding RFM Analysis</h3><p>RFM analysis is a data analysis technique used in customer segmentation to gain insights into their buying behavior. The acronym RFM stands for recency, frequency, and monetary. Recency refers to the number of days since a customer’s last purchase, frequency tells us how often a customer makes a purchase, and monetary value reflects how much a customer spends on purchases during a given period.</p><p>To establish the frequency of purchases, you can choose a period that is most suitable for the analysis, such as weekly or monthly, depending on the data available. In this article, we will use the periodicity of the transaction data to determine the frequency.</p><p>In Python, we can develop the RFM matrix using code that considers the recency, frequency, and monetary value of each customer, for gaining insights into customer behavior. Below is the code for constructing the RFM matrix:</p><pre>#Frequency<br>df_frequency = df.groupby(&#39;CustomerNo&#39;).agg({&#39;TransactionNo&#39;:&#39;count&#39;})<br>df_frequency.rename(columns={&#39;TransactionNo&#39;:&#39;frequency&#39;}, inplace=True)<br>df_frequency.head()<br><br>#Recency<br>df_recency = df.groupby(&#39;CustomerNo&#39;).agg({&#39;Date&#39;:&#39;max&#39;})<br>df_recency.columns = [&#39;LastDate&#39;]<br>df_recency[&#39;recency&#39;] = (df_recency[&#39;LastDate&#39;].max() - df_recency[&#39;LastDate&#39;]).dt.days<br>df_recency.drop(columns=&#39;LastDate&#39;, inplace=True)<br>df_recency.head()<br><br>#Monetary value<br>df_monetary = df.groupby(&#39;CustomerNo&#39;).agg({&#39;Value&#39;:&#39;sum&#39;}).rename(columns={&#39;Value&#39;:&#39;monetary&#39;})<br>df_monetary.head()<br><br>#RFM Matrix<br>rfm = pd.merge(pd.merge(df_frequency,df_recency,on=&#39;CustomerNo&#39;),df_monetary,on=&#39;CustomerNo&#39;)<br>rfm.head()</pre><h3>Implementing Customer Segmentation with K-means</h3><p>Once the RFM matrix is created, we will group customers and generate descriptions. To achieve this, we will apply the K-means clustering algorithm, which is used to group similar data points, and use the Elbow Method to determine the optimal number of clusters. The optimal_clusters_number function will assist us in identifying the most suitable number of clusters without relying exclusively on the Elbow graph. The code is shown below:</p><pre>#defining the inertia<br>def cluster_quantity(data):<br>    wcss = []<br>    for k in range(1,10):<br>        kmeans = KMeans(n_clusters = k).fit(X=data)<br>        wcss.append(kmeans.inertia_)<br>    return wcss<br>    <br>    <br>#Visualization of Elbow Method<br>cluster_list = []<br>for i in rfm.columns:<br>    plt.figure(figsize=(10,5))<br>    plt.plot(cluster_quantity(rfm[i].values.reshape(-1,1)))<br>    plt.ylabel(i)<br>    plt.xlabel(&#39;Quantity of clusters&#39;)<br>    plt.show()<br>    cluster_list.append(cluster_quantity(rfm[i].values.reshape(-1,1)))<br><br>#defining the number of clusters<br>def optimal_clusters_number(wcss):<br>    x1, y1 = 2,wcss[0]<br>    x2, y2 = 20,wcss[len(wcss)-1]<br>    <br>    distance = []<br>    for i in range(len(wcss)):<br>        x0 = i+2<br>        y0 = wcss[i]<br>        numerator = abs((y2 - y1)*x0 - (x2 - x1)*y0 + x2*y1 - y2*x1)<br>        denominator = math.sqrt((y2 - y1)**2 + (x2 - x1)**2)<br>        distance.append(numerator/denominator)<br>    return distance.index(max(distance)) + 2<br>    <br>#Optimal number<br>for i,v in enumerate(rfm.columns):<br>    kmeans=KMeans(n_clusters=optimal_clusters_number(cluster_list[i]))<br>    rfm[v+&#39;_cluster&#39;] = kmeans.fit_predict(rfm[v].values.reshape(-1,1))<br>    print(f&quot;Optimal cluster number for {v}&quot;,optimal_clusters_number(cluster_list[i]))</pre><p>To ensure a consistent cluster labeling, we will sort the clusters based on the average value of their RFM variables, and then use the index as the new group label. To achieve this, we need to reindex each RFM matrix variable dataframe based on the sorted group labels. It’s important to note that K-means is a stochastic algorithm, meaning that the clusters may vary each time we run it due to the random centroid selection. To address this issue, we can follow the steps below.</p><pre>#ordering the values of the clusters<br>def sort_cluster(cluster_name,target_name,df,asc=True):<br>    group_by_cluster = df.groupby(cluster_name)[target_name].mean().reset_index().sort_values(by=target_name,ascending=asc).reset_index(drop=True)<br>    group_by_cluster[&#39;index&#39;] = group_by_cluster.index<br>    join_cluster = pd.merge(df,group_by_cluster[[cluster_name,&#39;index&#39;]],on=cluster_name).drop([cluster_name],axis=1)<br>    df_final = join_cluster.rename(columns={&#39;index&#39;:cluster_name})<br>    return df_final<br>    <br>rfm = sort_cluster(&#39;frequency_cluster&#39;,&#39;frequency&#39;,rfm)<br>rfm = sort_cluster(&#39;recency_cluster&#39;,&#39;recency&#39;,rfm,asc=False)<br>rfm = sort_cluster(&#39;monetary_cluster&#39;,&#39;monetary&#39;,rfm)<br><br>#An overall customer score based on clusters will be created.<br>rfm[&#39;Score&#39;] = rfm[&#39;frequency_cluster&#39;]+rfm[&#39;recency_cluster&#39;]+rfm[&#39;monetary_cluster&#39;]<br><br>#Segmentations made after visual analysis.<br>rfm[&#39;Segment&#39;] = &#39;Lost&#39;<br>rfm.loc[rfm[&#39;Score&#39;]==1,&#39;Segment&#39;] = &#39;Almost lost&#39;<br>rfm.loc[rfm[&#39;Score&#39;]==2,&#39;Segment&#39;] = &#39;Hibernating&#39;<br>rfm.loc[rfm[&#39;Score&#39;]==3,&#39;Segment&#39;] = &#39;Cannot lose them&#39;<br>rfm.loc[rfm[&#39;Score&#39;]==4,&#39;Segment&#39;] = &#39;At risk&#39;<br>rfm.loc[rfm[&#39;Score&#39;]==5,&#39;Segment&#39;] = &#39;Nearly at risk&#39;<br>rfm.loc[rfm[&#39;Score&#39;]==6,&#39;Segment&#39;] = &#39;Potentially loyal&#39;<br>rfm.loc[rfm[&#39;Score&#39;]&gt;=7,&#39;Segment&#39;] = &#39;Champions&#39;</pre><p>Please keep in mind that we determined the optimal number of clusters for each variable of the RFM matrix and then added the values of the clusters taken from the three variables. We got the descriptions for each segmentation by analyzing a scatter plot. You can make modifications if you want. The descriptions for each segment are provided below:</p><ul><li>Champions: “Recently made high-value purchases and continue to do so frequently.”</li><li>Potentially loyal: “Made above-average purchases in terms of recency, frequency, and monetary value.”</li><li>Nearly at risk: “Made several purchases in the past, but haven’t made any recently.”</li><li>At risk: “Made frequent purchases in the past, but haven’t made any recently.”</li><li>Hibernating: “Made few purchases in the past, and haven’t made any recently.”</li><li>Almost lost: “Haven’t made any purchases recently, and have low frequency and monetary values.”</li><li>Lost: “Haven’t made any purchases recently, and have low frequency and monetary values.”</li></ul><p>But how can we perform periodic analysis in RFM segmentation? One way is to divide the data into quarters, although other time periods could be used. I chose quarterly for convenience. To help with this activity, we can use the function below, which groups customers by quarter and allows us to evaluate the model’s performance over time. Additionally, a function has been created to generate visualizations of the results both in quantitative and percentage terms. It’s important to note that we can also analyze the data using the quarterly accumulation of values by setting the parameter cum=True in the period_segmentation function.</p><pre>#Function for temporal analysis of customer groups<br>def period_segmentation(df,period,cum=False):<br>    concat_dataframe = []<br>    for m in period:<br>        if cum == True:<br>            #Frequency<br>            df_frequency = df[df.Date.dt.to_period(&#39;Q&#39;) &lt;= m].groupby(&#39;CustomerNo&#39;).agg({&#39;TransactionNo&#39;:&#39;count&#39;}).rename(columns={&#39;TransactionNo&#39;:&#39;frequency&#39;})<br>            #Recency<br>            df_recency = df[df.Date.dt.to_period(&#39;Q&#39;) &lt;= m].groupby(&#39;CustomerNo&#39;).agg({&#39;Date&#39;:&#39;max&#39;})<br>            df_recency.columns = [&#39;LastDate&#39;]<br>            df_recency[&#39;recency&#39;] = (df_recency[&#39;LastDate&#39;].max() - df_recency[&#39;LastDate&#39;]).dt.days.drop(columns=&#39;LastDate&#39;)<br>            #Monetary value<br>            df_monetary = df[df.Date.dt.to_period(&#39;Q&#39;) &lt;= m].groupby(&#39;CustomerNo&#39;).agg({&#39;Value&#39;:&#39;sum&#39;}).rename(columns={&#39;Value&#39;:&#39;monetary&#39;})<br>        else:<br>            #Frequency<br>            df_frequency = df[df.Date.dt.to_period(&#39;Q&#39;) == m].groupby(&#39;CustomerNo&#39;).agg({&#39;TransactionNo&#39;:&#39;count&#39;}).rename(columns={&#39;TransactionNo&#39;:&#39;frequency&#39;})<br>            #Recency<br>            df_recency = df[df.Date.dt.to_period(&#39;Q&#39;) == m].groupby(&#39;CustomerNo&#39;).agg({&#39;Date&#39;:&#39;max&#39;})<br>            df_recency.columns = [&#39;LastDate&#39;]<br>            df_recency[&#39;recency&#39;] = (df_recency[&#39;LastDate&#39;].max() - df_recency[&#39;LastDate&#39;]).dt.days.drop(columns=&#39;LastDate&#39;)<br>            #Monetary value<br>            df_monetary = df[df.Date.dt.to_period(&#39;Q&#39;) == m].groupby(&#39;CustomerNo&#39;).agg({&#39;Value&#39;:&#39;sum&#39;}).rename(columns={&#39;Value&#39;:&#39;monetary&#39;})<br><br>        #RFM Matrix<br>        rfm = pd.merge(pd.merge(df_frequency,df_recency,on=&#39;CustomerNo&#39;),df_monetary,on=&#39;CustomerNo&#39;)<br>    <br>        #Optimal cluster number. <br>        for i,v in enumerate(rfm.columns):<br>            n_clusters = optimal_clusters_number(cluster_list[i])<br>            #2 is the minimal number accept<br>            if n_clusters&gt;=2:<br>                kmeans=KMeans(n_clusters=n_clusters)<br>            else:<br>                kmeans=KMeans(n_clusters=3)<br>            rfm[v+&#39;_cluster&#39;] = kmeans.fit_predict(rfm[v].values.reshape(-1,1))<br>            #print(f&quot;Optimal cluster number for {v}&quot;,optimal_clusters_number(cluster_list[i]))<br>            <br>        <br>        #Ordering the cluster values to make it easier to describe them.<br>        rfm = sort_cluster(&#39;frequency_cluster&#39;,&#39;frequency&#39;,rfm)<br>        rfm = sort_cluster(&#39;recency_cluster&#39;,&#39;recency&#39;,rfm,asc=False)<br>        rfm = sort_cluster(&#39;monetary_cluster&#39;,&#39;monetary&#39;,rfm)   <br>        <br>        #An overall customer score based on clusters will be created.<br>        rfm[&#39;Score&#39;] = rfm[&#39;frequency_cluster&#39;]+rfm[&#39;recency_cluster&#39;]+rfm[&#39;monetary_cluster&#39;]<br>        <br>        #Segmentations made after visual analysis.<br>        rfm[&#39;Segment&#39;] = &#39;Lost&#39;<br>        rfm.loc[rfm[&#39;Score&#39;]==1,&#39;Segment&#39;] = &#39;Almost lost&#39;<br>        rfm.loc[rfm[&#39;Score&#39;]==2,&#39;Segment&#39;] = &#39;Hibernating&#39;<br>        rfm.loc[rfm[&#39;Score&#39;]==3,&#39;Segment&#39;] = &#39;Cannot lose them&#39;<br>        rfm.loc[rfm[&#39;Score&#39;]==4,&#39;Segment&#39;] = &#39;At risk&#39;<br>        rfm.loc[rfm[&#39;Score&#39;]==5,&#39;Segment&#39;] = &#39;Nearly at risk&#39;<br>        rfm.loc[rfm[&#39;Score&#39;]==6,&#39;Segment&#39;] = &#39;Potentially loyal&#39;<br>        rfm.loc[rfm[&#39;Score&#39;]&gt;=7,&#39;Segment&#39;] = &#39;Champions&#39;<br>        <br>        rfm[&#39;Period&#39;] = m<br>        concat_dataframe.append(rfm)<br>    return pd.concat(concat_dataframe, axis=0)<br>    <br>#Set the period here. In this case, the period used was the quarter.<br>period = df.Date.dt.to_period(&#39;Q&#39;)<br>period = period.unique()<br><br>#Final dataset with the matrix divided by period for analysis<br>rfm_final = period_segmentation(df,period,cum=False)<br><br>#Grouping the dataset and converting the date into a year and quarter format. You can choose another period.<br>rfm_period = rfm_final.groupby([&#39;Period&#39;,&#39;Segment&#39;]).agg({&#39;Segment&#39;:&#39;count&#39;}).rename(columns={&#39;Segment&#39;:&#39;Client&#39;}).reset_index()<br>rfm_period.Period = rfm_period.Period.apply(lambda x: pd.to_datetime(str(x.year)+&#39;-&#39;+str(x.quarter))).dt.strftime(&#39;%YQ%m&#39;)<br>#Percentage<br>rfm_period[&#39;Client_%&#39;] = round((rfm_period[&#39;Client&#39;]/rfm_period[&#39;Client&#39;].sum()) * 100,2)<br><br>def plot_segmento(x,y,hue,data):<br>    sns.set(palette=&#39;muted&#39;,color_codes=True,style=&#39;whitegrid&#39;)<br>    plt.figure(figsize=(15,8))<br>    sns.lineplot(x=x,y=y,hue=hue,data=data,style=hue,markers=True)<br>    plt.show()<br>    <br>plot_segmento(x=&#39;Period&#39;,y=&#39;Client&#39;,hue=&#39;Segment&#39;,data=rfm_period)</pre><h3>Percentage view</h3><p>The percentage view shows us the categories with the highest relative increase, since they have different sizes. This allows us to compare the categories with each other, regardless of their total size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*Puf9w59dE9N1MjRf-S_h2A.jpeg" /></figure><h3>Quantitative view</h3><p>The quantitative view demonstrates the absolute growth of the categories. This allows us to make comparisons between them over different periods of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*wWNC-NF17TuCNRwvVC1dHw.jpeg" /></figure><h3>Accumulated view</h3><p>The accumulated view helps us understand how the categories are changing over time, showing us their trends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*XZSbdfMlIhC0X-xE8P7iMw.jpeg" /></figure><p>The graphs below provide some insights. For instance, the non-cumulative quarterly analysis shows an increase in the number of customers in the “Cannot lose them”, “Hibernating”, and “Almost lost” segments, while the others remain relatively stable. In the cumulative view by quarter, there is a decline in the absolute number of customers in the “Champions” segment. Additionally, in terms of the relative frequency percentage, the “Lost” segment has a higher proportion of customers than the “At risk” segment, which is different from the absolute view. Other valuable insights can be extracted from the data, depending on the creativity and specific needs of each analyst.</p><h3>Conclusion</h3><p>In summary, customer segmentation, coupled with an analysis of past behavior to understand customer movement between clusters, is crucial for developing effective tactics to improve customer satisfaction and reduce expenses. For instance, segmenting customers into “At risk” or “Lost” groups can help businesses focus their efforts on retaining high-value customers and avoid spending unnecessary resources on low-value customers.</p><p>Implementing customer segmentation through RFM analysis and K-means clustering techniques can provide businesses with several benefits, including:</p><ul><li>Descriptive analysis of consumer segments over time to better understand changes and patterns.</li><li>Improved customer retention and contentment tactics by evaluating the performance of each group and identifying areas for enhancement.</li><li>More efficient use of resources by targeting marketing efforts to specific consumer groups, such as focusing on retaining “At risk” customers or re-engaging “Hibernating” customers, and Cost reduction by avoiding targeting low-value consumer groups and focusing on high-value segments.</li></ul><p>That’s all! Follow me for more articles. Don’t forget to comment. Hug!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1061c20771f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Maximize Your Income with Data Science: 4 Strategies for Career Advancement]]></title>
            <link>https://medium.com/@lucasnogsousa/how-to-maximize-your-income-with-data-science-3-strategies-for-career-advancement-e60a15c9e1b2?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e60a15c9e1b2</guid>
            <category><![CDATA[income]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[career-advancement]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[professional-development]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Sat, 11 Mar 2023 19:03:36 GMT</pubDate>
            <atom:updated>2023-03-11T19:04:35.083Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>As the demand for data-driven insights continues to grow, data science has become a vital tool for businesses across industries. In today’s job market, professionals who can harness the power of data to drive business results are highly sought after, and the potential for career advancement and increased income is significant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fUVlqMHsKx6XHzLTMrY3Zw.jpeg" /><figcaption>Source: Pexels</figcaption></figure><p>In this article, we’ll explore four key strategies for leveraging your career as data science professional and maximize your earning potential. Let’s dive in!</p><h3>Table of Contents</h3><ol><li>Introduction</li><li>Identify a Data Science problem to solve</li><li>Present your idea to stakeholders</li><li>Implement your Data Science solution</li><li>Present your results and ask for a promotion</li><li>Conclusion</li></ol><h3>Identify a Data Science problem to solve</h3><p>To start using Data Science to advance your career and increase your earning potential, the first step is to identify a relevant problem or opportunity that can be addressed using this powerful set of techniques. One way to go about this is to look for areas in your organization or industry that could benefit from better data analysis and insights. For example, you may want to identify areas where data is currently not being leveraged to its full potential, or where there is a clear need for more data-driven decision-making.</p><p>Once you’ve identified a potential problem or opportunity, you can start to explore how Data Science techniques might be applied to address it. For instance, you might use machine learning algorithms to predict customer behavior, or develop a data-driven marketing strategy to boost sales. Other potential projects might involve optimizing supply chain operations, analyzing financial data to identify areas for cost savings, or developing predictive maintenance models for industrial equipment.</p><p>I recently identified an opportunity to apply a machine learning model to predict customer CLV and churn. This enabled my company, a service provider, to retain a customer who was considering cancelling their contract. I wrote an article about this project, which can be found at the following link: <a href="https://medium.com/@lucasnogsousa/analyzing-customer-lifetime-value-clv-and-churn-with-python-83db54e409ec">https://medium.com/@lucasnogsousa/analyzing-customer-lifetime-value-clv-and-churn-with-python-83db54e409ec</a></p><p>The key is to identify a problem or opportunity that is both relevant to your organization or industry, and can be effectively addressed using Data Science techniques. By doing so, you can demonstrate your value as a Data Science professional, and position yourself for career advancement and increased income.</p><h3>Present your idea to stakeholders</h3><p>Once you have identified a Data Science problem to solve, it’s time to present your idea to the relevant stakeholders. In order to effectively pitch your project to your supervisor or other key decision makers, it’s important to highlight the potential benefits and outcomes of the project. One approach is to frame the project in terms of its potential impact on the company’s bottom line, emphasizing the value that Data Science can bring to the organization. It’s also helpful to provide concrete examples or case studies that demonstrate the potential benefits of the project. Finally, be prepared to answer any questions or concerns that stakeholders may have, and be open to feedback and suggestions. By effectively presenting your idea and making a strong case for its potential value, you can increase the likelihood of securing support and resources for your project.</p><h3>Implement your Data Science solution</h3><p>Now that you have identified a relevant problem and pitched your idea to stakeholders, it’s time to implement your Data Science solution. This stage is critical to the success of your project and involves careful planning and execution. It’s important to work closely with stakeholders and keep them updated throughout the implementation process to ensure their approval and support.</p><p>To start, you should design a clear plan for executing the project, including timelines, milestones, and resource allocation. You should also consider potential challenges that may arise, such as data quality issues, resource limitations, or technical hurdles. It’s important to have contingency plans in place to address these challenges and ensure the project stays on track.</p><p>Throughout the implementation process, it’s important to continuously assess and evaluate the progress of your project. This will help you identify any issues or areas that need improvement, and allow you to make necessary adjustments to ensure the project’s success.</p><p>Remember, getting approval from stakeholders is critical to the success of your project. Keep them engaged and informed throughout the implementation process, and be open to feedback and suggestions. By working collaboratively with stakeholders, you can ensure that your Data Science solution is effective and meets the needs of your organization.</p><h3>Present your results and ask for a promotion</h3><p>When presenting your project results, it is essential to showcase your achievements and how they contribute to the company’s bottom line. Be sure to highlight the financial benefits of your project, such as increased revenue or cost savings. Additionally, emphasizing the skills and expertise you’ve gained through the project can help make a persuasive case for your promotion.</p><p>It’s also important to remember that while seeking a promotion within your current company is a viable option, exploring opportunities at other companies can also be a way to increase your income. By keeping an eye out for positions that align with your skills and interests, you can position yourself for long-term career growth and success. Remember that you can even work from home online as a <a href="https://hop.clickbank.net/?affiliate=lucasns&amp;vendor=clkearners">freelance virtual assistant</a> for example. If you don’t have ideas on how to start, maybe this is a good starting point for you.</p><h3>Conclusion</h3><p>In conclusion, we have explored four key strategies for leveraging Data Science to advance your career and increase your earning potential. By identifying a relevant problem or opportunity, effectively pitching your project, and implementing a successful Data Science solution, you can create significant value for your company and yourself. Remember to present your results in a clear and compelling way, and don’t be afraid to ask for that well-deserved promotion. By taking action and applying these strategies, you can position yourself as a valuable asset to any organization or even consider other career opportunities in the market. The world of Data Science is ever-evolving, and it’s up to you to seize the opportunities and maximize your potential.</p><p>That is all! Don’t forget to follow me! Hug!</p><h3>Disclaimer</h3><p><em>The content of this article is for informational purposes only and should not be considered as professional or financial advice. Readers are encouraged to seek professional guidance or specialized consultation before making any decisions that may affect their careers or finances. The author shall not be held responsible for any damages or losses resulting from the use or application of the information contained in this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e60a15c9e1b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extracting Information from Images with OCR and Pytesseract]]></title>
            <link>https://medium.com/@lucasnogsousa/extracting-information-from-images-with-ocr-and-pytesseract-ebd90c1617a4?source=rss-a304f6ccdc2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebd90c1617a4</guid>
            <category><![CDATA[c-sharp-programming]]></category>
            <category><![CDATA[pytesseract]]></category>
            <category><![CDATA[ocr-tesseract]]></category>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[text-recognition]]></category>
            <dc:creator><![CDATA[Lucas Nogueira]]></dc:creator>
            <pubDate>Fri, 10 Mar 2023 22:28:35 GMT</pubDate>
            <atom:updated>2026-09-11T14:31:57.458Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Hello! Have you ever needed to transcribe text from an image, such as information from a scanned document, and even when using Python libraries that claim to convert images to text, the information was not transcribed or was only partially transcribed? In this case, the solution to this problem may lie in using OCR (Optical Character Recognition) with the Pytesseract library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5jmZSZ_BtwOHGkJBsdYwQ.jpeg" /><figcaption>Source: Pexels</figcaption></figure><p>OCR is a technology that allows the conversion of images into editable text, which is very useful in various situations. The Pytesseract library is a Python interface for the Tesseract OCR library, which is one of the most accurate OCR technologies currently available. Using Pytesseract, text recognition in an image can be done easily and quickly.</p><h3>Table of contents</h3><ol><li>Introduction</li><li>Installation and package importation, and image loading</li><li>Image preprocessing</li><li>Text recognition</li><li>Conclusion</li></ol><h3>Package importation and image loading</h3><p>To begin, it is necessary to install the Pytesseract library and the Tesseract OCR tool. Since we will be using Google Colab, we can use the following commands below:</p><pre>!sudo apt install tesseract-ocr<br>!pip install pytesseract</pre><p>Once the library is installed, the next step is to import the necessary packages and load the image to be converted into text. For this, we can use the following libraries and functions:</p><pre># importing packets<br>import pytesseract<br>import numpy as np<br>import requests<br>from PIL import Image</pre><p>After installing the libraries, we will download the image from the Pixabay website in Colab using the Requests library and load the image. Pytesseract supports various image formats, including JPG, PNG, and BMP. To load the image, we just need to use the Pillow library (also known as PIL) and the Image.open() function:</p><pre># request the image from pixabay, which is a free download image bank<br>url = &quot;https://cdn.pixabay.com/photo/2015/12/07/23/28/thankful-1081614_960_720.jpg&quot;<br>response = requests.get(url)<br>with open(&quot;thankful.jpg&quot;, &quot;wb&quot;) as f:<br>    f.write(response.content)<br><br># open the imported image<br>img = Image.open(&#39;/content/thankful.jpg&#39;)</pre><p>To visualize the image, we will use the img.show() command:</p><pre># view image<br>img.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*ubB2LRjNQJfRzh_ObfTwag.png" /></figure><p>With the image loaded and displayed, the next step is to use Pytesseract to perform text recognition. This can be done by calling the function image_to_string(), which takes the image as a parameter and returns the recognized text:</p><pre># read the recognized text from the image<br>text = pytesseract.image_to_string(img) <br>print(text)</pre><h3>Image preprocessing</h3><p>As you may have noticed, Pytesseract was not able to recognize the text in the image. This happened because some image processing is required, as Tesseract OCR requires images to be in a specific format for OCR to work correctly. This includes converting the image to grayscale, applying a good resolution, appropriate contrast and color levels, as well as smoothing filters and noise removal.</p><h4>Conversion to grayscale</h4><p>First, we will convert the image to a grayscale format. To do this, we can use the img.convert() command, passing the argument ‘L’, which converts the image to an 8-bit grayscale format.</p><pre># convert to grayscale<br>img_gray = img.convert(&quot;L&quot;)<br>text = pytesseract.image_to_string(img_gray)<br>print(text)</pre><h4>Adjustment of image resolution, contrast, and edge detection</h4><p>As you may have noticed, just converting the image to grayscale was not enough for the text to be recognized in the image. Therefore, some other treatments must be performed. We can try adjusting the image resolution, adjusting the image contrast, and detecting edges. See the code below, to do so:</p><pre>#importing packets<br>from PIL import Image, ImageEnhance, ImageOps, ImageFilter<br><br># open the image and convert it to grayscale<br>img = Image.open(&#39;/content/thankful.jpg&#39;).convert(&#39;L&#39;)<br><br># increase the image resolution<br>img = img.resize((img.width*3, img.height*3))<br><br># adjust the image contrast<br>enhancer = ImageEnhance.Contrast(img)<br>img = enhancer.enhance(3.5)<br><br># apply edge detection<br>img = img.filter(ImageFilter.FIND_EDGES)<br><br># convert the image to the format accepted by Tesseract OCR<br>img = ImageOps.invert(img)<br><br># perform OCR on the image<br>text = pytesseract.image_to_string(img, lang=&#39;eng&#39;)<br><br># print the recognized text<br>print(text)<br><br># results<br>Today Te<br>| am<br>ret i<br>et for: =<br>_ Today.,</pre><h4>Gaussian smoothing</h4><p>We can already see some scribbles, but it was not enough to detect all the text, as it was not clear enough. In this case, we will try the Gaussian smoothing, which is a form of smoothing to reduce noise and improve text readability.</p><pre># open the image<br>img = Image.open(&#39;/content/thankful.jpg&#39;)<br><br># apply Gaussian filter for smoothing<br>img = img.filter(ImageFilter.GaussianBlur(radius=2))<br><br># convert the image to the format accepted by Tesseract OCR<br>img = ImageOps.invert(img)<br><br># execute OCR on the image<br>text = pytesseract.image_to_string(img, lang=&#39;eng&#39;)<br><br># print the recognized text<br>print(text)<br><br># results<br>T oday</pre><h3>Text recognition</h3><h4>Median filter</h4><p>Using Gaussian smoothing, only one word appeared, which is Today. Now, we will try a noise removal technique with median filter that aims to eliminate small intensity variations that can impair object segmentation in the image or text recognition. The median filter is a type of non-linear filter that replaces the pixel value with the median of the pixel values in a neighborhood around it. This helps remove random noise that causes abrupt intensity variations in small regions of the image without affecting important edges or details.</p><p>In the context of text recognition in images, noise removal with median filter can help improve image quality by reducing intensity variations that can hinder character recognition by Tesseract OCR. It is a common technique for image preprocessing before applying OCR.</p><pre># open the image and convert it to grayscale<br>img = Image.open(&#39;/content/thankful.jpg&#39;).convert(&#39;L&#39;)<br><br># adjust the image contrast<br>enhancer = ImageEnhance.Contrast(img)<br>img = enhancer.enhance(2.5)<br><br># apply median filter to remove noise<br>img = img.filter(ImageFilter.MedianFilter(size=5))<br><br># execute OCR on the image<br>text = pytesseract.image_to_string(img, lang=&#39;eng&#39;)<br><br># print the recognized text<br>print(text)<br><br># results<br>T oday<br><br>lam \<br>\ THANKFUL<br>for<br>Today</pre><p>Finally, by converting the image to grayscale, using the median filter to remove noise along with the contrast adjustment, we were able to extract the text from the image. The text reads “Today I am so THANKFUL for Today.” Other techniques can be tested to obtain better results.</p><h3>The Same Pipeline on .NET</h3><p>For anyone doing this in C# rather than Python, the preprocessing sequence you assembled by hand — grayscale, contrast, median filter — exists as built-in input filters, which also removes the risk of the polarity and edge-detection mistakes above:</p><pre>PM &gt; Install-Package IronOcr</pre><p>csharp</p><pre>using IronOcr;<br><br>var ocr = new IronTesseract();<br>ocr.Configuration.PageSegmentationMode = <br>TesseractPageSegmentationMode.SparseText; // - psm 11<br><br>using var input = new OcrInput();<br>input.LoadImage(&quot;thankful.jpg&quot;);<br><br>input.ToGrayScale(); // your convert(&#39;L&#39;)<br>input.DeNoise(); // your MedianFilter<br>input.Deskew();<br><br>var result = ocr.Read(input);<br>Console.WriteLine(result.Text);</pre><p>The filters are tuned for OCR specifically, so there’s no equivalent of FIND_EDGES to reach for by mistake, and they apply to the input that actually gets read. Same Tesseract 5 engine underneath, so the accuracy ceiling is identical — what it saves is the trial-and-error you documented here. There’s also no apt install tesseract-ocr step, since the engine and language data ship in the package.</p><p>.NET only, so not a swap for a Python project. Commercial with a free 30-day trial.</p><p>📚<a href="https://ironsoftware.com/csharp/ocr/"> IronOCR for .NET</a> ·<a href="https://ironsoftware.com/csharp/ocr/tutorials/how-to-read-text-from-an-image-in-csharp-net/"> Read text from images in C#</a> ·<a href="https://ironsoftware.com/csharp/ocr/licensing/"> Free trial</a></p><h3>Conclusion</h3><p>Text recognition in images using OCR with the Pytesseract library is a very useful solution for converting information contained in scanned documents or other types of images. With Pytesseract, the recognition process is fast and simple, and it is possible to obtain quite accurate results, although correct data handling is not dispensable. It’s worth trying and exploring this tool to discover how it can help optimize your work and increase your productivity.</p><p>So that’s it! Don’t forget to follow me! Thank you very much! Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebd90c1617a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>