<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by CarlaZr on Medium]]></title>
        <description><![CDATA[Stories by CarlaZr on Medium]]></description>
        <link>https://medium.com/@carla.zarazir?source=rss-2ac42e3278fc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ddoHNphQaMOMZ31xmkHYMA.jpeg</url>
            <title>Stories by CarlaZr on Medium</title>
            <link>https://medium.com/@carla.zarazir?source=rss-2ac42e3278fc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:54:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@carla.zarazir/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Classifying Social Media User Messages for a Cinema Platform]]></title>
            <link>https://medium.com/@carla.zarazir/classifying-social-media-user-messages-for-a-cinema-platform-4307b29f1c0a?source=rss-2ac42e3278fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/4307b29f1c0a</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[CarlaZr]]></dc:creator>
            <pubDate>Mon, 19 Jan 2026 14:12:02 GMT</pubDate>
            <atom:updated>2026-01-19T14:12:02.387Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s been a while since my last post (life, work, and everything in between happened), but I’m excited to share a recent project that got me back into coding.</p><p>I built a machine learning model to help a cinema client to automatically label incoming customer messages from social media.</p><p>The goal was simple on paper:<br>Turn thousands of unstructured messages into a clean, hierarchical taxonomy that teams could actually use.</p><p>The reality was not.</p><h3>The Data Challenge</h3><p>The raw dataset contained user-generated messages from social media channels, in Arabic and English. It came with predictable but painful issues:</p><ul><li>Mixed language and spelling variations</li><li>Emojis, punctuation-heavy messages, and empty text</li><li>Copied threads where agent replies are embedded inside the same text blob</li><li>Timestamped conversation dumps (especially in direct messages)</li></ul><p>Before classification could even be considered, the real task was extracting only what the user actually wrote.</p><p>I started by setting up my Python environment.</p><pre>import re<br>import sys<br>from pathlib import Path<br>from typing import Tuple, Optional, Iterable<br>import pandas as pd</pre><h4>Step 1: Cleaning and building a “User Message” field</h4><p>The cleaning pipeline focused on three main things:</p><p><strong>a) Filtering a specific time window</strong></p><p>The dataset included multiple periods, but the analysis needed a defined date window. The script parses the original timestamp and filters by a chosen year and month set.<br>In this example, the scope is December 2025.</p><pre><br>YEAR_FILTER = 2025<br>MONTHS_FILTER = {12}  # December</pre><p>The timestamp is split into separate date and time fields, then filtered:</p><pre>def split_created_time(series: pd.Series):<br>    dt = pd.to_datetime(series, errors=&quot;coerce&quot;, utc=False)<br>    created_date = dt.dt.date.astype(&quot;string&quot;)<br>    created_time = dt.dt.time.astype(&quot;string&quot;)<br>    return dt, created_date, created_time<br><br>def filter_by_date(df: pd.DataFrame, created_col: str) -&gt; pd.DataFrame:<br>    dt, cdate, ctime = split_created_time(df[created_col])<br><br>    df = df.copy()<br>    insert_at = df.columns.get_loc(created_col) + 1<br>    df.insert(insert_at, &quot;Created Date&quot;, cdate)<br>    df.insert(insert_at + 1, &quot;Created Time&quot;, ctime)<br><br>    mask = (dt.dt.year == YEAR_FILTER) &amp; (dt.dt.month.isin(MONTHS_FILTER))<br>    df = df.loc[mask].copy()<br><br>    # drop original timestamp if you want<br>    df = df.drop(columns=[created_col], errors=&quot;ignore&quot;)<br>    df.reset_index(drop=True, inplace=True)<br>    return df</pre><p><strong>b) Dropping Noise</strong></p><p>Some rows are not useful for classification:</p><ul><li>empty text</li><li>“story expired” type messages</li><li>very short emoji-only content</li></ul><p>These were removed to avoid polluting the taxonomy counts.</p><pre>ONLY_EMOJI_PUNCT = re.compile(r&quot;^[^\w\u0600-\u06FF]+$&quot;, re.UNICODE)<br><br>def should_drop_user_text(txt: str) -&gt; bool:<br>    if not isinstance(txt, str):<br>        return True<br>    s = txt.strip()<br>    if not s:<br>        return True<br>    if ONLY_EMOJI_PUNCT.fullmatch(s) and len(s) &lt;= 4:<br>        return True<br>    return False</pre><p><strong>c) Dropping Unused Columns</strong></p><p>The dataset included many operational fields that were irrelevant for text analysis.<br> These were removed early to keep the working dataframe lightweight.</p><pre>COLUMNS_TO_DROP = [<br>    &quot;ACL Account ID&quot;,&quot;Author ID&quot;,&quot;Author name&quot;,&quot;Managed Profile ID&quot;,&quot;Profile name&quot;,&quot;Profile link&quot;,<br>    &quot;Content ID&quot;,&quot;Parent ID&quot;,&quot;Labels&quot;,&quot;URL&quot;,&quot;Salesforce case URLs&quot;,&quot;Internal notes&quot;,&quot;Comment count&quot;,<br>    &quot;Like count&quot;,&quot;Rating score&quot;,&quot;Origin&quot;,&quot;Case care ID&quot;,&quot;Created by&quot;,<br>]<br><br># Drop unused columns<br>    df = df.drop(columns=[c for c in COLUMNS_TO_DROP if c in df.columns], errors=&quot;ignore&quot;)</pre><p><strong>d) Building User Message based on Content Type.</strong></p><p>Messages arrived under different content types.</p><ul><li>Direct messages: timestamped conversation dumps</li><li>Comments, mentions, replies: user text sometimes mixed with agent responses</li></ul><p>The approach:</p><ul><li>For DMs, split the conversation and keep only lines that look like user text</li><li>For public threads, remove embedded agent replies</li><li>Ignore unsupported content types</li></ul><pre>def is_dm(content_type: str) -&gt; bool:<br>    return str(content_type).strip().lower() in {&quot;direct message&quot;, &quot;dm&quot;, &quot;directmessage&quot;}<br><br>def is_public_thread(content_type: str) -&gt; bool:<br>    ct = str(content_type).strip().lower()<br>    return any(k in ct for k in [&quot;comment&quot;, &quot;mention&quot;, &quot;reply&quot;])<br><br>def extract_user_message(message: str, content_type: str):<br>    if not isinstance(message, str) or not message.strip():<br>        return None<br><br>    if is_dm(content_type) or is_public_thread(content_type):<br>        cleaned = strip_agent_lines(message)<br>        return cleaned or None<br><br>    return None</pre><p><strong>e) Removing Agent Replies Without Losing User Content</strong></p><p>This was the most sensitive part of cleaning</p><p>Agent replies often appear inline with user text and include:</p><ul><li>Agent replies starting with a recognizable prefix</li><li>Templated service phrases</li><li>Signatures or system text</li></ul><pre>#Remove Agent Reply <br><br>TS_BOUNDARY = re.compile(r&quot;(?=(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}))&quot;)<br><br>AGENT_PREFIX = re.compile(r&quot;^\s*(?:brand|support|agent)\b[:：\-]*\s*&quot;, re.IGNORECASE)<br>AGENT_TONE = re.compile(r&quot;(?:we will contact you|please provide|thank you for contacting)&quot;, re.IGNORECASE)<br><br>def strip_agent_lines(text: str) -&gt; str:<br>    if not isinstance(text, str):<br>        return &quot;&quot;<br><br>    if TS_BOUNDARY.search(text):<br>        chunks = [c for c in TS_BOUNDARY.split(text) if c and not TS_BOUNDARY.fullmatch(c)]<br>    else:<br>        chunks = text.replace(&quot;\r\n&quot;, &quot;\n&quot;).replace(&quot;\r&quot;, &quot;\n&quot;).split(&quot;\n&quot;)<br><br>    kept = []<br>    for ch in chunks:<br>        s = re.sub(r&quot;^\d{4}-\d{2}-\d{2}T[^\s]*\s*&quot;, &quot;&quot;, str(ch)).strip()<br>        if not s:<br>            continue<br><br>        looks_agent = bool(AGENT_PREFIX.match(s)) or bool(AGENT_TONE.search(s))<br>        if not looks_agent:<br>            kept.append(s)<br><br>    return &quot;\n&quot;.join(kept).strip()</pre><h4>Step 2: Rule-based hierarchical classification</h4><p>Once messages were clean, each one was classified into a three-level hierarchy:</p><ul><li>Main Request</li><li>Sub Request 1</li><li>Sub Request 2</li></ul><p>The classifier is rule-based and designed to handle Arabic and English.</p><p>It performs five steps:</p><ul><li>Normalize the text (Arabic + English)</li><li>Detect high-priority cases first (failed booking with charge)</li><li>Score messages against a list of ordered taxonomy rules</li><li>Refine the most specific label (Sub Request 2)</li><li>Export labeled data and summary tables to Excel</li></ul><p><strong>a) Text Normalization (English &amp; Arabic)</strong></p><p>Normalization makes matching stable across spelling variations and mixed-language text.</p><pre><br><br>AR_DIACRITICS = re.compile(r&quot;[\u064B-\u0652\u0670\u0640]&quot;) <br>AR_PUNCT = re.compile(r&quot;[^\w\s%]|[،؛؟٬]+&quot;, flags=re.UNICODE)  # keep % (offers)<br><br>AR_NUMS = &quot;٠١٢٣٤٥٦٧٨٩&quot;<br>EN_NUMS = &quot;0123456789&quot;<br>AR_TO_EN_NUM_MAP = str.maketrans(AR_NUMS, EN_NUMS)<br><br>def normalize_text(s: str) -&gt; str:<br>    if not isinstance(s, str):<br>        return &quot;&quot;<br>    s = s.strip().lower()<br>    s = AR_DIACRITICS.sub(&quot;&quot;, s)<br>    s = (s.replace(&quot;أ&quot;, &quot;ا&quot;).replace(&quot;إ&quot;, &quot;ا&quot;).replace(&quot;آ&quot;, &quot;ا&quot;)<br>           .replace(&quot;ى&quot;, &quot;ي&quot;).replace(&quot;ئ&quot;, &quot;ي&quot;).replace(&quot;ؤ&quot;, &quot;و&quot;))<br>    s = s.translate(AR_TO_EN_NUM_MAP)<br>    s = AR_PUNCT.sub(&quot; &quot;, s)<br>    s = re.sub(r&quot;\s+&quot;, &quot; &quot;, s).strip()<br>    return s</pre><p>The normalization process I used:</p><ul><li>Lowercase + trim</li><li>Remove Arabic diacritics</li><li>Unify common Arabic letter variants (أ/إ/آ → ا, ى → ي, etc.)</li><li>Convert Arabic numerals to English numerals (٠١٢… → 012…)</li><li>Remove punctuation while keeping meaningful tokens (for example % for discounts)</li></ul><p><strong>b) Building Blocks Lexicon &amp; Patterns</strong></p><p>The classifier uses a set of regex-based lexicons to detect themes. Each lexicon is a regex string representing multiple ways users might express the same intent.</p><p>Examples of what’s covered:</p><ul><li>refunds and deductions</li><li>login / OTP / password reset</li><li>booking failures and technical errors</li><li>location and showtimes</li><li>offers and promo codes</li><li>customer service follow-up, escalation, contact channels</li><li>experience issues, staff complaints</li></ul><p>Some cases are more urgent, especially when the user reports a payment deduction plus a booking failure. I treated this as a priority path so it gets classified before generic rules.</p><pre><br><br>AR_DEDUCT    = r&quot;تم الخصم|charged|deducted|money was taken&quot;<br>AR_FAIL_CORE = r&quot;فشل|تعذر|crash|freeze|timeout|failed|error&quot;<br>AR_BLACK     = r&quot;black ?screen|شاشة سوداء&quot;<br>AR_SLOW      = r&quot;long loading|slow loading|تحميل طويل&quot;<br>AR_TIMEOUT   = r&quot;timeout|time.?out|مهلة|انتهى الوقت&quot;<br>AR_FAILED_MSG= r&quot;failed booking message|فشل الحجز|رفض الحجز&quot;<br>AR_BOOK_VERB = r&quot;\bbook(?:ing)?\b|حجز|تذاكر|ticket|tickets&quot;<br><br>def detect_failed_booking(txt: str):<br>    charged = re.search(any_words(AR_DEDUCT), txt, re.I)<br>    failed  = (<br>        re.search(AR_FAIL_CORE, txt, re.I)<br>        or re.search(AR_FAILED_MSG, txt, re.I)<br>        or re.search(AR_BLACK, txt, re.I)<br>        or re.search(AR_SLOW, txt, re.I)<br>        or re.search(AR_TIMEOUT, txt, re.I)<br>    )<br>    booking_context = re.search(AR_BOOK_VERB, txt, re.I) or re.search(<br>        r&quot;\b(checkout|payment|visa|mastercard|apple ?pay)\b&quot;, txt, re.I<br>    )<br><br>    if charged and (failed or booking_context):<br>        if re.search(any_words(r&quot;crash|freeze|app.*closed|علق|توقف&quot;), txt, re.I):<br>            return (&quot;Failed booking&quot;, &quot;Failed with Charge&quot;, &quot;App Crashed&quot;)<br>        if re.search(AR_BLACK, txt, re.I):<br>            return (&quot;Failed booking&quot;, &quot;Failed with Charge&quot;, &quot;Black Screen&quot;)<br>        if re.search(AR_SLOW, txt, re.I):<br>            return (&quot;Failed booking&quot;, &quot;Failed with Charge&quot;, &quot;Long Loading&quot;)<br>        if re.search(AR_TIMEOUT, txt, re.I):<br>            return (&quot;Failed booking&quot;, &quot;Failed with Charge&quot;, &quot;Time Out&quot;)<br>        return (&quot;Failed booking&quot;, &quot;Failed with Charge&quot;, &quot;Failed Booking Message&quot;)<br><br>    return None</pre><p>This block reduces misclassification by catching high-risk payment-related complaints early.</p><p><strong>c) Taxonomy rules and scoring</strong></p><p>The classifier uses a list of rules. Each rule contains:</p><ul><li>the target label (Main, Sub1, Sub2)</li><li>a list of patterns that should match</li></ul><p>Then it scores each rule by how many patterns match and picks the best match.</p><pre>RULES = [<br>    ((&quot;Customer Service&quot;, &quot;Contact Channels&quot;, &quot;Phone / Email / WhatsApp&quot;),<br>        [r&quot;customer\s*service|support|خدمة العملاء&quot;, r&quot;phone|email|whatsapp|اتصال|ايميل|واتس&quot;]),<br><br>    ((&quot;App/Website&quot;, &quot;App&quot;, &quot;OTP Not Received&quot;),<br>        [r&quot;\bapp\b|تطبيق|ios|android&quot;, r&quot;\botp\b|رمز|كود&quot;, r&quot;لم يصل|ما وصل|not.*receive&quot;]),<br><br>    ((&quot;Movie&quot;, &quot;Showtime&quot;, &quot;How to Book / Can I Book?&quot;),<br>        [r&quot;showtime|schedule|مواعيد|عرض&quot;, r&quot;\bbook(?:ing)?\b|حجز|احجز&quot;]),<br><br>    ((&quot;Offers&quot;, &quot;Offer Issues&quot;, &quot;Offer Not Working&quot;),<br>        [r&quot;offer|discount|خصم|promo ?code|كوبون&quot;, r&quot;not working|invalid|expired|لا يعمل|غير صالح&quot;]),<br><br>    ((&quot;General Inquiry&quot;, &quot;General&quot;, &quot;Other / Clarification&quot;), [r&quot;.+&quot;]),<br>]</pre><p>Code example Scoring and best match</p><pre><br><br>def score_rules(txt: str):<br>    best = None<br>    best_score = -1<br>    best_hits = []<br><br>    for (main, sub1, sub2), patterns in RULES:<br>        score = 0<br>        hits = []<br>        for pat in patterns:<br>            if re.search(pat, txt, re.I):<br>                score += 1<br>                hits.append(pat)<br><br>        if score &gt; best_score:<br>            best = (main, sub1, sub2)<br>            best_score = score<br>            best_hits = hits<br><br>    return best, best_score, best_hits</pre><p><strong>d)Final classifier function</strong></p><p>This combines everything:</p><ul><li>normalize</li><li>priority detection</li><li>rule scoring</li><li>return labels plus debug info</li></ul><pre>def classify_hierarchy(message: str):<br>    txt = normalize_text(message)<br><br>    fb = detect_failed_booking(txt)<br>    if fb:<br>        main, sub1, sub2 = fb<br>        return main, sub1, sub2, 3, &quot;priority: failed booking with charge&quot;<br><br>    (main, sub1, sub2), score, hits = score_rules(txt)<br><br>    sub2 = refine_sub2(main, sub1, sub2, txt)<br>    return main, sub1, sub2, score, &quot; | &quot;.join(hits)</pre><p><strong>e)Export labeled data and summaries to Excel</strong></p><p>After labeling, I exported:</p><ul><li>the labeled dataset</li><li>summary counts for Main × Sub1</li><li>full summary counts for Main × Sub1 × Sub2</li></ul><pre><br><br>def export_labeled_excel(df: pd.DataFrame, text_col: str, out_path: str):<br>    results = df[text_col].fillna(&quot;&quot;).astype(str).apply(classify_hierarchy)<br><br>    out = df.copy()<br>    out[[&quot;Main Request&quot;, &quot;Sub Request 1&quot;, &quot;Sub Request 2&quot;, &quot;Match Strength&quot;, &quot;Matched Patterns&quot;]] = \<br>        pd.DataFrame(results.tolist(), index=out.index)<br><br>    summary_main_sub1 = (<br>        out.groupby([&quot;Main Request&quot;, &quot;Sub Request 1&quot;], dropna=False)<br>           .size().reset_index(name=&quot;Count&quot;)<br>           .sort_values(&quot;Count&quot;, ascending=False)<br>    )<br><br>    summary_full = (<br>        out.groupby([&quot;Main Request&quot;, &quot;Sub Request 1&quot;, &quot;Sub Request 2&quot;], dropna=False)<br>           .size().reset_index(name=&quot;Count&quot;)<br>           .sort_values(&quot;Count&quot;, ascending=False)<br>    )<br><br>    with pd.ExcelWriter(out_path, engine=&quot;xlsxwriter&quot;) as writer:<br>        out.to_excel(writer, index=False, sheet_name=&quot;Labeled&quot;)<br>        summary_main_sub1.to_excel(writer, index=False, sheet_name=&quot;Summary Main-Sub1&quot;)<br>        summary_full.to_excel(writer, index=False, sheet_name=&quot;Summary Full&quot;)<br></pre><p>This produced a clean, structured Excel file that stakeholders could immediately use for reporting, monitoring, and operational decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*_OzI2vTTEng5v-cgDigYrg.png" /><figcaption>Output Example</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4307b29f1c0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Escape the Lab: My Journey to a Happier Career in Data Analytics]]></title>
            <link>https://medium.com/@carla.zarazir/escape-the-lab-my-journey-to-a-happier-career-in-data-analytics-6a1b778c19e4?source=rss-2ac42e3278fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a1b778c19e4</guid>
            <category><![CDATA[career-change]]></category>
            <category><![CDATA[personal]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[CarlaZr]]></dc:creator>
            <pubDate>Sat, 23 Sep 2023 14:12:42 GMT</pubDate>
            <atom:updated>2023-09-23T14:12:42.878Z</atom:updated>
            <content:encoded><![CDATA[<p>Escape the Lab: My Journey to a Happier Career in Data Analytics</p><p><strong><em>Note</em></strong><em>: It’s been a while since I’ve put pen to paper — or rather, fingers to keyboard — to write an article. Life has been busy, but I’m excited to be back. I have a lot of data science techniques and insights in mind that I can’t wait to share with you. Today, I’m sharing a deeply personal story about a transformative period in my life and career. Here I go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*os4OSOThl8l-2kngQolD5w.jpeg" /></figure><p>Earning a Ph.D. in Food Science and Technology was supposed to be the highlight of my career. Instead, it became the origin of a crisis. Despite dedicating years of my life to intensive study and research, I found myself enveloped in dissatisfaction and regret.</p><p>For six tiring years, I served as a researcher in the food science industry—an occupation that chipped away at my enthusiasm day by day. The routine was monotonous: diving into topics, selecting techniques and chemicals, executing experiments in the lab, and then analyzing the resulting data. The worst part? The work environment was toxic, making the job even harder to bear.</p><p>The COVID-19 pandemic escalated my discontent into a full-blown depression. I felt trapped, and uninspired. But in the midst of this darkness, I discovered that the one bright spot in my job is data analysis.</p><p>Eager to make a career shift, I devoted myself to self-education. I began with Excel tutorials on YouTube, progressed to basic SQL training, and eventually enrolled in DataCamp&#39;s Data Scientist track. Python coding became my new language, projects turned into my new passion, and data science competitions became my new challenge.</p><p>With my newfound skills and goals, I had the courage to leave my toxic job behind. I committed myself to searching for a fulfilling role in the data field—a gamble that felt both terrifying and liberating.</p><p>Then, as if the universe had been waiting for me to make the first move, an amazing opportunity appeared. A company reached out to me, and before I knew it, I was employed in an environment that was as uplifting as it was stimulating.</p><p>Today, I work alongside fantastic colleagues in a positive setting, continually evolving in my role as a data analyst. Every day presents a new learning experience, a new challenge, and most importantly, a newfound appreciation for the work I do.</p><p>So here I am, no longer just a number in a job that made me unhappy. Now, I’m a passionate data analyst, excited to learn more and grow in a career I truly love. This transition has taught me that it&#39;s never too late to change your path and chase what truly makes you happy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a1b778c19e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Power of Data Science in the World of Food: A Tasty Revolution]]></title>
            <link>https://medium.com/@carla.zarazir/the-power-of-data-science-in-the-world-of-food-a-tasty-revolution-6643c0477f5b?source=rss-2ac42e3278fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/6643c0477f5b</guid>
            <category><![CDATA[food]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[CarlaZr]]></dc:creator>
            <pubDate>Tue, 04 Jul 2023 11:25:15 GMT</pubDate>
            <atom:updated>2023-07-04T18:02:34.935Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wulcr5N9ZAzv8PIiPDyofA.jpeg" /><figcaption>Reference: <a href="https://www.hpe.com/us/en/about/doppler.html">Link</a></figcaption></figure><h3><strong>Introduction:</strong></h3><p>In today’s digital age, data science is revolutionizing every industry, and the food industry is no exception. Data scientists are no longer confined to traditional domains like finance and technology; they are now adding flavor to the future of food. With the power of data analytics and machine learning, food companies are able to make more informed decisions, improve operational efficiency, and ultimately deliver better products and experiences to their customers.</p><p>In this article, we’ll explore some exciting applications of data science in the food industry and see how they are transforming the industry.</p><h4>1. Precision Agriculture: Sowing Seeds of Success</h4><p>Data science is revolutionizing agriculture. Farmers are no longer limited to making decisions based on intuition and experience. They can now use data science to make informed decisions about everything from how much water to use to what crops to plant. Data scientists use different sources of data including satellite imagery, weather data, soil samples, and crop yield records, to create complex models that predict crop growth patterns, optimize irrigation, and maximize yields. The result? Sustainable farming practices that reduce resource wastage and sprout efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2NWqaVhiI_aOQU3miqh9hQ.jpeg" /><figcaption>Reference: <a href="https://anolytics.blogspot.com/2020/04/how-to-get-machine-learning-training.html">Link</a></figcaption></figure><h4>2. Food Quality and Safety: Keeping Our Plates Safe</h4><p>Data science plays an increasingly important role in ensuring food safety and quality. Data scientists use a variety of data sources such as sensor data, supply chain information and historical records to analyze potential risks and ensure compliance with food safety regulations. They’re like food detectives, detecting anomalies, tracing the origins of contamination, and banishing potential health hazards. This approach ensures we savor our food with peace of mind.</p><h4>3. Personalized Nutrition: Crafting Meals Just for You</h4><p>Personalized nutrition is the future of eating but is still in its early stages. People can now get meal plans based on their individual needs instead of following a one-size-fits-all diet. Data science has made all this possible.</p><p>For example, data scientists can use data on genetics, dietary habits, and health outcomes to create personalized nutrition plans. These plans can take into account a person’s individual metabolism, allergies, and preferences. They can also be tailored to specific health goals, such as weight loss, muscle gain, or improved heart health. It&#39;s like having a personal chef who knows your taste buds inside out. Bon appétit!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hKKnmQSsNLTH9-1jSc8FLw.png" /><figcaption>Reference: <a href="https://biotherapeuticsinc.com/index.php/page/informatics-technology">Link</a></figcaption></figure><h4>4. Consumer Insights and Demand Forecasting</h4><p>Data science is changing the way food companies understand our cravings and predict our food preferences. By collecting data from social media platforms, online reviews, and purchase data, data scientists can learn about our tastes, buying habits, and the latest food trends. This information can be used by businesses to create effective marketing strategies, promote new products, and manage their ingredients more efficiently.</p><p>For example, data scientists can use social media data to track what people are interested in, what they think of different products and restaurants and what they are actually buying. By combining all of this data, data scientists can create a detailed picture of our food preferences.</p><p>This information can then be used to make better decisions about everything from product development to marketing campaigns. By accurately forecasting demand, they serve up just the right amount, minimizing waste and leaving us hungry for more.</p><h3>Conclusion</h3><p>The combination of data science and food science is revolutionizing the food industry. Data scientists are using data to improve every step of the food production process, from farm to fork.</p><p>The possibilities of data science in the food industry are endless as data availability grows and analytical techniques become more advanced. Data scientists are always creating new ways to make food more sustainable, safe, and personalized. In the future, food will be more than just a delight for our taste buds. It will also be a sustainable, safe, and personalized experience that caters to consumer unique needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6643c0477f5b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Power of K-Means Clustering]]></title>
            <link>https://medium.com/@carla.zarazir/unraveling-the-power-of-k-means-clustering-54404573cab6?source=rss-2ac42e3278fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/54404573cab6</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[k-means]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[CarlaZr]]></dc:creator>
            <pubDate>Tue, 25 Apr 2023 07:44:30 GMT</pubDate>
            <atom:updated>2023-04-26T07:07:59.151Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*FDx_g6wg4ZhakVp30tazEA.jpeg" /></figure><h3>Introduction</h3><p>Unsupervised machine learning involves training a computer to work with unlabeled and unclassified data, enabling the algorithm to operate on this data without supervision. Unlike supervised learning, there is no prior training data available, so the machine’s task is to organize unsorted data based on similarities, patterns, and variations.</p><p>Clustering is an unsupervised machine learning technique which aims to identify subgroups in datasets. One popular algorithm for clustering is <em>KMeans,</em> which divides observations into a number of groups (k clusters). This algorithm is often applied in fields such as market segmentation where the objects within the same cluster are more similar to one another than to those outside the cluster.</p><p>I’ll be implementing the KMeans clustering technique on the fish dataset.</p><h4>KMeans clustering in Python</h4><p>Import the relevant libraries and load the data</p><pre># Import necessary packages<br>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br><br>from sklearn.cluster import KMeans<br>from sklearn.pipeline import make_pipeline<br>from sklearn.preprocessing import StandardScaler<br><br><br># Load the fish data from the CSV file and store it in the &#39;fish&#39; variable<br>fish = pd.read_csv(&#39;datasets/fish.csv&#39;, header=None)</pre><p>KMeans, as implemented in the scikit-learn library, does not directly accept DataFrames as input. Hence, it is essential to transform the data into an array that contains only the numerical values of the features that needed to be clustered.</p><pre># Convert the fish data to a numpy array and store the data points (columns 1-5) in the &#39;points&#39; variable<br># and the fish species (column 0) in the &#39;Species&#39; variable<br>arr = fish.to_numpy()<br>points = arr[:, 1:6]<br>Species = arr[:, 0]</pre><h4>Determine the best number of clusters</h4><p>Identifying the appropriate number of clusters is essential in clustering analysis. Several techniques are available, but one commonly used method is the <em>Elbow method</em>, which is a heuristic approach to determining the optimal number of clusters.</p><pre># Define a range of values for k, the number of clusters to use in k-means clustering<br>ks = range(1, 9)<br><br># Create an empty list to store the inertia values for each k<br>inertias = []<br><br># Iterate over the range of k values<br>for k in ks:<br>    # Create a KMeans object with k clusters and fit it to the data points<br>    kmeans = KMeans(n_clusters=k)<br>    kmeans.fit(points)<br>    <br>    # Append the inertia value of the k-means clustering to the &#39;inertias&#39; list<br>    inertias.append(kmeans.inertia_)<br>    <br># Plot the values of k versus the corresponding inertia values as a line plot with markers<br>plt.plot(ks, inertias, marker=&#39;o&#39;)<br>plt.xlabel(&#39;Number of clusters&#39;)<br>plt.ylabel(&#39;Inertia&#39;)<br>plt.title(&#39;Elbow method for fish dataset&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*sK1t662QiSp2hePCk2_FLw.png" /></figure><p>The Elbow Curve shows the sum of squared distances of each point from the clusters (This can be found out using <strong>kmeans.inertia_</strong> method) versus the number of clusters. According to the Elbow curve, the ideal number of clusters should be equal to 4.</p><h4>Perform K-Means clustering with optimal <em>k</em>.</h4><p>The following steps show how to perform k-means clustering on the dataset using the optimal value for <em>k</em> of 4:</p><pre># Create a StandardScaler object to standardize the data points<br>scaler = StandardScaler()<br><br># Create a KMeans object with 4 clusters<br>km = KMeans(n_clusters=4, random_state=45)<br><br># Create a pipeline to chain the scaler and KMeans objects together<br>pipeline = make_pipeline(scaler, km)<br><br># Fit the pipeline to the data points<br>pipeline.fit(points)<br><br># Predict the labels of the data points using the pipeline<br>labels = pipeline.predict(points)<br><br># Create a DataFrame to store the predicted labels and the true fish species<br>df = pd.DataFrame({&#39;Labels&#39;: labels, &#39;Species&#39;: Species})<br><br># Print the DataFrame to see the predicted labels and true fish species side-by-side<br>print(df)</pre><p>First of all, a <strong>StandardScaler</strong> object is created: This object is used to standardize the data points, meaning that it scales the features so that they have a mean of 0 and a standard deviation of 1. This is important because KMeans is a distance-based algorithm and can be sensitive to the scale of the features. By standardizing the data, you ensure that the algorithm treats all features equally.</p><p>Then, a <strong>KMeans</strong> object with 4 clusters is created: This object will perform the KMeans clustering algorithm on the data points, attempting to find 4 distinct clusters.</p><p>Also, the make_pipeline function is used to chain the StandardScaler and KMeans objects together. This enables a more efficient workflow by allowing the data to be processed by both the scaler and the KMeans algorithm in one step.</p><p>After fitting the pipeline to the data points, the labels of the data points are predicted with the predict method. In addition, this method assigns each data point to one of the 4 clusters.</p><p>A DataFrame is created using Pandas to store the predicted labels and the true fish species side-by-side. This can be useful for comparing the results of the clustering algorithm to the actual species labels, which can help evaluate the performance of the algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9b4W9_lg_wuaAY-jzBhqaQ.png" /></figure><p>Finally Cross-tabulation is used to summarize the relationship between two categorical variables, in this case, the predicted labels and the true fish species. By displaying the counts of data points in each combination of predicted label and true species, we can gain insight into how well our clustering algorithm is grouping similar data points together.</p><pre># Create a cross-tabulation table to count the number of data points <br># in each combination of predicted label and true fish species<br><br>ct = pd.crosstab(df[&#39;Labels&#39;], df[&#39;Species&#39;])<br>print(ct)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UHgYitniy12PtX5bKXnYyA.png" /></figure><p>The cross-tabulation table provides valuable insights into the clustering results, revealing how the KMeans algorithm has grouped the fish species based on their shared characteristics. The table highlights the distinct nature of Pike fish species, while also showcasing the similarities between other species such as Roach, Smelt, and Bream. This analysis demonstrates the importance of clustering algorithms in uncovering hidden patterns within data and can serve as a foundation for further exploration or decision-making in various applications, and different fields.</p><p>In conclusion, KMeans clustering is a simple yet effective technique for clustering data points into groups based on their similarity. It is widely used in various fields such as marketing, biology, and image processing. Understanding the concepts of k-means clustering and how to apply it in practice can be a valuable skill for data analysts and machine learning practitioners.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54404573cab6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scraping Lebanese food off the web]]></title>
            <link>https://medium.com/@carla.zarazir/scraping-lebanese-food-off-the-web-61db6a4e414a?source=rss-2ac42e3278fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/61db6a4e414a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[food]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[CarlaZr]]></dc:creator>
            <pubDate>Sat, 15 Apr 2023 06:19:19 GMT</pubDate>
            <atom:updated>2023-04-15T13:24:39.394Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ik9nuNCIjrWHI52SD88j4Q.jpeg" /></figure><p>Are you tired of spending a copious amount of time just copying and pasting data from websites for your project? Well, you’re not alone. To streamline the traditionally tedious process of obtaining data from websites, an increasing number of data scientists are now adopting a highly efficient method, known as <em>Web Scraping</em>.</p><p><strong>What is Data Scraping?</strong></p><p><em>Data Scraping</em> is an automated method of extracting data from websites that can save you time and effort. However, accessing this data can be a challenge, as many websites do not provide direct means of accessing their content.</p><p>This article covers the process of scraping Lebanese recipes and their corresponding nutritional facts from a specific website, including selecting appropriate tools and libraries, identifying the relevant data, and organizing it into a usable format.</p><h4>1- Packages and tools</h4><p>I used the following packages to conduct web scraping quickly and efficiently: Pandas, Requests, and BeautifulSoup.</p><p><strong>Pandas </strong>is a powerful library that offers data manipulation and analysis tools, making it easy to organize and analyze scraped data.</p><p><strong>Requests</strong> is another Python library that simplifies sending HTTP requests and handling the responses from websites.</p><p>Finally,<strong> BeautifulSoup</strong> is a Python package that helps in extracting data in a structured format by parsing HTML and XML documents.</p><pre>import pandas as pd<br>import requests<br>from bs4 import BeautifulSoup</pre><h4>2- Inspect option</h4><p>The <em>Inspect</em> option enables data scientists to explore the structure of a website’s HTML code. The option can be found in most web browsers, and it allows users to hover over different elements on a website to view their corresponding HTML code. <em>Inspect</em> can identify specific tags, classes, and IDs that contain the data to be scrapped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AHHmxkW_Tb4YkPybJaZ8MQ.png" /></figure><h4>3- The code</h4><pre># Initialize empty lists to store scraped data<br>name=[]<br>portion=[]<br>table =[]<br><br># Loop through the website pages to scrape data from each one<br>for i in range(1,11):<br>    # Set the URL for the current page to scrape<br>    url = f&quot;https://www.myfitnesspal.com/nutrition-facts-calories/lebanese/{i}&quot;<br><br>    # Send an HTTP request to the URL and retrieve its HTML content<br>    request = requests.get(url)<br><br>    # Use BeautifulSoup to parse the HTML content and extract relevant data<br>    soup = BeautifulSoup(request.content, &#39;html.parser&#39;)<br><br>    # Find all sections of the HTML content that contain the relevant data<br>    sections = soup.find_all(&#39;div&#39;, class_=&#39;MuiBox-root css-rr9uo8&#39;)<br><br>    # Loop through each section and extract the recipe name, portion size, and nutritional information<br>    for s in sections:<br>        titles = s.find(&#39;p&#39;, class_=&#39;MuiTypography-root MuiTypography-body1 css-qumjp8&#39;).text<br>        portion_size=s.find(&#39;p&#39;, class_=&#39;MuiTypography-root MuiTypography-body1 css-w1kjmb&#39;).text<br>        nutrition_table= s.find(&#39;div&#39;, class_=&#39;MuiBox-root css-2imjyh&#39;).text<br><br>        # Add the scraped data to their respective lists<br>        name.append(titles)<br>        portion.append(portion_size)<br>        table.append(nutrition_table)</pre><p>With this code, I was able to extract data from multiple pages and collect information on recipe names, portion sizes, and nutritional information.</p><p>I used the <em>Requests</em> to make HTTP requests and retrieve the HTML content from each page, then with the help of <em>BeautifulSoup</em> I parsed the HTML and extracted the relevant data. I also used a loop to iterate through multiple pages of the website, ensuring that I captured all the data.</p><h4>4- Cleaning and organizing the data</h4><p>Cleaning and organizing the data is an essential step before analyzing or visualizing it. Web scraping can sometimes produce unstructured or messy data, which can be difficult to interpret.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w72SHqoXPaFQzP6WPaDzUQ.png" /></figure><p>Therefore, cleaning and organizing the data is necessary to make it usable and meaningful. This involves removing any unwanted characters, splitting columns into multiple columns, dropping unnecessary columns, and reorganizing the data in a logical way. The final result greatly helps in creating a more structured and organized dataset that is easier to analyze.</p><pre># Create a Pandas DataFrame using the scraped data<br>df= pd.DataFrame({&#39;names&#39;:name, &#39;portion size&#39;:portion, &#39;nutrients&#39;:table})<br><br># Split the &#39;nutrients&#39; column into four separate columns based on the delimiter &quot;•&quot;<br>df[[&#39;calories&#39;, &#39;carbs&#39;, &#39;fat&#39;, &#39;protein&#39;]] = df[&#39;nutrients&#39;].str.split(&#39;•&#39;, expand=True)<br><br># Split the &#39;portion size&#39; column into two separate columns based on the delimiter &quot;,&quot;<br>df[[&#39;category&#39;, &#39;portion_size&#39;]] = df[&#39;portion size&#39;].str.split(&#39;,&#39;, expand=True)<br><br># Remove unwanted characters and units of measurement from the &#39;calories&#39; column<br>df[&#39;calories&#39;] = df[&#39;calories&#39;].str.strip(&#39;Calories:&#39;)<br><br># Remove unwanted characters and units of measurement from the &#39;carbs&#39; column<br>df[&#39;carbs&#39;] = df[&#39;carbs&#39;].str.strip(&#39;Carbs:&#39;)<br>df[&#39;carbs&#39;] = df[&#39;carbs&#39;].str.strip(&#39;g&#39;)<br><br># Remove unwanted characters and units of measurement from the &#39;fat&#39; column<br>df[&#39;fat&#39;] = df[&#39;fat&#39;].str.strip(&#39;Fat:&#39;)<br>df[&#39;fat&#39;] = df[&#39;fat&#39;].str.strip(&#39;g&#39;)<br><br># Remove unwanted characters and units of measurement from the &#39;protein&#39; column<br>df[&#39;protein&#39;] = df[&#39;protein&#39;].str.strip(&#39;Protein:&#39;)<br>df[&#39;protein&#39;] = df[&#39;protein&#39;].str.strip(&#39;g&#39;)<br><br># Drop the &#39;nutrients&#39; and &#39;portion size&#39; columns from the DataFrame<br>df=df.drop([&#39;nutrients&#39;,&#39;portion size&#39;], axis=1)<br><br># Reorder the columns in the DataFrame for ease of use<br>df= df[[&#39;names&#39;, &#39;category&#39;, &#39;portion_size&#39;, &#39;calories&#39;, &#39;carbs&#39;, &#39;fat&#39;, &#39;protein&#39;]]<br><br>recipe = df.copy()</pre><p>The next step in data processing is to identify and remove any duplicate values in the dataset by using the <strong>duplicated()</strong> method in <em>Pandas</em>. By applying this method to a specific column in the <em>DataFrame</em>, such as ‘names’, a Boolean array indicating which rows contain duplicate values is obtained. The <strong>drop_duplicates() </strong>method is then applied to remove these duplicate rows.</p><pre>dup = recipe[&#39;names&#39;].duplicated()<br>recipe.drop_duplicates(subset=&#39;names&#39;, inplace =True)</pre><h4>5- <strong>Output</strong></h4><p>The <strong>to_csv() </strong>method enables saving data stored in a <em>DataFrame</em> object to a CSV (Comma-Separated Values) file format. This makes it easy to share and exchange data with others, as well as to import it into other applications for further analysis or visualization.</p><pre>recipe.to_csv(&#39;Lebanese recipes.csv&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*l97pXwdzH1-g91L0mA2bEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*NdYdFFJSoQfyytzkpWSq9g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61db6a4e414a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A/B Testing in Python: A Case Study Comparing Two Marketing Campaigns]]></title>
            <link>https://medium.com/@carla.zarazir/a-b-testing-in-python-a-case-study-comparing-two-marketing-campaigns-eae6883e72cc?source=rss-2ac42e3278fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/eae6883e72cc</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[ab-testing]]></category>
            <dc:creator><![CDATA[CarlaZr]]></dc:creator>
            <pubDate>Tue, 28 Mar 2023 19:23:54 GMT</pubDate>
            <atom:updated>2023-03-30T07:18:52.678Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*JRrN55Ih7CKVX4nguEeFVg.png" /></figure><h3><strong>I. Introduction</strong></h3><p>A/B testing, commonly referred to as split testing, is an essential instrument for businesses looking to make intelligent decisions grounded in user behavior. This approach enables organizations to adopt data-backed strategies, enhance user engagement, and minimize risks associated with changes.</p><h4>Objective of the article</h4><p>In this article, I evaluated the outcomes of two marketing approaches to identify the most effective strategy for upcoming promotional campaigns.</p><p>In the <a href="https://statso.io/a-b-testing-case-study/">dataset</a> I analyzed, there were two marketing campaigns — a control group and a test group. The main goal is to compare the click-through rates (CTR) of the two campaigns to determine which one was better at engaging users.</p><h3>II. A/B Testing using Python</h3><p>Let’s import the relevant libraries and load the data</p><pre>#Import the relevant libraries<br>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br>#Load the control and test data<br>control = pd.read_csv(&#39;control_group.csv&#39;, sep=&#39;;&#39;)<br>test = pd.read_csv(&#39;test_group.csv&#39;, sep = &#39;;&#39;)<br>control.head()<br>test.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YRb3A7SuijpfeSiUvKnv4Q.png" /><figcaption>Control Campaign Dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L7o7ZlncfOCu6BJ-hPfhAA.png" /><figcaption>Test Campaign Dataset</figcaption></figure><h4>Data pre-processing</h4><p>The dataset contains some gaps in the data, requiring attention to ensure accurate analysis. It is important to address the missing values through appropriate techniques, which can help maintain the integrity of the dataset. Additionally, converting date entries into a standardized datetime format is essential for streamlining data manipulation and facilitating seamless comparisons and calculations.</p><pre>#Convert date to datetime<br>control[&#39;Date&#39;] = pd.to_datetime(control[&#39;Date&#39;])<br>test[&#39;Date&#39;] = pd.to_datetime(test[&#39;Date&#39;])<br><br>#Fill missing values with mean<br>control = round(control.fillna(control.mean(numeric_only=True)))<br><br>#Concat both dataframes<br>data = pd.concat([control, test], axis =0)<br><br>data.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kRfB_C4ElXxShW2uXIWlMw.png" /></figure><p>For a balanced comparison, I will examine the dataset to ensure that both campaigns have an equal number of samples using the value_counts() function:</p><pre>#Count the campaign samples<br>campaigns = data[&#39;Campaign Name&#39;].value_counts()<br>print(campaigns)</pre><pre>Output:<br>Control Campaign    30<br>Test Campaign       30<br>Name: Campaign Name, dtype: int64</pre><h4><strong>Finding the best Marketing Strategy</strong></h4><p>Initiating the A/B testing process involves a thorough examination of the relationship between different variables collected from both campaigns and the associated expenditure.</p><pre># Set up the figure and axes<br>fig, axes = plt.subplots(nrows=3, ncols=3, figsize=(18, 12), sharex=True, gridspec_kw={&#39;height_ratios&#39;: [1, 1, 2]})<br><br># Merge the bottom two subplots<br>gs = axes[2, 0].get_gridspec()<br>for ax in axes[2, :]:<br>    ax.remove()<br><br># Create the larger plot at the bottom<br>ax_big = fig.add_subplot(gs[2, :])<br><br># Get unique campaign names<br>campaign_names = data[&#39;Campaign Name&#39;].unique()<br><br># Create the plots<br>for campaign in campaign_names:<br>    campaign_data = data[data[&#39;Campaign Name&#39;] == campaign]<br>    sns.regplot(ax=axes[0,0], data=campaign_data, x=&#39;Spend [USD]&#39;, y=&#39;Reach&#39;, ci=None, label=campaign, scatter_kws={&#39;alpha&#39;: 0.5})<br>    sns.regplot(ax=axes[0,1], data=campaign_data, x=&#39;Spend [USD]&#39;, y=&#39;# of Website Clicks&#39;, ci=None, label=campaign, scatter_kws={&#39;alpha&#39;: 0.5})<br>    sns.regplot(ax=axes[0,2], data=campaign_data, x=&#39;Spend [USD]&#39;, y=&#39;# of Searches&#39;, ci=None, label=campaign, scatter_kws={&#39;alpha&#39;: 0.5})<br>    sns.regplot(ax=axes[1,0], data=campaign_data, x=&#39;Spend [USD]&#39;, y=&#39;# of View Content&#39;, ci=None, label=campaign, scatter_kws={&#39;alpha&#39;:0.5})<br>    sns.regplot(ax=axes[1,1], data=campaign_data, x=&#39;Spend [USD]&#39;, y=&#39;# of Add to Cart&#39;, ci=None, label=campaign, scatter_kws={&#39;alpha&#39;:0.5})<br>    sns.regplot(ax=axes[1,2], data=campaign_data, x=&#39;Spend [USD]&#39;, y=&#39;# of Purchase&#39;, ci=None, label = campaign, scatter_kws={&#39;alpha&#39;:0.5})<br>    sns.regplot(ax=ax_big, data=campaign_data, x=&#39;Spend [USD]&#39;, y=&#39;# of Impressions&#39;, ci=None, label=campaign, scatter_kws={&#39;alpha&#39;: 0.5})<br><br># Add legend<br>handles, labels = axes[0,0].get_legend_handles_labels()<br>fig.legend(handles, labels, loc=&#39;upper right&#39;)<br><br># Set the figure title<br>fig.suptitle(&#39;Correlation plots&#39;)<br><br># Display the plots<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3-kfgJQsVxn8z8-Gc5g_BA.png" /><figcaption>Correlation plots between the variables and the amount spent on both campaign</figcaption></figure><p>The analysis reveals that the most prominent and logical correlations with the ‘amount spent’ variable are found in two key metrics: reach and the number of impressions. To gain further insights into the performance of each campaign, it is essential to compute and visualize the Click-Through Rate (CTR) using these variables.</p><pre>data[&#39;CTR&#39;] = round((data[&#39;# of Website Clicks&#39;]/ data[&#39;# of Impressions&#39;]), 2)<br><br>ctr = data.groupby([&#39;Campaign Name&#39;, &#39;Date&#39;])[&#39;CTR&#39;].mean().to_frame()<br><br>#Create a pivot table<br>ctr_pivot= ctr.pivot_table(columns= &#39;Campaign Name&#39;, index=&#39;Date&#39;, <br>values = &#39;CTR&#39;)<br><br>#Plot the data as a line plot<br>ctr_pivot.plot(kind=&#39;line&#39;, figsize=(15, 5), style=&#39;o--&#39;, markersize=6)<br><br>#add title and axis labels<br>plt.title(&#39;Monthly website clicks by Campaign&#39;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.ylabel(&#39;Click-through rates&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fP57YTb4L5ZkdJVBT3uGgA.png" /></figure><p>Up next, the monthly expenditure is determined for each campaign:</p><pre>#amount spent by each campaign monthly<br>month=round(data.groupby([&#39;Campaign Name&#39;, &#39;Date&#39;], <br>as_index=False)[[&#39;Spend [USD]&#39;]].mean())<br><br>#Sort the monthly amount spent descending<br>month_sorted = month.sort_values(&#39;Spend [USD]&#39;, ascending=False)<br><br>#Calulate the mean of the amount spent by month of each campaign<br>month_v_spend = month_sorted.groupby([&#39;Campaign Name&#39;, &#39;Date&#39;])[&#39;Spend [USD]&#39;].mean().to_frame()<br><br>#Create a pivot table to organize data by Campaign Name and Month<br>pivot_table = month_v_spend.pivot_table(values=&#39;Spend [USD]&#39;, index=&#39;Date&#39;, columns=&#39;Campaign Name&#39;)<br><br>#Plot the data as a line plot<br>pivot_table.plot(kind=&#39;line&#39;, figsize=(15, 5), style=&#39;o--&#39;, markersize=6)<br><br>#add title and axis labels<br>plt.title(&#39;Amount spent on each Campaign&#39;)<br>plt.xlabel(&#39;Month&#39;)<br>plt.ylabel(&#39;Spend [USD]&#39;)<br>plt.show()<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-t4Ddd7vIRbeAZZ5o01kGg.png" /></figure><p>The results show that between month 01 and month 07, the amount spent does not impact the CTR for the control group, while it does lead to an increase in the test campaign. However, in the months 08 and 09, the amount spent on each campaign has a significant effect on the CTR, particularly for the test campaign.</p><h4>Hypothesis testing</h4><p>During hypothesis testing, two opposing hypotheses are considered: the null hypothesis (H0) and the alternative hypothesis (H1). H0 assumes there is no important difference between the control and test groups. However, H1 claims there is a significant difference between them.</p><p>H0 : Test — Control = 0</p><p>H1: Test — Control &gt; 0</p><p>Given the relatively small sample sizes of both the control and test groups, each containing only 30 observations, it’s important to carefully consider the assumptions of the statistical methods used for analysis. One common assumption in many statistical tests is the normality of the data. To check whether this assumption is met, a normality test like the Shapiro-Wilk test is performed<strong>.</strong></p><h4>The Shapiro-Wilk test</h4><pre>from scipy.stats import shapiro<br><br>def test_normality(data):<br>    # Perform Shapiro-Wilk test<br>    stat, p = shapiro(data)<br><br>    # Print the test results<br>    print(&#39;Shapiro-Wilk test statistic:&#39;, stat)<br>    print(&#39;p-value:&#39;, p)<br><br>    # Interpret the test results<br>    alpha = 0.05<br>    if p &gt; alpha:<br>        print(&#39;The data looks normally distributed (fail to reject H0)&#39;)<br>    else:<br>        print(&#39;The data does not look normally distributed (reject H0)&#39;)<br><br># Test normality of Control Campaign CTR<br>print(&#39;for the control campaign:&#39;)<br>test_normality(control_ctr)<br><br># Test normality of Test Campaign CTR<br>print(&#39;\nfor the test campaign:&#39;)<br>test_normality(test_ctr)</pre><pre>for the control campaign:<br>Shapiro-Wilk test statistic: 0.9384782314300537<br>p-value: 0.08277233690023422<br>The data looks normally distributed (fail to reject H0)<br><br>for the test campaign:<br>Shapiro-Wilk test statistic: 0.834705114364624<br>p-value: 0.00030149263329803944<br>The data does not look normally distributed (reject H0)</pre><p>According to the outcomes of the Shapiro-Wilk test, the CTR data for the Control Campaign appears to follow a normal distribution, whereas the Test Campaign’s CTR data does not. As a result, a non-parametric test that doesn’t rely on the assumption of normality is employed to compare the CTRs of both the Control and Test Campaigns.</p><p>In addition, it is preferable to perform a bootstrap distribution to obtain more reliable and accurate statistical insights. Bootstrapping is a resampling technique that involves generating multiple samples from the original dataset by drawing observations with replacement. This method creates a large number of samples, which can then be used to estimate the distribution of a test statistic, such as the mean difference between the two groups.</p><p>Employing the bootstrap distribution assists in addressing the challenges tied to small sample sizes, while increasing the validity of the drawn statistical conclusions. Moreover, bootstrapping offers a method for determining confidence intervals concerning the difference between control and test groups, which can help determine if the observed differences are statistically significant.</p><pre>#bootstrap distribution<br>boot_diff = []<br>for i in range(1000):<br>    boot_control = control_ctr.sample(frac=1, replace=True)<br>    boot_test = test_ctr.sample(frac=1, replace=True)<br>    boot_diff.append(boot_test.mean() - boot_control.mean())<br><br>#Convert the list into a DataFrame<br>boot_diff_df = pd.DataFrame(boot_diff)<br><br>#Plot a kde graph of the bootstrap differences<br>boot_diff_df.plot(kind=&#39;kde&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*iSO7hR3z2NDDbhqSc-n-qg.png" /></figure><p>A bootstrap distribution of the disparities between the control and test groups’ click-through rates (CTR) was generated, followed by the creation of a Kernel Density Estimation (KDE) graph for the bootstrap differences. This visualization facilitates the examination of the distribution’s shape, central tendency, and dispersion, simplifying the process of detecting patterns and trends.</p><p>Subsequently, a 95% confidence interval was computed.</p><pre>#Calculating a 95% confidence interval<br>confidence_interval = pd.Series(boot_diff).quantile([0.025, 0.975])<br>print(confidence_interval)</pre><pre>Output:<br>0.025    0.026000<br>0.975    0.077342<br>dtype: float64</pre><p>The result shows that the 95% confidence interval for the difference in the mean of the CTR variable between the control and test groups is (0.026, 0.077). In other words, there is a 95% certainty that the performance difference between the test and control campaigns falls within 2.6% and 7.7%.</p><p>Since the confidence interval does not include zero, it suggests that the observed differences in CTR between the control and test groups are statistically significant. In addition, the confidence interval indicates that the test campaign outperforms the control campaign by at least 2.6% and as much as 7.7% in terms of CTR.</p><h4>Mann-Whitney U test <strong>and p-value</strong></h4><p>According to the the Shapiro-Wilk test, the data is not normally distributed. In this case, a non-parametric test such as the Mann-Whitney U test would be more appropriate to compare the CTR between the Control Campaign and Test Campaign.</p><pre># Create a DatFrame with two columns Campaign name and CTR<br>df = data[[&#39;Campaign Name&#39;, &#39;CTR&#39;]]<br><br># Reshape the DataFrame to a wide format<br>df_wide = df.pivot(columns = &#39;Campaign Name&#39;, values=&#39;CTR&#39;)<br><br>import pingouin as pg<br><br># Conduct a Mann-Whitney U test<br>test_results = pg.mwu(x=df_wide[&#39;Test Campaign&#39;],<br>y=df_wide[&#39;Control Campaign&#39;], alternative=&#39;greater&#39;)<br><br>print(test_results)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*fbxeDsdAEBR8qOQ2VOXXUg.png" /><figcaption>Mann-Whitney U test</figcaption></figure><p>The p-value (0.000109) is very low, which indicates that there is strong evidence to reject the null hypothesis in favor of the alternative hypothesis that the test group values are greater than the control group values. In other words, the findings indicate that the test campaign is performing significantly better than the control campaign in terms of CTR.</p><h3>III. Conclusion</h3><p>In conclusion, this analysis demonstrates the importance of carefully considering the assumptions of statistical tests when analyzing data, particularly when dealing with small sample sizes. By performing appropriate tests and using techniques such as bootstrapping, more accurate and meaningful results can be obtained. These findings can provide valuable insights for decision-making in various fields such as marketing, finance, and healthcare.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eae6883e72cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing Covid-19 Vaccines Across the Globe: A Python-Based Data Analysis Approach]]></title>
            <link>https://medium.com/@carla.zarazir/analyzing-covid-19-vaccines-across-the-globe-a-python-based-data-analysis-approach-6c8239451934?source=rss-2ac42e3278fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c8239451934</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[covid19]]></category>
            <dc:creator><![CDATA[CarlaZr]]></dc:creator>
            <pubDate>Mon, 06 Mar 2023 21:56:41 GMT</pubDate>
            <atom:updated>2023-03-07T10:55:49.142Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uU2HBg7mPaT0kdII2TbkJw.jpeg" /><figcaption>Via Unsplash</figcaption></figure><p>The COVID-19 pandemic has changed the world in ways that were previously unimaginable. Since its emergence in late 2019, the virus has infected millions of people worldwide and caused hundreds of thousands of deaths. While many countries have implemented various measures such as social distancing and lockdowns to control the spread of the virus, the development of vaccines has been a critical step in the fight against COVID-19. In this article, I will walk you through the analysis of using Covid-19 vaccines across different countries in Python.</p><p>Although several vaccines have been developed to fight COVID-19 and none have been shown to be 100% effective. Nonetheless, most manufacturers claim that their vaccines are highly effective in providing immunity against the virus. This has prompted many countries to aim for high vaccination rates, reducing their dependence on any single vaccine.</p><h3>COVID-19 analysis with Python</h3><p>For the purpose of analyzing COVID-19 vaccines, I will be working with a <a href="https://www.kaggle.com/datasets/gpreda/covid-world-vaccination-progress">dataset</a> sourced from Kaggle.</p><p>The steps required for Data Analysis:</p><ul><li>Importing all the relevant packages</li><li>Loading the Data</li><li>Transforming data (data cleaning /data wrangling)</li><li>Exploratory data analysis and visualization</li></ul><h4>Step 1 — Importing necessary libraries:</h4><p>Before we start analyzing data, we need to import the necessary libraries. Some common libraries I used in this analysis include NumPy, Pandas, Matplotlib, Seaborn and Plotly.</p><ul><li><strong>NumPy</strong> is a library for numerical computing in Python. It provides powerful tools for working with arrays and matrices, which are commonly used in data analysis.</li><li><strong>Pandas </strong>is a library for data manipulation and analysis in Python. With Pandas, data is loaded, cleaned, transformed and analyzed quickly and easily. It also provides convenient methods for grouping data, filtering data, and handling missing data.</li><li><strong>Matplotlib and Seaborn </strong>are libraries for data visualization in Python. Matplotlib is a general-purpose plotting library that allows you to create a wide range of visualizations, including line plots, scatter plots, bar plots, histograms, and more. Seaborn is a higher-level library that builds on top of Matplotlib, providing additional functionality for statistical data visualization. It includes tools for creating heatmaps, pair plots, and other types of advanced visualizations.</li><li><strong>Plotly</strong> is a library for creating interactive visualizations in Python. It provides tools for creating a wide range of charts and graphs, including line charts, scatter plots, bar charts, and 3D surface plots.</li></ul><pre>#import all relevant libraries<br>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import plotly.express as px</pre><h4>Step 2 — Loading the data</h4><p>Once the libraries are imported, we need to load the data (CSV file) into out Python environment.</p><pre>data = pd.read_csv(&#39;country_vaccinations.csv&#39;)<br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwec-bFmrZ_3FhGtR2Vf8Q.png" /></figure><h4>Step 3 — Data Wrangling</h4><p>Data wrangling, also known as data munging, is the process of cleaning, transforming, and preparing raw data for analysis. It is an important step in the data analysis process as it ensures that the data is accurate, consistent, and ready for analysis. Some common tasks involved in data wrangling:</p><ul><li>D<strong>ata cleaning </strong>involves identifying and handling missing data, removing duplicates, and correcting any errors in the data.</li><li><strong>Data transformation </strong>involves converting the data into a format that is suitable for analysis. This may involve scaling the data or converting categorical variables to numerical variables.</li><li><strong>Data reduction </strong>involves reducing the size of the dataset by removing unnecessary variables or observations.</li></ul><pre># Drop irrelevant variables<br>data = data.drop([&#39;source_name&#39;, &#39;source_website&#39;], axis=1)<br><br>#Identifying and treating missing values<br>data.isnull().sum()<br>data=data.fillna(0)<br><br>#Convert the date to datetime<br>data[&#39;date&#39;] = pd.to_datetime(data.date)<br><br></pre><p><strong>Step 4 — Exploratory Data Analysis and Visualization</strong></p><p>Before we begin analyzing the vaccines taken by countries, let’s first take a closer look at this data.</p><p>I used the value_counts() method to count the number of vaccines in each country.</p><pre>data.country.value_counts()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*86aOp4Q-QrPgdKKKk7CT4Q.png" /></figure><p>In the above dataset, countries of the United Kingdom (England, Scotland, Wales, and Northern Ireland) are listed separately, even though they are part of the same country. So, let’s fix it and aggregate the data for the United Kingdom as a whole.</p><pre>def filter_countries(data):<br>    # Create a list of country names to exclude<br>    excluded_countries = [&quot;England&quot;, &quot;Scotland&quot;, &quot;Wales&quot;, &quot;Northern Ireland&quot;]<br>    <br>    # Use the apply method to create a boolean mask of countries to keep<br>    mask = ~data.country.isin(excluded_countries)<br>    <br>    # Apply the mask to the DataFrame to filter out the excluded countries<br>    filtered_data = data[mask]<br>    <br>    return filtered_data</pre><p>This function is filtering the DataFrame to exclude any rows where the country is not one of the four countries that make up the United Kingdom (i.e., England, Scotland, Wales, or Northern Ireland).</p><p>Let’s explore the vaccines that are included in this dataset. The <strong>value_counts()</strong> method is used to obtain a count of unique values in the ‘vaccines’ column of the DataFrame.</p><pre>data.vaccines.value_counts()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yQmi76JT4tfUsYuThre-zw.png" /></figure><p>I will now create a new DataFrame by selecting only the vaccine and country columns, in order to investigate which vaccine is administered in each country.</p><pre>def vaccine_countries(data):<br>    # Create an empty dictionary to hold vaccine-country mappings<br>    vaccine_dict = {}<br>    <br>    # Loop over unique vaccine names in the DataFrame<br>    for x in df.vaccines.unique():<br>        # Extract the country names where this vaccine is being used<br>        countries = set(data.loc[data.vaccines == x, &quot;country&quot;])<br>        # Add the vaccine-country mapping to the dictionary<br>        vaccine_dict[x] = countries<br>    <br>    # Loop over the dictionary items and print the vaccine-country mappings<br>    for vaccine, countries in vaccine_dict.items():<br>        print(f&quot;{vaccine}:&gt;&gt;{countries}&quot;)<br>        <br>vaccine_countries(df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XM5iKqvD6Y5jIHST3Fa4AA.png" /></figure><p>Let’s visualize this data to examine the specific vaccine combinations that each country is using.</p><pre>import plotly.express as px<br>import plotly.offline as py<br>import plotly.io as pio<br><br># Create the choropleth map<br>vaccine_map = px.choropleth(data, locations = &#39;iso_code&#39;, color = &#39;vaccines&#39;)<br><br># Update the layout<br>vaccine_map.update_layout(height=300, margin={&quot;r&quot;:0,&quot;t&quot;:0,&quot;l&quot;:0,&quot;b&quot;:0})<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*ReJKGm-a3HxpyKhaHQVjWw.png" /></figure><p>Conclusion:</p><p>Based on the analysis of Covid-19 vaccine distribution across different countries, it can be concluded that there is a significant variation in the types of vaccines being used. The Pfizer-BioNTech vaccine is the most commonly used vaccine across the world, followed by the AstraZeneca vaccine. However, some countries have opted for other vaccines such as Sinovac, Sinopharm, and Sputnik V.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c8239451934" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>