<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yash Khandelwal on Medium]]></title>
        <description><![CDATA[Stories by Yash Khandelwal on Medium]]></description>
        <link>https://medium.com/@yashkhandelwal07?source=rss-abe69fbfb436------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zOcw5YFF0VRENxNfV9DTqQ.jpeg</url>
            <title>Stories by Yash Khandelwal on Medium</title>
            <link>https://medium.com/@yashkhandelwal07?source=rss-abe69fbfb436------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:27:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yashkhandelwal07/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Explaining Every Rupee: How We Built Reliable LLM Support Bots for Delivery Partners]]></title>
            <link>https://medium.com/swiggy-bytes/explaining-every-rupee-how-we-built-reliable-llm-support-bots-for-delivery-partners-066c4f23e875?source=rss-abe69fbfb436------2</link>
            <guid isPermaLink="false">https://medium.com/p/066c4f23e875</guid>
            <category><![CDATA[customer-experience]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yash Khandelwal]]></dc:creator>
            <pubDate>Fri, 29 May 2026 11:28:46 GMT</pubDate>
            <atom:updated>2026-05-29T12:40:06.399Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l67BkTa6fAre2B1dtoJa4Q.png" /></figure><h3>The Problem</h3><p>A Delivery Executive (DE) finishes their day, opens the app, and expects a clear breakdown of what they earned. Instead, they often see a payout that does not fully match their expectations — an incentive missing, a deduction they do not recognise, or a credited amount lower than what they had mentally calculated.</p><p>At Swiggy, these seemingly simple questions translated into a massive operational support challenge, with close to <strong>9 lakh monthly tickets</strong> being raised across <strong>payout</strong> and <strong>incentive-related</strong> issues — all routed to human support queues.</p><p><strong><em>For context</em></strong><em>, each order has a payout that the DE always earns. Incentives are additional bonuses on top of this base, unlocked only when specific conditions (e.g., order count, time windows) are met. In short, incentives represent potential earnings a DE can qualify for, while payouts are the final credited amount after applying incentives along with other earnings and adjustments.</em></p><p>The questions themselves were repetitive and deceptively simple:</p><blockquote>“Kal ka incentive kyun nahi mila?”</blockquote><blockquote>“I earned ₹1000 yesterday… but only ₹800 got credited — what happened to the rest?”</blockquote><blockquote>“₹440 क्यों काट लिए? ये TDS hai ya onboarding fees? Samajh nahi aa raha.”</blockquote><p>What made resolution especially difficult was that these were not generic support queries. They were financial explanations. A DE was not asking for a status update or a navigation hint; they were asking us to explain earned money, missed milestones, taxes, onboarding fees, penalties, and final credited payouts with accuracy and speed. Even for human agents, these answers were not straightforward. The truth lived across complex incentive slabs, milestone tiers, peak-slot conditions, and payout adjustments spread across multiple systems.</p><p>That gap between a simple user question and a deeply fragmented backend truth created three problems at once. It slowed resolution, it increased operational load, and most importantly, it created a trust problem. In a financial-support workflow, even a small factual mistake does not feel like a bad response; it feels like the system cannot be trusted with money.</p><h4>Legacy Support</h4><p>Before conversational support bots, these conversations were handled by human support queues. Resolution times stretched to roughly<strong> ~20 minutes</strong>, while CSAT (customer satisfaction score) stayed low. The issue was that explaining incentive and payout logic requires cross-referencing multiple conditions, thresholds, and deductions before you can even begin to phrase the answer clearly.</p><p>We also knew a generic LLM-first approach would not be enough. If you ask a model to directly compute financial truth from a raw, ambiguous context, you are effectively asking it to arbitrate. That is not what we wanted. Our problem was not to generate plausible answers; Our problem was to generate correct explanations of authoritative answers.</p><h3>The core insight: the LLM explains, it does not arbitrate</h3><p>The architectural principle that shaped the entire system was simple: we did not ask the LLM to decide eligibility or compute payouts. We used deterministic systems to resolve the truth first, then used the LLM to explain that truth clearly, conversationally, and in the DE’s preferred language.</p><p>That decision changed the role of the model completely. Instead of being a financial reasoner, it became a grounded explainer. Logic resolution moved upstream into code. Language generation stayed downstream. This separation is what made the system reliable enough for a financial-support use case.</p><h3>What we built</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LIlsINf2aZk18BS0RvqCvg.png" /><figcaption>The payout amounts, incentive values, and dates shown in these screenshots are illustrative examples created for demonstration purposes only. They do not represent actual delivery partners, real transactions, or real financial values.</figcaption></figure><p>We built two AI-powered bots for DEs. The <strong>Daily Incentive (DI)</strong> Bot explains daily incentives: whether a DE is qualified, which milestone they achieved, which condition they missed, and what happened next. The <strong>Payout</strong> Bot explains weekly payout outcomes: total earnings, deductions, and why the final credited amount differs from what the DE expected. Both bots support pan-India multilingual interactions and share the same core design philosophy: deterministic financial resolution, LLM-powered explanation, and graceful human escalation when the system reaches the edge of what it can honestly answer.</p><p>From there, the engineering problem became how to combine grounded data retrieval, deterministic condition evaluation, reliable language generation, multilingual presentation, and escalation into one production system.</p><h3>High-level architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QP0_eX3xYuMs-mcncSAwDg.png" /><figcaption>a high level architecture of the bot</figcaption></figure><h3>At a high level, the system works in six stages:</h3><ol><li>A DE starts a chat inside the Swiggy DE app.</li><li>The request first goes through a backend gateway service that connects the app with the support bot.</li><li>The backend forwards the query to a Databricks Model Serving endpoint, where a LangGraph-based agent handles reasoning and tool orchestration.</li><li>Whenever factual payout or incentive data is needed, the agent calls internal APIs for incentive history, payout breakdowns, and deductions.</li><li>Before any information reaches the LLM, deterministic preprocessing logic converts raw payout conditions into explanation-ready signals to ensure accuracy and consistency.</li><li>The model generates a grounded response in English, which is then translated into the DE’s preferred language through a post-processing layer.</li></ol><h3>Why we chose a graph-based agent</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ynD2N_7YBQULCZo1VuggDA.png" /><figcaption><strong>Note:</strong> Self-correction limits (e.g., max 3 retries) are enforced for loop handling. For casual greetings like “hi”, the system skips the data-fetching and validator nodes entirely, routing directly to the end</figcaption></figure><p><strong>Components of agent graph:</strong></p><p><strong>llm_node</strong> is the brain. It receives the full conversation history, renders it through the system prompt template, and invokes the LLM. If the LLM needs data, it returns an <em>AIMessage</em> with a <em>tool_calls</em> field populated. If it has enough context to respond, it returns the final answer.</p><p><strong>tool_node</strong> is a prebuilt <em>ToolNode</em> from LangGraph. It inspects the last <em>AIMessage</em> for any tool call presence, executes the requested tool (<em>get_incentive_details</em>), and appends the result back into state as a <em>ToolMessage</em>. No custom routing code — LangGraph handles dispatch.</p><p><strong>validator_node</strong> is a custom post-processing check that runs only when the LLM produces a final response. It cross-references claimed incentive amounts in the <em>AIMessage</em> against the raw tool output still in state. On a mismatch, it injects a correction <em>HumanMessage</em> and routes back to <em>llm_node</em> for regeneration. On a match, it passes through to END.</p><p>The support flow is multi-turn by nature. A DE might first ask why they missed an incentive, then follow up on a specific milestone, and then ask whether a peak slot could still qualify them. A single monolithic prompt is not a good fit for that shape of interaction. We used LangGraph’s StateGraph to build a ReAct-style loop with three core nodes: an LLM node for reasoning, a tool node for grounded data retrieval, and a validator node for post-generation safety checks.</p><p>We also needed the bot to behave like a real support system across multiple turns, not like a stateless demo. The graph persists conversation history using Redis checkpointing, which allows the same conversation to resume with context intact while isolating different support streams when needed. For longer interactions, we trim the context passed to the LLM while preserving full state in Redis, so the model stays within token limits without losing system truth.</p><h3>Deterministic preprocessing before the LLM</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Cv_YVliqXvhw7e-079Ssw.png" /><figcaption>the data preprocessing pipeline, here — In the DI bot, incentives are payout schemes, milestones are threshold slabs checked as pass/fail for each incentive, and peak slots are high-demand time windows used as a fallback level when day-level conditions fail. Please note — in the above pipeline, the examples shared does not represent actual conditions or values for incentives for DEs at Swiggy.</figcaption></figure><p>This is where the system earns its reliability. The <em>get_incentive_details</em> tool is the only data source exposed to the model, but even its output is not raw backend data. Before anything reaches the LLM, we run a deterministic preprocessing pipeline that turns fragmented, ambiguous incentive data into a narrow, pre-evaluated representation of outcomes.</p><p>The condition evaluation engine parses raw condition strings and evaluates them directly against the DE’s actual metrics. It supports direct comparisons, single counters, and composite conditions, then emits a normalised verdict instead of passing raw logic downstream. If a day-level condition fails, the system can re-evaluate the corresponding peak-slot (e.g., peak lunch hour incentives) rule and surface the most relevant failure context before the model ever sees the case.</p><p><strong>Condition Evaluation example in Python -</strong></p><p><strong>Condition:</strong></p><blockquote><em>trips_completed &gt; x &amp; login_minutes &gt; y</em></blockquote><p><strong>Actual:</strong></p><blockquote><em>trips_completed = a (lower than x) , login_minutes = b (high than y)</em></blockquote><p><strong>Output:</strong></p><blockquote><em>FAIL — trips_completed: a</em></blockquote><p>We also trim milestone ladders aggressively. Incentive schemes often contain multiple slabs, but only a small slice is relevant to the explanation. We drop milestones below the highest achieved tier and keep only the first failure after that. This reduces the context to the decision boundary: what the DE achieved, and what they missed next.</p><p><strong>Trimming Example -</strong></p><p><strong>Before:</strong></p><blockquote>₹200 ✓ ₹400 ✓ ₹700 ✗ ₹1000 ✗ ₹1500 ✗</blockquote><p><strong>After:</strong></p><blockquote>₹400 ✓ ₹700 ✗</blockquote><p>The result is the key principle behind the whole system: the LLM explains, but it does not arbitrate. All complex logic resolution happens upstream in code; the model receives clear, minimal, unambiguous truth and turns it into a conversational explanation.</p><h3>The validator as a trust boundary</h3><p>In financial support, even a ₹100 hallucination is a trust failure. That is why we added a validator node after the model generates a final response. The validator extracts all incentive amounts claimed in the generated message, compares them against the ground-truth amounts still present in tool state, and if a mismatch is found, injects a correction message and routes the response back to LLM for regeneration.</p><p>The regeneration rate is low, but the cases it catches are exactly the ones that matter most: amount swaps, misread condensed outputs, and other subtle failures that would quickly erode trust with DEs.</p><pre>if not amounts_match(claimed_amounts, ground_truth_amounts):<br>  correction = HumanMessage(<br>  content=f&quot;[SYSTEM] Your stated amounts do not match the data. &quot;<br>  f&quot;Ground truth: {ground_truth_amounts}. Please regenerate.&quot;<br>  return {&quot;messages&quot;: [correction]}<br>return {} # Passes through to END</pre><h3>Multilingual Support without multilingual reasoning complexity</h3><p>The system supports multilingual conversations, but we made an important design choice: translation lives outside the agent graph. The graph always reasons in English; language is treated as a presentation concern rather than a reasoning concern. That kept the graph simpler and avoided extra state or tool calls for language resolution, which already existed in backend session context.</p><h4>The one way translation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U8Fi-dP9kdN3vAOulE_qJg.png" /><figcaption>example of one-way translation</figcaption></figure><p>The intuitive approach is bidirectional: translate the DE’s input to English, pass it to the LLM, then translate the response back. We tried this — and ran into a key issue:</p><ul><li><strong>Poor input translation quality</strong> — short, noisy queries like <em>“mujhe kal ka incentive nahi mila”</em> (often in Romanized Hindi) were inconsistently translated, which impacted intent understanding.</li></ul><p>In practice, the LLM already handles multilingual input well — including Romanized Hindi, native scripts, and mixed-language queries. Since DE queries are typically short and intent-driven, input understanding was not the bottleneck.</p><p>The real challenge was the output: generating clear, structured, and accurate incentive breakdowns in the user’s preferred language. Instead of relying on the LLM for multilingual generation, we constrained it to produce a clean English explanation and translated only the final response.</p><p>We found that a full input-output translation pipeline was suboptimal. Translating DE inputs into English before reasoning introduced errors for short, noisy, and Romanized queries, while the LLM could already understand multilingual input reasonably well. In practice, the bigger challenge was generating clean, semantically consistent and trustworthy responses in the DE’s preferred language.</p><p>We enforce this with a simple (and slightly hacky) trick — <strong>appending an instruction directly to the user message</strong>, asking the LLM to respond in English. Since we already know the DE’s preferred language, we add this instruction in their own language, making it more likely the model follows it.</p><pre>user_message = f&quot;&quot;&quot;<br>{user_query}<br><br>(कृपया जवाब केवल English में दें और साफ़ तरीके से incentive explain करें)<br>&quot;&quot;&quot;</pre><p>This way:</p><ul><li>The model fully understands the user’s query in their native language</li><li>But is nudged (very explicitly) to respond only in English</li><li>Avoiding inconsistent multilingual generation while keeping input natural</li></ul><p>We let the model understand the input natively, nudged it to respond in English, and translated only the final message field while leaving the routing intent untouched. That improved reliability and <strong>reduced translation cost by ~50%.</strong></p><h4>Evaluating different translator Models</h4><p>We evaluated Sarvam, Google Translate, and Azure translators for multilingual translation quality -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ARLUcgO6kfHQBjudZLq99A.png" /><figcaption>evaluation of different translators used in the Daily Incentive (DI) Bot. These results are based on internal evaluations and experiments conducted within the DI Bot workflow as of May 2025.</figcaption></figure><p>The main issue with traditional translators was not basic language conversion, but <strong>loss of domain control</strong>. In our use case, the bot explains financial constructs such as <strong>incentives, slabs, peak incentives, and payout-related terms</strong>. Generic translators often translated these terms too literally or into culturally plausible but product-incorrect alternatives. For example, words like <strong>incentive</strong> could drift toward meanings such as <strong>bonus</strong>, <strong>promotion</strong>, or other loosely related terms, which reduced precision in a financial-support flow.</p><p><strong>Why Azure worked better:</strong> Azure Translator gave us an important control lever that the broader setup needed — the ability to <strong>preserve selected words / entities</strong> using a dictionary or preserve-list style mechanism. That meant business-critical terms could remain untranslated, or be translated in a standardised way, instead of being reinterpreted each time by the translation system. This was especially important for bot handling payout and incentive explanations, where terminological consistency directly affects DE trust.</p><p>The key shift was:</p><blockquote><strong>Don’t translate business logic. Translate presentation.</strong></blockquote><p>This gave the DI bot a more trustworthy multilingual UX while also reducing translation cost significantly.</p><h3>Human escalation as part of the honesty model</h3><p>Not every incentive or payout dispute should be solved by a bot. Some cases are genuinely out of scope: unexplained data discrepancies, policy exceptions, manual-review scenarios, or urgent situations where a human should take over immediately. When the model detects these patterns, it returns a structured escalation intent, and the backend routes the conversation to a human support agent along with the full context.</p><p>This is an important product principle. The goal is not maximum bot stubbornness. The goal is correct resolution. A system that knows when to stop is more trustworthy than one that tries to answer everything.</p><h3>Production Learnings: Latency Matters</h3><p>Our biggest production bottleneck was not the LLM — it was data retrieval.</p><p>Initially, incentive and DE history data were fetched via the Databricks SQL connector. Because this required heavy joins across multiple tables, <strong>every new date query by a user incurred an 8–10 second latency penalty</strong>, severely dragging down the interactive experience.</p><p>For a conversational support flow, that was unacceptable. We moved retrieval to an internal aggregated API built by the engineering team, reducing fetch latency from seconds to milliseconds and significantly improving the real-time experience.</p><p>This was a useful reminder of a broader truth in applied AI systems: production quality depends as much on serving architecture and data contracts as on prompt or model quality.</p><h3>Results</h3><p>The system now resolves incentive- and payout-related questions in near real time, <strong>reducing resolution times</strong> from ~20 minutes to <strong>a few seconds</strong>. <strong>CSAT improved by 40%</strong>, and the bots now resolve <strong>~90%</strong> of incentive and payout-related queries end to end.</p><p>Those numbers matter, but the bigger outcome is qualitative. We moved the user experience from “please wait while an agent investigates multiple systems” to “here is a grounded explanation of what happened, in your language, with a human fallback when needed.” That is a better support experience, a better operational model, and a more trustworthy way to explain financial outcomes to delivery partners.</p><h3>Conclusion</h3><p>The real challenge in this project was never building a chatbot. It was building a system that could explain money clearly, quickly, and correctly. That required us to treat language generation and financial logic as two different responsibilities. Deterministic systems resolved truth. The LLM turned that truth into a clear explanation. Validators protected accuracy. Translation stayed downstream. Escalation stayed honest.</p><blockquote><strong>Author</strong>: <a href="https://medium.com/u/abe69fbfb436">Yash Khandelwal</a></blockquote><blockquote><strong>Contributors:</strong> <a href="https://medium.com/u/adb0ad89cfb">devayani</a> and <a href="https://medium.com/u/b606a27eba38">Rishabh Patil</a></blockquote><blockquote><strong>Special Thanks</strong> to <a href="https://medium.com/u/0d0c381fa186">Prateek Saxena</a> for their contribution and <a href="https://medium.com/u/10c3c71bd5f1">Meghana Negi</a>, <a href="https://medium.com/u/38e2a378ccf2">Sanskarjain</a> for guidance throughout this project.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=066c4f23e875" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swiggy-bytes/explaining-every-rupee-how-we-built-reliable-llm-support-bots-for-delivery-partners-066c4f23e875">Explaining Every Rupee: How We Built Reliable LLM Support Bots for Delivery Partners</a> was originally published in <a href="https://medium.com/swiggy-bytes">Swiggy Bytes — Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science Hackathons: Winner’s Guide (Part 1)]]></title>
            <link>https://yashkhandelwal07.medium.com/data-science-hackathons-winners-guide-part-1-fecbf3b2c023?source=rss-abe69fbfb436------2</link>
            <guid isPermaLink="false">https://medium.com/p/fecbf3b2c023</guid>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yash Khandelwal]]></dc:creator>
            <pubDate>Tue, 28 Nov 2023 13:46:13 GMT</pubDate>
            <atom:updated>2023-11-28T13:49:17.306Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XB0OpXrvtI-61pRUOAhdGQ.jpeg" /><figcaption>Hackathons can be rewarding!!!</figcaption></figure><p>Get ready to up your game in data science hackathons! In this blog, I’ll be sharing the winning solution approaches &amp; best practices in Data Science &amp; Machine Learning. It’s a practical guide to help you navigate and triumph in data hackathons, distilled from the lessons learned through my own National victories in past hackathons. Let’s dive into the essentials that can boost your chances of emerging victorious in any data science challenge.</p><h3>Where to participate?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8sV0YYB2GHuNOnT4BqAGxw.jpeg" /><figcaption>Source: Unsplash</figcaption></figure><p>The platforms, such as <a href="https://www.kaggle.com/competitions">Kaggle</a>, <a href="https://datahack.analyticsvidhya.com/contest/all/">DataHack</a>, <a href="https://machinehack.com/hackathons">MachineHack</a>, <a href="https://unstop.com/hackathons">Unstop</a>, etc. offer a chance to apply your skills in Python, Data Analytics, Machine Learning, etc in real-world challenges. Participating in these platforms is not merely about clinching the top spot; it’s a journey of hands-on learning and community engagement. These hackathons provide an invaluable chance to test your capabilities, learn from the diverse approaches of fellow participants, and stay attuned to the ever-evolving landscape of data science. Consider it your ticket to continuous improvement, community collaboration, and a noteworthy addition to your professional portfolio!</p><h3>Problem Statements</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rw5LDHRkBY2aajU84p2Dfg.jpeg" /><figcaption>Source: Unsplash</figcaption></figure><p>Let’s now see a glimpse of intriguing problem statements you’ll encounter -</p><p><strong>1. Supervised/ Unsupervised Machine Learning (ML):</strong></p><p>Example: Predicting the sales of a mega mart and making recommendations for sustainable sales.</p><p><strong>2. Time Series Forecasting:</strong></p><p>Example: Forecasting PM2.5 concentrations for the first 3 days of 2023 for 34 Indian Cities.</p><p><strong>3. Deep Learning:</strong></p><p>Example: Estimating the intensity of cyclones from satellite imagery using CNN.</p><p><strong>How the Game Works: Behind the Scenes</strong></p><p>In hackathons, participants submit the predictions from their model for evaluation. A private leaderboard is used to track participants’ performance, but the scores are hidden until the end. Winners are determined based on the predictive accuracy of their models through various metrics depending on the type of predicting task. The target is to develop a ML model that is perfect fit on the training data and it doesn’t overfit, giving the best metric value on the leaderboard.</p><p>Here’s how it works: you build a model to predict things, like sales or pollution. You submit your predictions, but you don’t know how well you did until the end. There’s a special scoreboard that keeps track, but it keeps the scores a secret until it’s over. The winners are the ones whose predictions were super close to what really happened.</p><p><strong>Lets now look at the state-of-the-art techniques and best ML practices to top the Private leaderboard!</strong></p><h3>Supervised/ Unsupervised Machine Learning (ML)</h3><p>Let’s take the problem statement mentioned before into account — “Predicting the sales of a mega mart and making recommendations for sustainable sales” and see how my winning solution approach &amp; techniques were.</p><p><strong>1. Understanding the Data (Link to the dataset, </strong><a href="https://machinehack.com/hackathons/data_analytics_championship/overview">Hackathon Link</a><strong>) -</strong></p><p>The challenge revolves around predicting the sales of a mega mart. In this regression task, the training data comprises 87,864 observations with 12 features, including 7 categorical and 5 continuous ones. The test data consists of 37,656 observations, and the target feature is the Sales of items in the mega mart.</p><p><strong>2. Data Preparation</strong></p><p>The journey to a successful model kicks off with meticulous data preparation. This involves transforming raw data into an informative dataset, laying the foundation for a machine learning model that not only comprehends the intricacies of the training data but also excels in making accurate predictions on real-world data.</p><p>Let’s, look at the findings in the data and the preparation methods that worked best on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*0hLwpxEFSIT6oBZ9lIbx_A.png" /><figcaption>The dataset</figcaption></figure><p><strong><em>Outlier Detection:</em></strong><em> Inter Quartile Range Method</em></p><p>Applying the Interquartile Range (IQR) Method to identify and handle outliers ensures a cleaner dataset. The code snippet demonstrates how to cap extreme values within the acceptable range.</p><pre># Assuming df is the DataFrame <br>Q1 = df[‘Sales’].quantile(0.25) <br>Q3 = df[‘Sales’].quantile(0.75) <br>IQR = Q3 — Q1 <br>lower_bound = Q1–1.5 * IQR <br>upper_bound = Q3 + 1.5 * IQR <br>df[‘Sales’] = df[‘Sales’].apply(lambda x: upper_bound if x &gt; upper_bound else (lower_bound if x &lt; lower_bound else x))</pre><p><strong><em>Null Value Imputation: </em></strong><em>No null values found!</em></p><p>A check for null values is conducted, and in this case, none are found, simplifying the data preparation process.</p><p><strong><em>Categorical Encoding: </em></strong><em>One Hot Encoding (</em>When categorical variables have no inherent order or ranking), <em>Ordinal Encoding </em>(When categorical variables have a clear order or rank).</p><p>For example, in this dataset — Outlet_ID, Item_Type, Outlet_Year, Outlet_Type have no order while Outlet_Size, Outlet_location_type, Item_FC have a visible order.</p><pre># One Hot Encoding <br>df_encoded = pd.get_dummies(df, columns=[‘categorical_feature’], drop_first=True) # Ordinal Encoding <br>ordinal_mapping = {‘low’: 0, ‘medium’: 1, ‘high’: 2} <br>df[‘ordinal_feature’] = df[‘ordinal_feature’].map(ordinal_mapping)</pre><p><strong>Feature Engineering: </strong>Creating<strong> </strong>meaningful features that enhance a machine learning model’s performance. For example in this data — <em>Years from present</em> (2023 — Outlet_year)</p><pre># Assuming df is your DataFrame<br>df[‘new_feature’] = df[‘feature1’] * df[‘feature2’]</pre><p><strong><em>Feature Selection:</em></strong></p><p><em>Permutation Feature Importance:</em> is a technique that measures the impact of each feature on a model’s performance by randomly shuffling the values of a single feature and observing the effect on model accuracy.</p><pre>from sklearn.inspection import permutation_importance <br># Assuming model is your trained machine learning model <br>perm_importance = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=42) <br>feature_importance = perm_importance.importances_mean</pre><p>XGBoost Feature Importance: XGBoost, a powerful gradient boosting algorithm, provides a built-in method for feature importance. It calculates how each feature contributes to reducing the impurity during tree construction.</p><p>import xgboost as xgb # Assuming X_train, y_train are your training data model = xgb.XGBClassifier() model.fit(X_train, y_train) # Accessing feature importance scores feature_importance = model.feature_importances_#import csv12345678makefile</p><p>Both techniques offer insights into feature importance, helping you identify key variables and refine your feature set for improved model performance.</p><p><strong>3. Machine Learning — Modelling</strong></p><p>The modelling begins with splitting the training data into training and validation. There are major two techniques for the same — 1.<em> K Fold Cross Validation </em>2. <em>Train-test-split</em></p><p>For this dataset, train-test-split is itself efficient because of abundant data, prioritising simplicity and faster computation.</p><pre>from sklearn.model_selection import train_test_split <br># Assuming X is your feature matrix and y is your target variable <br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) #import csv12345python</pre><p><strong>Modelling</strong></p><p>Efficient regression model development begins with a baseline model, often a simple linear regression, serving as a performance benchmark. Evaluate its results using appropriate metrics. Explore feature relationships and engineer informative features. Even try to alter the data preparation with suitable methods. Experiment with various models, fine-tune hyperparameters, and consider ensemble methods for improved robustness.</p><p>This is the order in which I begin my modelling experimentation.</p><p><strong>1. Linear Regression</strong></p><p><strong>2. Lasso &amp; Ridge Regression</strong> (Regularization, will prevent over fitting)</p><p><strong>3. Decision Tree</strong> (Will give better metric values on training data through high predictive performance)</p><p><strong>4. Random Forest</strong> (Will solve the overfitting issue of decision trees)</p><p><strong>5. XGBoost, AdaBoost, LightGBM, CatBoost</strong> (Boosting models sequentially build a robust learner by focusing on correcting errors made by weak models, effectively reducing bias, lowering variance, and excelling in capturing intricate patterns)</p><p>Expect the best performance from the boosting models. But where does the crux of winning solution lies then?</p><p>There are few more, less common techniques — <strong>Stacking Ensemble</strong> &amp; <strong>Weighted Average Ensemble. </strong>Stacking, with its meta-model learning from varied base model predictions, often outperforms individual models. Average ensemble methods, on the other hand, leverage collective wisdom to create a more balanced and robust prediction by mitigating biases or errors present in individual models. Both approaches contribute to enhanced model performance through effective combination and adaptation to different aspects of the underlying data.</p><p>Both of these models had the best performance on this data. These models topped the leaderboard!</p><pre>#Code for Stacking Ensembling<br>from sklearn.ensemble import StackingRegressor <br>from sklearn.linear_model import LinearRegression <br>from sklearn.ensemble import RandomForestRegressor<br>from sklearn.tree import DecisionTreeRegressor <br># Assuming base models are your individual regression models <br>base_models = [(‘linear’, LinearRegression()), (‘rf’, RandomForestRegressor()), (‘tree’, DecisionTreeRegressor())] <br>stacking_model = StackingRegressor(estimators=base_models, final_estimator=LinearRegression()) <br>stacking_model.fit(X_train, y_train)</pre><pre>#Code for Weighted Average Ensemble<br># Assuming model1, model2, model3 are your regression models<br>weights = [0.4, 0.3, 0.3] <br># Adjust weights based on model performance<br>predictions_model1 = model1.predict(X_test) <br>predictions_model2 = model2.predict(X_test) <br>predictions_model3 = model3.predict(X_test) <br>weighted_average_predictions = (weights[0] * predictions_model1 + weights[1] * predictions_model2 + weights[2] * predictions_model3) / sum(weights) </pre><p><strong>End Note</strong></p><p>Creating an efficient machine learning (ML) model involves foundational steps. Starting with a comprehensive understanding of the data, addressing missing values, and outliers with patience. Thoughtful feature engineering enhances the model’s pattern recognition. Systematic hyperparameter tuning fine-tunes performance, and careful model selection aligns with specific tasks. Ensemble methods, cross-validation, and regularization contribute to robustness. Incorporate machine learning explainability for transparency in model decisions, crucial for business understanding. <strong>Lastly, continuous iteration based on evaluation insights ensures ongoing enhancement of predictive capabilities, aligning the model with both technical and business objectives of problem statement of the hackathon.</strong></p><p>Remember, to win a hackathon, an efficient ML model &amp; your leaderboard standing may not be enough. You might have to also deliver an outstanding presentation on your ML solution in order to win 🏆🏅</p><p>Here is the link to my winning solution for this problem statement which was a part of <a href="https://analyticsindiamag.com/analytics-olympiad-organised-by-shiv-nadar-university-aim-ends-on-a-high-note/">Analytics Olympiad 2021</a> : <a href="https://github.com/YashK07/Analytics-Olympiad-2021-My-Winning-Solution"><strong>Github</strong></a><strong> Link</strong>(also, have a look on how a presentation for your solution should look like)</p><p>Here are a few helpful resource links that might be useful along this blog:</p><ol><li><strong>Stacking Ensemble in Depth with code -</strong><a href="https://www.analyticsvidhya.com/blog/2021/08/ensemble-stacking-for-machine-learning-and-deep-learning/">https://www.analyticsvidhya.com/blog/2021/08/ensemble-stacking-for-machine-learning-and-deep-learning/</a></li><li><strong>Lifecycle of a Data Science Project -</strong><a href="https://www.analyticsvidhya.com/blog/2021/10/introduction-to-the-lifecycle-of-data-science-project/#h2_9">https://www.analyticsvidhya.com/blog/2021/10/introduction-to-the-lifecycle-of-data-science-project/#h2_9</a></li></ol><h3>Time Series Forecasting</h3><p>This is another challenging problem statement in Data Science hackthons.</p><p>Let’s continue with Time Series Forecasting in the next blog titled — “Data Science Hackathons: A Practical Guide (Part 2)”. Happy Reading! 😊💻</p><h3>About the Author</h3><p>Yash Khandelwal is a final year student pursuing Mathematics &amp; Computing at Birla Institute of Technology, Mesra. He was a National Winner of EXL EQ 2023, National Finalist of Smart India Hackathon 2022 and a National Runner up of Analytics Olympiad 2021.</p><p>Connect with him on LinkedIn — <a href="https://www.linkedin.com/in/yash-khandelwal-a40484bb/">https://www.linkedin.com/in/yash-khandelwal-a40484bb/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fecbf3b2c023" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Big Data Analytics: How it works and its benefits]]></title>
            <link>https://medium.com/analytics-vidhya/big-data-analytics-how-it-works-and-its-benefits-2903761571d4?source=rss-abe69fbfb436------2</link>
            <guid isPermaLink="false">https://medium.com/p/2903761571d4</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yash Khandelwal]]></dc:creator>
            <pubDate>Tue, 13 Dec 2022 06:39:22 GMT</pubDate>
            <atom:updated>2022-12-13T06:39:22.986Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yO-v4R68Oy7Fbw9YAjJp-g.jpeg" /></figure><h3>Introduction</h3><p>Data is so pervasive in today’s society that it’s impossible to account for all of the ways it influences your daily life. Every day, so much data is gathered that we now refer to these vast collections as “big data.” Before you understand how big data is used in your daily life, you must first comprehend what big data is and how it is collected.</p><h4>What is Big Data?</h4><p>We live in a generation where technology is used by every person in the world to do their most basic work. People around the world generate a lot of data without actually knowing it.</p><p>Big Data is the huge amount of data that is generated when users text, email, run an application, make any money transactions, etc. This enormous volume of data is stored in the form of raw unorganized clusters called Big data.</p><p>Big data is very difficult to handle, mostly because of the 3 Vs of Big Data i.e Volume, Velocity, and Variety. Lets have a look at the 3 Vs -</p><h4>3 Vs of Big Data -</h4><p>Volume — This refers to the amount of data generated when people interact with any application on the internet.</p><p>Velocity — It refers to the speed at which data is being generated. The more people use it the faster the Velocity.</p><p>Variety — Variety in Big Data refers to all different structured and unstructured data generated on the internet. Examples — tweets, pictures, videos, voicemail, etc.</p><p>These 3 Vs of Big Data can’t be processed by any normal software, so a different domain Big Data Analytics is used to address these challenges and derive significant results from them. Now besides what theoretically big data means, how do the companies/business employ from it? So, when used in conjunction with analytics, big data fusion helps firms to combine data from many sources to develop a more comprehensive and unified model and gain a better understanding of the data. To succeed, organizations also invest in big data fusion technologies, artificial intelligence (AI) and machine learning. AI allows companies &amp; firms to sift through data from different sources to create cohesive and accurate insights.</p><p>For now lets understand the Anlytics part because ML/AI is out of scope for this blog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kcgdlmbz6RSfbvQqlAisCg.jpeg" /><figcaption>Today, businesses, irrespective of their size, are using analytics tools to determine how data can be leveraged in business use-cases.</figcaption></figure><h4>What is Big Data Analytics?</h4><p>Big Data Analytics is the field where Big Data is used to uncover different patterns, customer preferences, and correlations through which Business decisions are made.</p><p>With such enormous data out there, different companies and startups use this data to gain an edge on their businesses. This practice of using data to your advantage is called Big Data Analytics.</p><p>With the ever-growing demand for Big Data Analytics, everyday new statistical methods and new algorithms are being developed to address the different needs of the customer and the businesses.</p><h4>How does Big Data Analytics work?</h4><p>Big data analytics works in different phases ranging from collecting data to cleaning to preprocessing and then analysis.</p><h4>Collecting Data:-</h4><p>Data collecting is the process of collecting data directly or indirectly from customers. Different Businesses have their own method of collecting data. These methods can be reviews, social media, surveys, volunteers, previous purchases data, etc.</p><h4>Data Preprocessing:-</h4><p>Data preprocessing is the stage in which the collected data or the raw data is transformed into well-ordered data sets for higher accuracy in the complete analytics process.</p><p>This stage ensures that the transformed data has no missing or noisy data.</p><h4>Data Cleaning:-</h4><p>In this stage, the transformed data is thoroughly filtered to ensure high data quality. Any incorrect, irrelevant, or incorrect data is removed or corrected in the data set.</p><h4>Data Analysis -</h4><p>It is the analytics part in which data is used to extract different business insights.</p><p>Depending upon the Business, different tools and software are used such as Machine Learning, Deep Learning, Artificial Intelligence (AI), Business Intelligence software, etc. However irrespective of the tools Big Data Analytics can be summarized into 4 categories:-</p><ol><li><strong>Descriptive Analysis </strong>— It provides insights into past events and trends and uncovers any pattern if it exists. Descriptive Analysis uses techniques like statistics, clustering, etc which is used in market basket analysis. Its application includes the company’s sales, profits, revenue, etc.</li><li><strong>Diagnostic Analysis</strong> — It is used for in-depth analysis to understand the root cause of a problem. Diagnostic Analysis is used when there is a need to find the reason behind a particular event. Different techniques like data mining, data recovery, etc are used in this process.</li><li><strong>Predictive Analysis</strong> — It makes use of past incidents and market trends to uncover different patterns in them. It uses these patterns to forecast future demands and trends.</li><li><strong>Prescriptive Analysis</strong> — It runs its own analysis on the result of the above analysis to explore several possibilities depending on different actions and suggests which action could be the best to take among them.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iYmSFNz1WYTi-Vcd1LN_Bg.jpeg" /><figcaption>Big Data helps a business organization achieve various goals.</figcaption></figure><h4>Advantages of Big Data Analytics:-</h4><p>Big Data Analytics is widely used in different Businesses. With Big Data Analytics these businesses get key insights on their business, which further enhances their chances of identifying opportunities and risks. The major advantages Big Data Analytics provide are :</p><ol><li><strong>Saving time and costs </strong>— Using Big Data Analytics businesses can optimize their business by focusing more on their profit-making sectors.</li><li><strong>Risk Management</strong> — Big Data Analytics helps to foresee any upcoming risks;</li><li><strong>Better decision making</strong> — Big Data Analytics helps businesses to make better decision-making models than the traditional methods.</li><li><strong>Better Products </strong>— Businesses can get a better understanding of the customer needs which helps them in making products more likable for the consumer.</li><li><strong>Bringing Innovation</strong> — Businesses can use Big Data Analytics in their research and Development (R&amp;D) sector to bring new innovations which the consumers want.</li><li><strong>Key Insights</strong> — Businesses can predict future trends and can get major insights into their customer buying pattern.</li></ol><p>Though there may be a lot of advantages of Big Data Analytics, it also comes with some challenges:-</p><h4>Challenges of Big Data Analytics: -</h4><ol><li><strong>Quality Data Collection</strong> — Data coming from multiple sources needs to be verified and verifying Big Data can become quite hectic for Businesses.</li><li><strong>Data Accessibility</strong> — With such enormous amounts of data, the handling of these data can become more complicated which can severely affect the predictions.</li><li><strong>Proper tools for analysis </strong>— With the ever-increasing different set of tools and software for analyzing data, selecting suitable tools for the analysis can become quite confusing for someone who is not so versed in Big Data Analytics.</li><li><strong>Data Security</strong> — It can be very complex to properly address the security concerns of such large and complicated data stored.</li><li><strong>Not for the short run</strong> — Big data analytics is not useful for the short run. To properly leverage the advantage of Big data it needs to be analyzed for the long run.</li></ol><h4>Conclusion:</h4><p>The application of Big Data Analytics is no exception to the saying “Data is the new oil”. It solely depends upon the person or the organization how to use this oil to fuel their business growth.</p><p>Big Data Analytics has helped businesses to attain new heights and will do the same in the future. Happy reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2903761571d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/big-data-analytics-how-it-works-and-its-benefits-2903761571d4">Big Data Analytics: How it works and its benefits</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guide on Word Embeddings in NLP]]></title>
            <link>https://yashkhandelwal07.medium.com/a-guide-on-word-embeddings-in-nlp-1976f1d0014d?source=rss-abe69fbfb436------2</link>
            <guid isPermaLink="false">https://medium.com/p/1976f1d0014d</guid>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[text-mining]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yash Khandelwal]]></dc:creator>
            <pubDate>Sun, 03 Jul 2022 05:15:42 GMT</pubDate>
            <atom:updated>2022-07-03T05:15:42.186Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iJV-YRTG-ohUQjRH" /></figure><h3>Introduction</h3><p>Natural Language Processing (NLP) is the interdisciplinary field of Computer Science, Artificial Intelligence, and Linguistics, concerned with the ability of computers to understand human language. <strong>Word Embeddings</strong> is an advancement in NLP that has have skyrocketed the ability of computers to better understand text-based content. It is an approach to represent words and documents in the form of numeric vectors allowing similar words to have similar vector representations. Isn’t making a lot of sense?… Okay, let us start from scratch to develop an understanding of Word Embeddings!</p><h3>Background</h3><p>Given a supervised learning<a href="https://www.kaggle.com/vstepanenko/disaster-tweets"> task</a> to predict which tweets are about real disasters and which ones are not (classification). Here the independent variable would be the tweets (text) and the target variable would be the binary values (1: Real Disaster, 0: Not real Disaster). Now, Machine Learning and Deep Learning algorithms only take numeric input. So how do we convert tweets to their numeric values? Let’s dive deep.</p><h4>Bag of Words (BOW)</h4><p>The bag of words is a type of vectorization technique of text where each value in the vector would represent the count of words of a document/sentence.</p><p>Let’s take a small part of disaster tweets, 4 tweets, to understand how BOW works:-</p><p>‘kind true sadly’,</p><p>‘swear jam set world ablaze’,</p><p>‘swear true car accident’,</p><p>‘car sadly car caught up fire’</p><p>To create BOW we use Scikit-learn’s <strong><em>CountVectorizer,</em></strong> which<strong><em> </em></strong>tokenizes a collection of text documents, builds a vocabulary of known words, and encodes new documents using that vocabulary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/0*xZCWG-00Lc_eM_Py" /><figcaption>Output of CountVectorizer</figcaption></figure><p>Here the rows represent each document (4 in our case), the columns represent the vocabulary (unique words in all the documents) and the values represent the count of the words of the respective rows. In the same way, we can apply <em>CountVectorizer </em>on the complete training data tweets (11,370 documents) and obtain a matrix that can be used along with the target variable to train a machine learning/deep learning model.</p><h4>Term Frequency — Inverse Document Frequency (TF-IDF)</h4><p>Let’s see how vectorization is done in TF-IDF -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/0*EzVL8_vsUSn2B57g" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*FK5PAXCjKykapwKh" /></figure><p>To create TF-IDF vectors, we use Scikit-learn’s <strong><em>TfidfVectorizer. </em></strong>After applying it to the previous 4 sample tweets, we obtain -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/0*oQpiB5cHNQLPwpkf" /><figcaption>Output of <em>TfidfVectorizer</em></figcaption></figure><p>Similar to BOW, the rows represent each document, the columns represent the vocabulary, and the values are tf-idf(i,j) obtained through the above formula. This matrix obtained can be used along with the target variable to train a machine learning/deep learning model.</p><h3>Challenges</h3><p>Now let’s discuss the challenges with the two text vectorization techniques we have discussed till now. In <strong>BOW</strong>,<strong> </strong>the size of the vector is equal to the number of elements in the vocabulary. If most of the values in the vector are zero then the bag of words will be a sparse matrix. Sparse representations are harder to model both for computational reasons and also for informational reasons. Also, in BOW there is a lack of meaningful relations and no consideration for the order of words. While the <strong>TF-IDF</strong> model contains the information on the more important words and the less important ones, it does not solve the challenge of high dimensionality, sparsity and unlike BOW it also makes no use of semantic similarities between words.</p><p><a href="https://github.com/YashK07/Turing-Blogs/blob/master/A_Guide_on_word_embeddings_in_NLP_.ipynb"><strong>The complete code of disaster tweet classification using BOW and TF-IDF.</strong></a></p><h3>Solution: Word Embedding</h3><p>Word Embeddings is a technique where individual words are represented as real-valued vectors in a lower-dimensional space. Representing words as real-valued vectors captures inter-word semantics. Each word is represented by a real-valued vector with tens or hundreds of dimensions. A word vector with 100 values represents 100 unique features.</p><p>Let us now discuss two different approaches to getting word embeddings. We’ll also look at the hands-on part!</p><h4>Word2Vec</h4><p>Word2Vec method was developed by Google in 2013. The method involves iteration over a corpus of text to learn the association between the words. It relies on a hypothesis that the neighboring words in a text have semantic similarities between each other. It uses<a href="https://www.machinelearningplus.com/nlp/cosine-similarity/"> <strong>cosine similarity</strong></a> metric to measure the semantic similarity. Cosine similarity is equal to Cos(<em>angle</em>) where<em> the angle </em>is measured between the vector representation of two words/documents.</p><p>Word2Vec has two neural network-based variants: <strong>Continuous Bag of Words (CBOW)</strong> and <strong>Skip-gram</strong>.</p><ol><li><strong>CBOW </strong>— Let us understand the concept of <em>context</em> and <em>current</em> word for CBOW.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*-_q6ZurYKQf5AvaX" /></figure><p>In CBOW, we define a window size. The middle word is the <em>current</em> word and the surrounding words (past and future words) are the <em>context</em>. CBOW utilizes the context to predict the current words. Each word is encoded using One Hot Encoding in the defined vocabulary and sent to CBOW neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/0*7NH0jNVtG2phS625" /></figure><p>The hidden layer is a standard fully-connected Dense layer. The output layer outputs probabilities for the target word from the vocabulary.</p><p>2. <strong>Skip-gram — </strong>Skip-gram is slightly different from CBOW, as it uses the current word to predict the context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/0*LNXs3SkwgLtYT8Kn" /></figure><p>The end goal of Word2Vec (both variants) is to learn the weights of the hidden layer. The hidden weights, we’ll use them as our word embeddings!! Let’s now see the <strong>code </strong>for creating custom word embeddings using Word2Vec-</p><p><strong>Import Libraries</strong></p><pre>from gensim.models import Word2Vec</pre><pre>import nltk</pre><pre>import re</pre><pre>from nltk.corpus import stopwords</pre><p><strong>Preprocess the Text</strong></p><pre>#Word2Vec inputs a corpus of documents splitted into consituent words</pre><pre>corpus = []</pre><pre>for i in range(0,len(X)):</pre><pre> tweet = re.sub(“[^a-zA-Z]”,” “,X[i])</pre><pre> tweet = tweet.lower()</pre><pre> tweet = tweet.split()</pre><pre> corpus.append(tweet)</pre><p>Here is the exciting part! Let’s try to see the most similar words (vector representations) of some random words from the tweets -</p><pre>model.wv.most_similar(‘disaster’)</pre><p>Output -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/0*-cN77EqBjkKyBOBF" /><figcaption>List of tuples of words and their predicted probability</figcaption></figure><p>The embedding vector of ‘disaster’ -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/0*b-Cdvnv7CivHs761" /><figcaption>dimensionality = 100</figcaption></figure><h3>GloVe: Global Vector for word representation</h3><p>GloVe method was developed at Stanford by Pennington, et al. Unlike Word2Vec, which creates word embeddings using local context, GloVe focuses on global context to create word embeddings which gives it an edge over Word2Vec. In GloVe, the semantic relationship between the words is obtained using a co-occurrence matrix.</p><p>Consider two sentences -</p><p>I am a data science enthusiast</p><p>I am looking for a data science job</p><p>The co-occurrence matrix for involved in GloVe would look like this for the above sentences -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EmAh5I29AH74vkPM" /></figure><p>Window Size = 1</p><p>Each value in this matrix represents the count of co-occurrence with the corresponding word in row/column. Observe here — this co-occurrence matrix is created using <strong>global </strong>word co-occurrence count (no. of times the words appeared consecutively; for window size=1). If a text corpus has 1m unique words, the co-occurrence matrix would be 1m x 1m in shape. The core idea behind GloVe is that the word co-occurrence is the most important statistical information available for the model to ‘learn’ the word representation.</p><p>Let’s now see an example from Stanford’s GloVe<a href="https://nlp.stanford.edu/pubs/glove.pdf"> paper</a> of how the co-occurrence probability rations work in GloVe. “For an example, consider the co-occurrence probabilities for target words <em>ice</em> and <em>steam</em> with various probe words from the vocabulary. Here are some actual probabilities from a corpus of 6 billion word:”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*VtJ7DKLDX50qD0kK" /></figure><p>Here,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/0*k25ZDq__tnP2JvtQ" /></figure><p>Let’s take k = solid i.e, words related to ice but unrelated to steam. The expected Pik /Pjk ratio will be large. Similarly, for words k which are related to steam but not to ice, say k = gas, the ratio will be small. For words k like water or fashion, that are either related to both ice and steam or neither to both respectively, the ratio should be approximately one. The probability ratio is able to better distinguish relevant words (solid and gas) from the irrelevant words (fashion and water) than the raw probability. It is also abe to better discriminate between two relevant words. Hence in GloVe, the starting point for word vector learning is ratios of co-occurrence probabilities rather than the probabilities themselves. To understand the derivation of loss function used in this method, refer — to the<a href="https://nlp.stanford.edu/pubs/glove.pdf"> paper</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/0*GtDP0kjOA3B8P2g9" /></figure><p><a href="https://nlp.stanford.edu/projects/glove/">Source</a>: Visualization of GloVe Word Embeddings. <strong>King — Man + Woman = Queen</strong></p><p>Enough of the theory. Time for the code!</p><p><strong>Import Libraries</strong></p><pre>import nltk</pre><pre>import re</pre><pre>from nltk.corpus import stopwords</pre><pre>from glove import Corpus, Glove</pre><p><strong>Text Preprocessing</strong></p><pre>#GloVe inputs a corpus of documents splitted into consituent words</pre><pre>corpus = []</pre><pre>for i in range(0,len(X)):</pre><pre>tweet = re.sub(“[^a-zA-Z]”,” “,X[i])</pre><pre>tweet = tweet.lower()</pre><pre>tweet = tweet.split()</pre><pre>corpus.append(tweet)</pre><p><strong>Train the word Embeddings</strong></p><pre>corpus = Corpus()</pre><pre>corpus.fit(text_corpus,window = 5)</pre><pre>glove = Glove(no_components=100, learning_rate=0.05)</pre><pre>#no_components = dimensionality of word embeddings = 100</pre><pre>glove.fit(corpus.matrix, epochs=100, no_threads=4, verbose=True)</pre><pre>glove.add_dictionary(corpus.dictionary)</pre><p><strong>Find most similar -</strong></p><pre>glove.most_similar(“storm”,number=10)</pre><p>Output -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/0*TWzD0AerOIR1OZ18" /><figcaption>List of tuples of words and their predicted probability</figcaption></figure><p><a href="https://github.com/YashK07/Turing-Blogs/blob/master/A_Guide_on_word_embeddings_in_NLP_.ipynb"><strong>Code for using pre-trained Word2Vec embeddings and GloVe embeddings.</strong></a></p><h3>Conclusion</h3><p>In this blog, we discussed the two techniques for vectorization in NLP that are Bag of Words and TF-IDF, their drawbacks, and how word-embedding techniques like GloVe and Word2Vec overcome their drawbacks by <strong>dimensionality reduction and context similarity. </strong>Word embeddings can be used to train deep learning models like GRU, LSTM, Transformers which have been successful in NLP tasks such as sentiment classification, name entity recognition, speech recognition, etc. Now you know how word embeddings have benefited your day-to-day life as well.</p><p>Happy Learning!!</p><h3>About the author -</h3><p>I am a grad in Mathematics and Computing currently perusing my masters from BIT Mesra in the same stream. My field of interest and expertise lies mainly in Data Analytics, Machine Learning &amp; Quant.</p><p><strong>Connect with me -</strong></p><p>Linkedin: <a href="https://www.linkedin.com/in/yash-khandelwal-a40484bb/">https://www.linkedin.com/in/yash-khandelwal-a40484bb/</a> Github: <a href="https://github.com/YashK07">https://github.com/YashK07</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1976f1d0014d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Life-cycle of Data Science project: Hands-on guide]]></title>
            <link>https://medium.com/analytics-vidhya/life-cycle-of-data-science-project-hands-on-guide-ee1b2dd5d45e?source=rss-abe69fbfb436------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee1b2dd5d45e</guid>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Yash Khandelwal]]></dc:creator>
            <pubDate>Thu, 06 Jan 2022 17:25:19 GMT</pubDate>
            <atom:updated>2022-02-25T16:37:35.374Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3B7uy7EftcBIe5vzvsSWw.jpeg" /><figcaption>Pic Source: <a href="https://unsplash.com/photos/eveI7MOcSmw">https://unsplash.com/photos/eveI7MOcSmw</a></figcaption></figure><h3>Introduction</h3><p>With the increasing demand for Data Scientists, more people are willing to enter into this field. It has become very important to showcase the right skills to stand out from the crowd to get placed in the top companies. This is where the projects come into play. Moreover, end-to-end projects give you exposure to a real-time work environment. <strong>In this guide, I’ll be explaining the complete life-cycle of a Data Science project with a hands-on demo. </strong>We’ll be taking a Machine Learning problem statement and building a web application as its solution using Flask and deploying it on Heroku, a cloud application platform. Let’s get started!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfDktE9S4VzZVf1GLQvJnw.jpeg" /><figcaption>Pic Source: <a href="https://bit.ly/3zodc6H">https://bit.ly/3zodc6H</a></figcaption></figure><h3>1. Understanding the Problem</h3><p>The life cycle of a project begins the understanding the problem statement. The problem statement (<a href="https://www.hackerearth.com/challenges/competitive/hackerearth-machine-learning-challenge-predict-burnout-rate/">source</a>) for the project is: To build a web-based application with a machine learning model at the backend that predicts the burnout rate of company employees based on various work-life factors such as - working hours, work-from-home availability, mental fatigue score, and the like.</p><p><strong>Case Study: </strong>Happy and healthy employees are undoubtedly more efficient at work. They assist the company to thrive. However, the scenario in most of the companies has changed with the pandemic. Since work from home, over 69% of the employees have been showing burnout symptoms (survey source: <a href="https://learnmore.monster.com/poll-results-from-work-in-the-time-of-coronavirus">Monster</a>). The burnout rate is indeed alarming. Many companies are taking steps to ensure their employees stay mentally healthy. As a solution to this, we’ll build a web app that can be used by companies to monitor the burnout of their employees. Also, the employees themselves can use it to keep a check on their burnout rate (no time to assess the mental health in the fast work-life 😔).</p><h3>2. Gathering Relevant Data</h3><p>There are many libraries in python — Beautiful Soap, Selenium for scraping data. Besides, there are also web scraping APIs like ParseHub, Scrappy, Octoparse that make this less time-consuming. Web scrapping is a crucial part of a Data Science project because the lifecycle depends on the quality and relevance of the <strong>Data</strong>.</p><p>In this project, the dataset has been taken from <a href="https://www.kaggle.com/blurredmachine/are-your-employees-burning-out">Kaggle</a>. Have a look at the data before reading further.</p><p><strong>Dataset</strong></p><p>The following are the data attributes and their description -</p><ul><li><strong>Employee ID</strong>: The unique ID allocated by the company to each employee.</li><li><strong>Date of Joining</strong>: The date when the employee had joined the company.</li><li><strong>Gender</strong>: The gender of the employee.</li><li><strong>Company Type</strong>: The type of company where the employee is working in (Service/Product).</li><li><strong>WFH Setup Available</strong>: If the work from home facility is available for the employee (Yes/No).</li><li><strong>Designation</strong>: The designation of the employee in his/her organization. In range — [0.0, 5.0], 0.0 is the lowest designation and 5.0 is the highest.</li><li><strong>Resource Allocation</strong>: The number of resources allocated to the employee to work, to be interpreted as the number of working hours. In range — [1.0, 10.0] (higher means more resources).</li><li><strong>Mental Fatigue Score</strong>: How much mentally tired is the employee in the working hours in the range — [0.0, 10.0] where 0.0 means no fatigue and 10.0 means completely fatigue.</li><li><strong>Burn Rate</strong>: The target value in the data of each employee telling the rate of burnout during working hours in the range — [0.0, 1.0] where the higher the value is more is the burnout.</li></ul><p><strong>Few important notes about the data:</strong></p><p>1. Difference between Stress and Burnout is that burnout is a different state of mind. Under stress, you still manage to cope with pressures. But once burnout takes hold, you’re out of gas and you’ve given up all hope of surmounting your deterrents.</p><p>2. When you’re suffering from burnout, you feel more than just being mentally fatigued.</p><h3>3. Data Preparation and EDA</h3><p>After collecting the data, data preparation comes into play. It involves cleaning and organizing the data, which is known to take up more than 80% of data scientists’ work. The real-world data is raw and can be full of duplicates, missing values, and wrong information. Hence the data needs to be cleaned.</p><p>Once the data has been organized, we extract the information enfolded in the data and summarize its main characteristics through exploratory data analysis. EDA is an important stage for a well-defined data science project. It is performed before the statistical or machine learning modeling phase.</p><p>Enough of theory let’s begin the hands-on part!</p><pre>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*Cs5uXOiJL5fXmiXUg28A2A.jpeg" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><ul><li>There are 5 categorical features and 4 numerical features. Out of the 5, all the features are useful except the <em>employee id</em> as it has nothing to do with the target i.e. <em>Burn Rate</em>. Let us first perform exploratory data analysis and alongside we’ll transform the categorical features into a form that can be understood by the model.</li></ul><pre>#Date and month might not be a useful feature. But the year of joining is. It has some significant information that the model can use.</pre><pre>df[&#39;Year of Joining&#39;] = df[&#39;Date of Joining&#39;].apply(lambda x : x.split(&#39;-&#39;)[0])</pre><pre>df[&#39;Year of Joining&#39;].describe()</pre><p><strong>Output: </strong>The feature has only one unique value i.e 2008. This may not be a useful feature. So we drop it.</p><pre>df.drop(&#39;Year of Joining&#39;,axis=1,inplace=True)</pre><p>Let’s do some EDA on other categorical features. Alongside we’ll also do the Data Preparation part -</p><h4>EDA</h4><pre>sns.boxplot(x = &#39;Gender&#39;, y = &#39;Burn Rate&#39;,data = df);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*5t12CDVNqfRPUX7UATf-aA.png" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><ul><li>The average (median) <em>Burn Rate</em> among the male employees is higher than that in female employees. Let’s find the possible reasons by comparing other factors like - designation, working hours, etc. among the two genders in the company.</li></ul><p><strong>Note: </strong>The boxplot shows that there are outliers in the <em>Burn Rate</em> records of female employees. We’ll have to take care of it.</p><pre>sns.countplot(x = &#39;Gender&#39;,palette=sns.color_palette(&quot;hls&quot;),data = df);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*61gRfk9QjwKhw5NYJMxVSQ.png" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><ul><li>The data has more records of female employees.</li></ul><pre>sns.countplot(x = &#39;Designation&#39;,hue =&#39;Gender&#39;,palette=&#39;Blues_d&#39;,data = df);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*i-lHynzSVSxX-KY4Az-5Vg.png" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><ul><li>There are more males working at higher designation (&gt;2.0).</li></ul><pre>sns.boxplot(x = &#39;Gender&#39;, y = &#39;Resource Allocation&#39;,palette=sns.color_palette(&quot;Paired&quot;),data = df);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*dtcT3J1-TG2u2w6bckdf2A.png" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><ul><li>While most of the female employees work up to 8 hours, the male employees work up to 10 hours. Average working (median) hours between both genders differ by 1 hour.</li></ul><p><strong>Note</strong> — Again we observe outliers in the feature — <em>Resource Allocation</em> in the female records.</p><pre>sns.boxplot(x = &#39;Company Type&#39;, y = &#39;Mental Fatigue Score&#39;,palette=sns.color_palette(&quot;hls&quot;),data = df);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*HV3Ai9YsA0nbjs41I7mBWw.png" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><ul><li>The fatigue scores remain equal for the two types of companies. <strong>Note- </strong>Outliers</li></ul><pre>sns.scatterplot(x = &#39;Mental Fatigue Score&#39;, y = &#39;Burn Rate&#39;,hue = &#39;Designation&#39;,palette=sns.color_palette(&quot;rocket&quot;),data = df);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*ztefIl8vFstAN9_67qNCVQ.png" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><ul><li>This indicates a very strong linear relationship between the <em>Fatigue Score</em> and <em>Burn rate</em>.</li></ul><pre>sns.heatmap(df.corr(),annot = True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*a7Q2FItb1_BgRmgCw4lCFA.png" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><ul><li>Apart from <em>Mental Fatigue Score</em>, <em>Burn Rate</em> shows a strong positive linear relationship with <em>Resource Allocation</em>.</li></ul><h4><strong>Categorical Encoding</strong></h4><p>This is the process of transforming the text data (data under categorical variable) into numbers so that the model is able to understand and extract valuable information from it. These are some powerful techniques of categorical encoding -</p><ul><li><strong>Ordinal Encoding — </strong>Involves the<strong> </strong>mapping of each unique label to an integral value. This type of encoding is advisable only if there is a known order/relationship between the categories. Eg. Take an example of a variable <em>Product Quality </em>that has values- ‘Excellent’, ‘Very Good’, ‘Good’, ‘Average’, and ‘Poor’. Since this data has an order i.e. ‘Excellent ‘&gt;’Very Good’&gt;’Good’&gt;’Average’&gt;’Poor’, we can use Ordinal Encoding to map- ‘Poor’: 0, ‘Average’: 1, ‘Good’: 2, ‘Very Good’: 3, ‘Excellent’: 4.</li><li><strong>One Hot Encoding — </strong>This is the transformation of data into binary values based on the occurrence of categories. This is used only when there is no order/relationship between the categories. For example - In our data, we have a variable called <em>Gender</em> which consists of values- Male and Female. Since there is no order, we use One Hot Encoding.</li></ul><p>Let’s continue the hands-on part -</p><pre>df.drop(&#39;Employee ID&#39;,axis=1,inplace=True)</pre><ul><li>We will be using ‘Date of Joining’ in the feature engineering part.</li></ul><pre>dat = pd.Series(df[&#39;Date of Joining&#39;])<br>df.drop(&#39;Date of Joining&#39;,axis=1,inplace=True)</pre><pre>#Applying One hot encoding -</pre><pre>cat = []<br>num = []</pre><pre>for feat in df.columns:<br> if(df[feat].dtype==&#39;object&#39;):<br>  cat.append(feat)<br> else:<br>  num.append(feat)</pre><pre>encoder = OneHotEncoder(handle_unknown=&#39;ignore&#39;, sparse=False)<br>dummy_df = pd.DataFrame(encoder.fit_transform((df[cat])))<br>dummy_df.index = df.index<br>dummy_df.columns = encoder.get_feature_names(cat)<br>df.drop(cat,axis=1,inplace = True)</pre><pre>df = pd.concat([df,dummy_df],axis=1)</pre><pre>df.drop([&#39;WFH Setup Available_No&#39;,&#39;Gender_Male&#39;,&#39;Company Type_Product&#39;],axis=1,inplace=True)</pre><h4><strong>Dealing with missing values</strong></h4><p>Some of the methods that are used to handle null values in the data are -</p><ul><li><strong>Random Sample Imputation: </strong>It involves taking random observations from the dataset and using it to replace null values. It is appropriate when the data are missing completely at random (<a href="https://stefvanbuuren.name/fimd/sec-MCAR.html">MCAR</a>).</li><li><strong>Imputing Mean/Median: </strong>This is the method of filling the missing data with the mean/median of their respective attribute. It is appropriate when the data are missing at random (<a href="https://stefvanbuuren.name/fimd/sec-MCAR.html">MAR</a>).</li><li><strong>Predicting the null values: </strong>When there is high collinearity between the attribute with null values and the one free of null values, it is suitable to use Linear Regression to predict and fill the missing values.</li></ul><pre>df.isnull().sum()</pre><ul><li>The null values in all the columns are missing at random. Probably the employees didn’t want to share/ hesitated to share the information like stress score, burn rate, and the working hours.</li><li>There are missing values in <em>Burn Rate</em> and <em>Mental Fatigue score</em>. Let’s visualize their distribution.</li></ul><pre>sns.heatmap(df.isnull())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*MHId3EYvsHmJ0INeIP4L2A.png" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><pre>df[df[‘Burn Rate’].isnull()][‘Mental Fatigue Score’].isnull().sum()</pre><ul><li>We observe that the null values in the two features - do not occur simultaneously (only 172 values occur simultaneously) for a given sample. So we impute the missing value of target through <em>Mental Fatigue Score</em> and vice versa using <strong>Linear Regression</strong> due to their strong collinearity. We will use median/mean imputation for the rest of the missing values (Where null values occur simultaneously).</li></ul><pre>x1 = pd.DataFrame(df[df[&#39;Burn Rate&#39;].isnull()][&#39;Mental Fatigue Score&#39;])</pre><pre>x1[&#39;Mental Fatigue Score&#39;].fillna(x1[&#39;Mental Fatigue Score&#39;].median(),inplace=True) <br>#Filling the simultaneously occuring null values for Mental Fatigue Score with median imputation.</pre><pre>#Imputation using Linear Regression</pre><pre>df_new = df[[&#39;Mental Fatigue Score&#39;,&#39;Burn Rate&#39;]] #the training data with no null values</pre><pre>df_new.dropna(inplace=True)</pre><ul><li>Using <em>Mental Fatigue Score </em>as the independent feature and <em>Burn Rate </em>as the dependent feature in the linear regression model.</li></ul><pre>X = df_new[[&#39;Mental Fatigue Score&#39;]] # X.shape should be (N, M) where M &gt;= 1<br>y = df_new[&#39;Burn Rate&#39;]</pre><pre>model = LinearRegression()<br>model.fit(X,y)</pre><pre>pred = model.predict(x1) <br>#the values to be filled in nan values of burn rate</pre><pre>ind = df[df[&#39;Burn Rate&#39;].isnull()].index</pre><pre>for j,i in enumerate(ind):<br> df[&#39;Burn Rate&#39;].iloc[i] = pred[j] <br>#Filling the missing values with   the predicted values</pre><ul><li>Apply the same steps to fill the missing values in <em>Mental Fatigue Score. </em>This time we use<em> Burn Rate </em>as the independent feature and<em> Mental Fatigue Score </em>as the dependent feature for the linear regression model.</li></ul><pre>X = df_new[[&#39;Burn Rate&#39;]] # X.shape should be (N, M) where M &gt;= 1<br>y = df_new[&#39;Mental Fatigue Score&#39;]</pre><pre>model = LinearRegression()<br>model.fit(X,y)</pre><pre>x2 = pd.DataFrame(df[df[&#39;Mental Fatigue Score&#39;].isnull()][&#39;Burn Rate&#39;])<br>pred = model.predict(x2)<br>ind = df[df[&#39;Mental Fatigue Score&#39;].isnull()].index</pre><pre>for j,i in enumerate(ind):<br> df[&#39;Mental Fatigue Score&#39;].iloc[i] = pred[j]</pre><ul><li>We are left with missing values in <em>Resource Allocation</em>. We will use the same technique i.e. Linear Regression to impute the null values of <em>Resource Allocation </em>since it has a high correlation with <em>Designation. </em>The independent feature will be<em> Designation </em>and the target feature will be<em> Resource Allocation.</em></li></ul><pre>df_new = df[[&#39;Designation&#39;,&#39;Resource Allocation&#39;]]<br>df_new.dropna(inplace=True)</pre><pre>X = df_new[[&#39;Designation&#39;]]<br>y = df_new[&#39;Resource Allocation&#39;]</pre><pre>model = LinearRegression()<br>model.fit(X,y)</pre><pre>des = pd.DataFrame(df[df[&#39;Resource Allocation&#39;].isnull([&#39;Designation&#39;])</pre><pre>pred2 = model.predict(des)</pre><pre>ind = df[df[&#39;Resource Allocation&#39;].isnull()].index</pre><pre>for j,i in enumerate(ind):<br> df[&#39;Resource Allocation&#39;].iloc[i] = pred2[j]</pre><h4><strong>Outliers</strong></h4><p>It’s time to remove the outliers that we detected in few features during EDA using boxplot visualization.</p><p>Outliers are points in the data that fall far away from the other observation. The presence of outliers can spoil and mislead the training process of the model. Hence it is important to handle them.</p><p>Here we’ll use log transformation on data to get rid of the outliers. We’ll use this method on the features with the modulus of their <a href="https://www.investopedia.com/terms/s/skewness.asp">skewness</a> &gt;0.5 since data with skewness less than 0.5 follows a fairly symmetrical distribution.</p><pre>for feat in num:<br> if((np.abs(df[feat].skew())&gt;0.5)):<br>  print(feat)</pre><pre>df[feat] =np.log1p(df[feat])[0]</pre><p><strong>Output: </strong>There are no such features that have skewness greater than 0.5. We have completed data preparation and EDA!</p><h3><strong>3. Feature Engineering</strong></h3><p>The process of Feature Engineering involves the extraction of features from the raw data based on its insights. Moreover, feature engineering also involves feature selection, feature transformation, and feature construction to prepare input data that best fits the machine learning algorithms. Feature Engineering influences the results of the model directly and hence it’s a crucial part of data science.</p><p>Let’s use <em>Date of Joining to </em>create a new feature - <em>Days Spent </em>that would contain the information about how many days has the employee worked in the company till date. <em>Burn Rate</em> for an employee who has worked for years will be perhaps much higher than a newly joined employee.</p><pre>present = date.today()</pre><pre>date_df = pd.DataFrame(dat)</pre><pre>date_df[&#39;Date of Joining&#39;] = date_df[&#39;Date of Joining&#39;].apply(lambda x : datetime.strptime(x,&quot;%Y-%m-%d&quot;).date())</pre><pre>df[&#39;Days Spent&#39;] = present - date_df[&#39;Date of Joining&#39;] <br>#Get the total number of days spent in the company</pre><pre>df[&#39;Days Spent&#39;] = df[&#39;Days Spent&#39;].apply(lambda x : int(str(x).split(&quot; &quot;)[0]))</pre><p>Let’s check if the new feature has any effect on the target -</p><pre>sns.scatterplot(x = df[‘Days Spent’],y = df[‘Burn Rate’]);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*N3YF26sAshdUgcvaQspXCw.jpeg" /><figcaption><strong>Output </strong>(Image by Author)</figcaption></figure><ul><li>The scatterplot shows that there are no trends in the data of <em>Burn Rate</em> vs <em>Days Spent</em>. So we drop <em>Days Spent</em> as it has no importance in the data.</li></ul><pre>df.drop(‘Days Spent’,axis=1,inplace=True)</pre><h3><strong>4. Model Building and Evaluation</strong></h3><p>Let’s find the performance of different ensemble techniques on the data - 1.XGBoost 2.AdaBoost 3.RandomForest.</p><p>Please read this <a href="https://www.analyticsvidhya.com/blog/2021/08/ensemble-stacking-for-machine-learning-and-deep-learning/">blog</a> for an understanding of ensemble techniques and their different types. We’ll use the ‘<a href="https://www.geeksforgeeks.org/ml-r-squared-in-regression-analysis/#:~:text=R%2Dsquared%20is%20a%20statistical,for%20r%2Dsquare%20is%201.&amp;text=The%20more%20the%20value%20of,than%20the%20average%20fitted%20model.">R-squared</a>’ as the metric since we are training a regression model.</p><pre>model1 = XGBRegressor()</pre><pre>X = df.drop(&#39;Burn Rate&#39;,axis=1)<br>y = df[&#39;Burn Rate&#39;]</pre><pre>score = cross_val_score(model1,X,y,cv = 5,scoring = &#39;r2&#39;) <br>#K fold cross validation</pre><pre>score.mean()</pre><p><strong>Output: </strong>0.9323</p><pre>model2 = AdaBoostRegressor()<br>score = cross_val_score(model2,X,y,cv = 5,scoring = ‘r2’)<br>score.mean()</pre><p><strong>Output: </strong>0.9134</p><pre>model3 = RandomForestRegressor()<br>score = cross_val_score(model3,X,y,cv = 5,scoring = ‘r2’)<br>score.mean()</pre><p><strong>Output: </strong>0.9216</p><p>All the 3 algorithms have given pretty good results. We can further improve the results by improvising the preceding stages of the lifecycle like <strong>scaling</strong> the data, changing the technique of <strong>null value imputation </strong>and <strong>categorical encoding</strong>, performing <strong>feature selection</strong> to get rid of unimportant features, <strong>hyperparameter optimization</strong>, etc.</p><h4><strong>Hyperparameter optimization</strong> -</h4><p>It is the technique of selecting the optimal set of hyperparameters for the ML algorithm that gives the best results on the chosen metric. Random Search, Grid Search, Optuna, etc. are some of the methods to tune the hyperparameters of a learning algorithm.</p><p>Let’s optimize the hyperparameters of XGBoost -</p><pre>regressor = XGBRegressor()<br>param_grid = {<br>&quot;colsample_bytree&quot; : np.arange(0.1,1,0.1),<br>&quot;gamma&quot; : np.arange(0.01,0.1,0.01),<br>&quot;learning_rate&quot; : np.arange(0.01,0.1,0.01),<br>&quot;max_depth&quot; : np.arange(2,10),<br>&quot;n_estimators&quot; : np.arange(1500,2500,100),<br>&quot;reg_alpha&quot; : np.arange(0,1,0.1),<br>&quot;reg_lambda&quot; : np.arange(0,1,0.1),<br>&quot;subsample&quot; : np.arange(0.1,1,0.1),<br>&quot;silent&quot;  : [1],<br>&quot;nthread&quot; : [-1],<br>}</pre><pre>model  = RandomizedSearchCV(<br>estimator= regressor,<br>param_distributions= param_grid,<br>n_iter=20,<br>scoring = &#39;r2&#39;,<br>verbose= 3,<br>n_jobs=1,<br>cv = 5)</pre><pre>model.fit(X,y)<br>model.best_score_</pre><p><strong>Output: </strong>0.9327</p><ul><li>The hyperparameter tuning has given a slight increase in the model’s score. So we go with this tuned model.</li></ul><p>Save the model -</p><pre>filename = ‘bunrout_model_xgb.pkl’<br>pickle.dump(model, open(filename, ‘wb’))</pre><p>It’s time for model deployment! Complete <a href="https://github.com/YashK07/Burnout-Rate-Prediction-Heroku/blob/main/Burn_out_rate_Model_Building.ipynb">code</a> for the previous stages of the project.</p><h3><strong>5. Deployment</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*OPWi1BO0I70iBolmStff_w.png" /><figcaption>Pic Source: <a href="https://bit.ly/3EDtb4k">https://bit.ly/3EDtb4k</a></figcaption></figure><h4>Introduction</h4><p>Now our aim is to build a web app that takes input information from the user and gives the prediction of Burn Rate for the user. To build it, we’ll use Flask, an API of Python that allows us to build up applications. After building the app, we’ll deploy it on Heroku.</p><p><strong>Note: </strong>Other platforms where we can deploy the ML models are - Amazon AWS, Microsoft Azure, Google Cloud, etc.</p><p>The input information that we would collect from the user will be the features on which our Burn Rate predictive model has been trained -</p><ul><li>Designation of work of the user in the company in the range - [0–5]: 5 is the highest designation and 0 is the lowest.</li><li>The number of working hours.</li><li>Mental Fatigue Score in the range - [0–10]: How much fatigue/tired does the user usually feels during working hours.</li><li>Gender: Male/Female</li><li>Type of the company: Service/Product</li><li>Do you work from home: Yes/No</li></ul><h4>Building app on the flask</h4><p>Create a .py file for the app. Import all the important libraries -</p><p>from flask import Flask,render_template</p><pre>import os<br>from flask import request<br>import pickle<br>import pandas as pd<br>import numpy as np<br>from xgboost import XGBRegressor</pre><ul><li>Initiate the Flask app and load the save model -</li></ul><pre>app = Flask(__name__) #Initialize the flask app<br>model = pickle.load(open(“model\\bunrout_model_xgb.pkl”, ‘rb’))</pre><ul><li>Do the app routing for the home function that will render HTML page -</li></ul><pre>@app.route(‘/’)<br>def home():<br> return render_template(‘index.html’)</pre><ul><li>Create a function ‘predict’ that will be the most important backend work i.e to give back predictions based on the user input -</li></ul><pre>def predict():</pre><pre> if(request.method==&quot;POST&quot;):<br>  int_feat = [&#39;Designation&#39;, &#39;Resource Allocation&#39;, &#39;Mental Fatigue Score&#39;, &#39;Gender&#39;, &#39;Company Type&#39;, &#39;WFH Setup Available&#39;]</pre><pre>  l = []</pre><pre>  for i in int_feat:<br>   val = int(request.form[i])<br>   l.append(val)</pre><pre>  #convert into array of shape -&gt; (1,6)</pre><pre> feat_arr = np.array(l).reshape(-1,1).reshape(1,6)</pre><pre> input = pd.DataFrame(feat_arr,columns = [&#39;Designation&#39;, &#39;Resource Allocation&#39;, &#39;Mental Fatigue Score&#39;, &#39;Gender_Female&#39;, &#39;Company Type_Service&#39;, &#39;WFH Setup Available_Yes&#39;])</pre><pre> prediction = float(model.predict(input)[0])<br> prediction = round(prediction, 2)</pre><pre> #top 25 percentile<br> if(prediction&lt;=0.3):<br>  feedback1 = &quot;Fantastic! You have a low burnout of {} .&quot;.format(prediction)<br>  <br>  return render_template(&quot;index_1.html&quot;,color = &quot;color:#33CC00;&quot;,feedback = feedback1)</pre><pre>#top 25 percentile to 75 percentile<br> elif((prediction&gt;0.3) &amp; (prediction&lt;=0.59)):<br>  feedback2 = &quot;Not bad...You have a moderate burnout of {} .&quot;.format(prediction)<br>   <br>  return render_template(&quot;index_1.html&quot;,color = &quot;color:#339900;&quot;,feedback = feedback2)</pre><pre>#top 75 percentile to 90 percentile<br> elif((prediction&gt;0.59) &amp; (prediction&lt;=0.80)):<br>  feedback3 = &quot;Oops!! You have a high burnout of {} .&quot;.format(prediction)<br>  <br>  return render_template(&quot;index_1.html&quot;,color = &quot;color:#FF0000;&quot;,feedback = feedback3)</pre><pre>#top 90 percentile to 99 percentile<br> elif((prediction&gt;0.8) &amp; (prediction&lt;=0.9)):<br>  feedback4 = &quot;Ouch!!! You have a very high burnout of {} .&quot;.format(prediction)<br>  <br>  return render_template(&quot;index_1.html&quot;,color =&quot;color:#CC0000;&quot;,feedback = feedback4)</pre><pre>#top 99 percentile<br> else:<br>  feedback5 = &quot;Sorry! You have an extremly high burnout of {} .&quot;.format(prediction)<br>  <br>  return render_template(&quot;index_1.html&quot;,color =&quot;color:#990000;&quot;,feedback = feedback5)</pre><ul><li>You can display some messages along with the prediction as in the above code to describe the degree of their burnout rate.</li><li>Define the main function -</li></ul><pre>if __name__ == &quot;__main__&quot;:<br> app.run(&#39;debug&#39;==True)</pre><p>Your Flask app is ready! Go through <a href="https://github.com/YashK07/Burnout-Rate-Prediction-Heroku/blob/main/templates/index_1.html">index.html</a> for the frontend part of the webapp.</p><p><strong>Note: </strong>Create the index.html and save it in a directory called — templates.</p><h4><strong>Deploying app on Heroku</strong></h4><p>Deployment on Heroku requires a reuirement.txt file. The requirements for this project -</p><pre>Flask==1.1.1<br>pandas==1.0.1<br>numpy==1.18.1<br>xgboost==0.90<br>gunicorn==20.1.0<br>scikit-learn&gt;=0.18</pre><p>Now create a Procfile which is a text file without a .txt extension. It defines the commands that are implemented by the app on startup. Procfile for this project -</p><pre>web: gunicorn app:app</pre><p>Now push the project into your Github and connect the repo to Heroku for deployment -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hu4JzmqdJTStsMX73Z9h4A.jpeg" /><figcaption>Deploying the Github repo of the project on Heroku (Image by Author)</figcaption></figure><p>Congratulations! We have successfully created and deployed our web application ✌️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SUBSfEmRq5EpDt7S" /><figcaption>Burnout Rate Prediction web app (Image by author)</figcaption></figure><p>Link of the web app: <a href="https://burnout-rate-prediction-api.herokuapp.com/">https://burnout-rate-prediction-api.herokuapp.com</a></p><p>It can be accessed from anywhere and be used to keep a check on the mental health of the employees.</p><h4><strong>Improvisation</strong></h4><p>After the deployment, improvising the lifecycle stages of the project for delivering the best solution is yet another important stage.</p><h3>Conclusion</h3><p>So we have come to the end of the guide. From understanding the problem statement to delivering an end-to-end solution, we have all covered this in a single blog!</p><p>Do check the whole project code from — <a href="https://github.com/YashK07/Burnout-Rate-Prediction-Heroku">https://github.com/YashK07/Burnout-Rate-Prediction-Heroku</a>. Please do start it 🌟 if you feel the app is worth it. Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee1b2dd5d45e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/life-cycle-of-data-science-project-hands-on-guide-ee1b2dd5d45e">Life-cycle of Data Science project: Hands-on guide</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[India vs Australia ODI cricket analysis using Data Visualization]]></title>
            <link>https://medium.com/analytics-vidhya/india-vs-australia-odi-cricket-analysis-using-data-visualization-4644d0dab2da?source=rss-abe69fbfb436------2</link>
            <guid isPermaLink="false">https://medium.com/p/4644d0dab2da</guid>
            <category><![CDATA[india-vs-australia]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[cricket]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <dc:creator><![CDATA[Yash Khandelwal]]></dc:creator>
            <pubDate>Tue, 24 Nov 2020 14:58:01 GMT</pubDate>
            <atom:updated>2020-11-27T16:45:10.599Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lCa5MKZc8oEqjt6wcEFuDQ.jpeg" /></figure><p>In this blog post, I will make a short analysis of India vs Australia One Day matches of past 35 years. The dataset has been taken from Kaggle. My python code for the following analysis and data visualizations -<a href="https://www.kaggle.com/yashk07/ind-vs-aus-odi-cricket-analysis-data-viz">https://www.kaggle.com/yashk07/ind-vs-aus-odi-cricket-analysis-data-viz</a></p><p><strong>The Dataset -</strong><a href="https://www.kaggle.com/palashiitk/cricket-odi-results">https://www.kaggle.com/palashiitk/cricket-odi-results</a></p><p>Dataset contain match records of 27 teams that have ever played One Day International cricket which include 24 countries and 3 tournament teams :- ICC World XI, Asia XI and Africa XI. I have used the data of India vs Australia ODI matches for the analysis.</p><h3><strong>Why India vs Australia?</strong></h3><p>India vs Australia is one of the biggest rivalries of International cricket. Limited overs format of the game between the two sides has always been a thrilling contest and hence producing some interesting stats to look for.</p><h3>Lets Begin!</h3><p><em>For the following Data Visualization, I have used these python libraries -Matplotlib, Seaborn and Plotly.</em></p><h4>Bar, Count, and Distribution Plots</h4><p>Lets look at the team scores when batting 1st and when chasing the target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FTeTRkDK4XZ9QBPe4LPCHQ.jpeg" /><figcaption>Batting stats in the 1st innings of Ind vs Aus matches</figcaption></figure><p>12 times India has hit 300+ runs when batting 1st in the match. While Australia leads with 17 times 300+ runs on board in the 1st innings of match. Shows that the bowling side of India has not really been effective against Aussies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2Ad1_J5VeFs9_s0wWG31Q.jpeg" /><figcaption>Batting stats when chasing in Ind vs Aus matches</figcaption></figure><p>18 times India has chased 275+ runs while Australia has chased 275+ runs 11 times. With likes of MS Dhoni, Virat Kohli and many other greats of modern day cricket, India has been highly consistent in chasing big scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ehG7gAuONF5wSO-rDVmDkw.png" /><figcaption><strong>Trends in results of Ind vs Aus matches while batting first in different venues -</strong></figcaption></figure><p>Majority of matches have been away games for India. The plot also shows how the pitch has favored match results in different venues.</p><p><strong>Boxplots and Stripplots</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*zza2pM9aaG5o9XDUVw_lkw.png" /><figcaption>Runs scored in 1st innings vs Years</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*ADR2fGWka2y7iBekIBRUqQ.png" /><figcaption>Runs scored in 2nd innings vs Years</figcaption></figure><p>These plots depict how much the game has changed over years. While average batting score was 150 in 1985, it is being consistently near 300 after 2013. India vs Australia matches has therefore been exciting to watch!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*RqrFw1Y2chnKmL1je_6Dfg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*W0JFCqbJ_TlaiL9LTfHRcA.png" /></figure><p>While India has been consistent at chasing scores at higher run rate(5–7 and 7+ on quite many games) and reaching huge targets which is evident from the previous plots, Australia has been more consistent in batting fearlessly in the 1st innings of match.</p><h4>Pieplots</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*STD-37OTmTMZ_fWouosOyQ.png" /><figcaption>Results for India while batting 1st against Australia</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*FAG_kzlFY3BpkGd6ZspSkQ.png" /><figcaption>Results for India while chasing against Australia</figcaption></figure><p>Australia has won 41 out of 70 matches while chasing and 29 out of 50 matches while defending the score with a win percentage of 58 or above. This shows how much balanced has been the yellow side throughout the 35 years of ODI cricket.</p><h3>Final Countplot</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eJvvh2FGRVZ6XhNfvkHjlg.png" /></figure><p><strong>Clearly, Australia has overshadowed India in the early years of ODI cricket and in between 2000 - 2011 which is often referred as the ‘golden era’ of Aussies led by Ricky Ponting. But in the recent years, India has dethroned Australian dominance from the One Day format of the game. It has been a tough competition over all.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4644d0dab2da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/india-vs-australia-odi-cricket-analysis-using-data-visualization-4644d0dab2da">India vs Australia ODI cricket analysis using Data Visualization</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>