<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shabeel Kandi on Medium]]></title>
        <description><![CDATA[Stories by Shabeel Kandi on Medium]]></description>
        <link>https://medium.com/@shabeelkandi?source=rss-4bb0bf33aed0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-MZBNajGHcRgV22fdjCTFQ.jpeg</url>
            <title>Stories by Shabeel Kandi on Medium</title>
            <link>https://medium.com/@shabeelkandi?source=rss-4bb0bf33aed0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 12:29:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shabeelkandi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Ultimate Guide to Setting Up Next.js and FastAPI]]></title>
            <link>https://medium.com/analytics-vidhya/building-full-stack-web-applications-with-python-the-ultimate-guide-to-setting-up-next-js-f090684377a1?source=rss-4bb0bf33aed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f090684377a1</guid>
            <category><![CDATA[boilerplate]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[nextjs]]></category>
            <category><![CDATA[python-web-developer]]></category>
            <dc:creator><![CDATA[Shabeel Kandi]]></dc:creator>
            <pubDate>Tue, 04 Jul 2023 22:55:47 GMT</pubDate>
            <atom:updated>2023-07-11T18:27:14.772Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*oq82hKNzEOX-pd0THOH4ag.png" /></figure><p>In this article, we will show you how to incorporate a FastAPI project into a Next.js application and deploy the entire project on Vercel.</p><p>In the past, we used the API Routes feature in Next.js to handle backend functionality. This involved creating custom request handlers for each route using JavaScript or TypeScript in the route.ts file. When deploying to Vercel, the code in the src/app/api directory would be deployed to a Node.js server.</p><p>Now, we have the option to create a separate api folder in the project’s root directory for our FastAPI project. This allows us to write the backend code using Python. When deploying the project to Vercel, the platform automatically detects the Python project in the api directory and deploys it as a Python serverless function.</p><h3>First lets see the Demo :</h3><p><a href="https://nextjs-fastapi-starter.vercel.app/">Create Next App</a></p><h3>Step 1: Setting up Next.js Project</h3><p>To create a Next.js project, use one of the following commands based on your package manager:</p><pre># For PNPM<br>pnpm create next-app nextjs-fastapi<br># For NPX<br>npx create-next-app@latest nextjs-fastapi<br># For Yarn<br>yarn create next-app nextjs-fastapi</pre><p>After running the command, select “Yes” for all prompts and choose the first import alias option. This enables the necessary features for your Next.js project.</p><p>Once the project is generated and dependencies are installed, open it in your preferred IDE or text editor.</p><h3>Step 2: Setting up the FastAPI Project</h3><p>To get started with the FastAPI project, follow these steps:</p><ul><li>Install the necessary modules and create a virtual environment using the command “python3 -m venv venv” in your IDE’s integrated terminal.</li><li>Create a .gitignore file to exclude certain files and directories from commits.</li><li>In the project’s root directory, create a folder named “api”. Make sure to use this specific name as Vercel expects it.</li><li>Inside the “api” folder, create a file called “index.py”. This file acts as the entry point for the Python project in Vercel.</li><li>Open the “index.py” file to automatically activate the required extensions and virtual environment in your IDE or text editor. If this doesn’t happen, manually activate the virtual environment by running “source venv/bin/activate” in the terminal.</li><li>Once the virtual environment is activated, install the necessary dependencies for your FastAPI project by running the command “pip install fastapi ‘uvicorn[standard]’” in a new terminal.</li><li>Now, let’s set up a basic FastAPI server with a health checker endpoint. We’ll make requests to this endpoint to verify if the requests are being properly proxied to the FastAPI server. Open the api/index.py file and add the following code:</li></ul><p><strong>api/index.py</strong></p><pre>from fastapi import FastAPI<br><br>app = FastAPI(docs_url=&quot;/api/docs&quot;, openapi_url=&quot;/api/openapi.json&quot;)<br><br><br>@app.get(&quot;/api/healthchecker&quot;)<br>def healthchecker():<br>    return {&quot;status&quot;: &quot;success&quot;, &quot;message&quot;: &quot;Integrate FastAPI Framework with Next.js&quot;}</pre><h3>Step 3: Proxy Requests from Next.js to the FastAPI App</h3><p>To configure API request handling for our Next.js project, we need to make some changes to the next.config.js file. Follow these steps:</p><ul><li>Open the next.config.js file in your project.</li><li>Replace the existing code with the following snippet:</li></ul><pre>/** @type {import(&#39;next&#39;).NextConfig} */<br>const nextConfig = {<br>  rewrites: async () =&gt; {<br>    return [<br>      {<br>        source: &quot;/api/:path*&quot;,<br>        destination:<br>          process.env.NODE_ENV === &quot;development&quot;<br>            ? &quot;http://127.0.0.1:8000/api/:path*&quot;<br>            : &quot;/api/&quot;,<br>      },<br>    ];<br>  },<br>};<br><br>module.exports = nextConfig;<br><br>In the code snippet above, we define a rewrite rule that redirects any request matching the pattern /api/:path* to a specific destination. During development, the destination is set to http://127.0.0.1:8000/api/:path*, which means that API requests will be forwarded to a FastAPI server running locally. In a production environment, the destination is set to “/api/”, indicating that API requests will be proxied to the same domain as the Next.js app.</pre><p>In the code snippet above, we have a rewrite rule that handles requests matching the pattern /api/:path*. These requests will be redirected to a specific destination.</p><p>During development, the destination URL is set to <a href="http://127.0.0.1:8000/api/:path*">http://127.0.0.1:8000/api/:path*</a>. This means that API requests made to our Next.js app will be sent to the FastAPI server running locally at <a href="http://127.0.0.1:8000/">http://127.0.0.1:8000</a>.</p><h3>Step 4: Running the projects on localhost</h3><p>To run the Next.js and FastAPI servers simultaneously instead of using separate terminals, we can use a package called concurrently. Follow these steps to set it up:</p><ul><li>In your integrated terminal, execute the command that corresponds to your package manager:</li></ul><pre># For PNPM<br>pnpm add concurrently<br><br># For NPM<br>npm i concurrently<br><br># For Yarn<br>yarn add concurrently</pre><ul><li>Once the package is installed, open the package.json file.</li><li>Replace the existing dev script in the package.json file with the following code:</li></ul><pre>{<br> &quot;scripts&quot;: {<br>    &quot;fastapi-dev&quot;: &quot;pip3 install -r requirements.txt &amp;&amp; python3 -m uvicorn api.index:app --reload&quot;,<br>    &quot;next-dev&quot;: &quot;next dev&quot;,<br>    &quot;dev&quot;: &quot;concurrently \&quot;pnpm run next-dev\&quot; \&quot;pnpm run fastapi-dev\&quot;&quot;<br>  },<br>}</pre><p>To run the FastAPI and Next.js servers concurrently, follow these steps:</p><ul><li>Make sure to replace “pnpm” with your appropriate package manager.</li><li>The “fastapi-dev” script installs the required Python packages and starts the FastAPI server with hot-reloading enabled.</li><li>The “next-dev” script starts the Next.js development server with hot-reloading enabled.</li><li>The “dev” script combines the execution of both servers using the concurrently package.</li><li>To set up the “fastapi-dev” script, you need to generate the requirements.txt file. Execute the command pip freeze &gt; requirements.txtto generate this file, which lists all the installed Python packages and their versions.</li></ul><p>After generating the requirements.txt file, run “pnpm dev” to start both the FastAPI and Next.js servers concurrently. Open your browser and go to <a href="http://localhost:3000/api/healthchecker">http://localhost:3000/api/healthchecker</a>. If you see the JSON response from the FastAPI server, it means that the Next.js app is properly forwarding requests to the FastAPI server.</p><h3>Step 5: Deploy the project on Vercel and share it with others!</h3><p>Now that we are sure that Next.js is forwarding API requests to the FastAPI server correctly, let’s deploy the project on Vercel and test if it works in a live environment. Deploying on Vercel is easy and beginner-friendly.</p><p>First, create a GitHub repository for the project and upload the source code. Then, log in to your Vercel account and click on “Add New”. From the dropdown menu, select “Project” and choose the repository where you uploaded the code.</p><p>You will be taken to the configuration page, where you can customize project settings like name and framework. Vercel will automatically detect Next.js and provide default configurations, which are usually fine. Once you’re satisfied, click “Deploy”.</p><p>Now, wait for Vercel to build and deploy the project. Once it’s done, you’ll get a domain for your project. Visit that domain, and you should see your Next.js app live. Go to the /api/healthchecker endpoint, and you should see the JSON object from the FastAPI server.</p><p>If you can see the JSON object, it means you followed all the steps correctly and the project is working as expected.</p><h3>Useful Links</h3><ul><li><a href="https://github.com/shabeelkandi/nextjs-fastapi">https://github.com/shabeelkandi/nextjs-fastapi</a></li><li><a href="https://vercel.com/templates/next.js/nextjs-fastapi-starter">https://vercel.com/templates/next.js/nextjs-fastapi-starter</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f090684377a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/building-full-stack-web-applications-with-python-the-ultimate-guide-to-setting-up-next-js-f090684377a1">The Ultimate Guide to Setting Up Next.js and FastAPI</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chat with an Excel dataset with OpenAI and LangChain]]></title>
            <link>https://medium.com/analytics-vidhya/chat-with-an-excel-dataset-with-openai-and-langchain-5520ce2ac5d3?source=rss-4bb0bf33aed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5520ce2ac5d3</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[gpt-4]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Shabeel Kandi]]></dc:creator>
            <pubDate>Sat, 08 Apr 2023 08:07:28 GMT</pubDate>
            <atom:updated>2023-04-24T18:33:39.695Z</atom:updated>
            <content:encoded><![CDATA[<p>In the article, I take you through how you can talk to any .csv dataset using LangChain and OpenAI api, in just about 10 lines of code.</p><p>Okay, step 1 is to get an OpenAI API key. You probably need to set up billing as well. You can use the link: <a href="https://platform.openai.com/account/api-keys">https://platform.openai.com/account/api-keys</a></p><p>Once that is sorted, make sure you install langchain, openai, chromadb and tiktoken python libraries.</p><pre>!pip install -q langchain openai chromadb tiktoken</pre><p>Great. In the next code snippet, I load the libraries, API key (use the one with the one you create), and a .csv dataset. For now, make sure the dataset is not too large. Anything above 2000 rows and you get wonky results. We are still in the early stages, do not worry, we will soon be able to get past this barrier.</p><pre>from langchain.document_loaders import CSVLoader<br>from langchain.indexes import VectorstoreIndexCreator<br>from langchain.chains import RetrievalQA<br>from langchain.llms import OpenAI<br>import pandas as pd<br>import os<br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;sk-xxxx&quot;<br><br># Load the dataset<br># link to download data I used: https://www.kaggle.com/datasets/ashishraut64/indian-startups-top-300?resource=download<br>loader = CSVLoader(file_path=&#39;Startups1.csv&#39;)    </pre><p>Next, we load this dataset to create indices for this document. LangChain uses ChromaDB here, since we have not specified any particular vector database. In simple terms, vectorising this dataset, puts it in a format that can be used by a language model to query this data (basically allowing you to talk to the data)</p><pre># Create an index using the loaded documents<br>index_creator = VectorstoreIndexCreator()<br>docsearch = index_creator.from_loaders([loader])</pre><p>We want a question-answering chain since we want to ask the data questions.</p><pre># Create a question-answering chain using the index<br>chain = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type=&quot;stuff&quot;, retriever=docsearch.vectorstore.as_retriever(), input_key=&quot;question&quot;)</pre><p>And that’s it. You have done it, you have build a model to query the dataset. See the below gif for results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-TX1xRvRsZDa8iqAhcDag.gif" /></figure><p>Checkout my github for the full code:</p><p><a href="https://github.com/shabeelkandi/Chat-with-an-Excel-dataset-with-LangChain">https://github.com/shabeelkandi/Chat-with-an-Excel-dataset-with-LangChain</a></p><p>Want to take this to the next level for your use case? Get in touch!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5520ce2ac5d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/chat-with-an-excel-dataset-with-openai-and-langchain-5520ce2ac5d3">Chat with an Excel dataset with OpenAI and LangChain</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flask For Data Scientists : Building a Searchable File Directory]]></title>
            <link>https://medium.com/analytics-vidhya/flask-for-data-scientists-building-a-searchable-file-directory-25b286332e80?source=rss-4bb0bf33aed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/25b286332e80</guid>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[flask-sqlalchemy]]></category>
            <dc:creator><![CDATA[Shabeel Kandi]]></dc:creator>
            <pubDate>Thu, 08 Apr 2021 23:09:12 GMT</pubDate>
            <atom:updated>2023-04-24T18:34:16.952Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Flask For Data Scientists </strong>: Building a Searchable File Directory</h3><p>You do not have to be an expert python web developer, but the using flask and the ability to build simple but powerful web apps with it, is a great tool in our toolkit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*Qc9AU5f933ALWMK3K_fOCg.png" /><figcaption>Visual of the simple web app once completed.</figcaption></figure><p>In this article I show you how to build a simple web application using flask. The app will be used to upload a file to a remote path, and then search and download them using this web app.</p><p>To jump straight to the code go to the github link below:</p><blockquote><a href="https://github.com/shabeelkandi/flask-file-upload-search-app">https://github.com/shabeelkandi/flask-file-upload-search-app</a></blockquote><p>In the following lines, I take you through some key aspects of the web app, and not an extensive review of the code.</p><p>First things first. <strong>What is our use case?</strong></p><p><em>Answer: </em>User uploads file. See’s a table of all files, and is able to search by filename and delete or download a file.</p><h3>Code Breakdown</h3><h4>Defining the file model.</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a9e00cd3bc76361f103f515379727b2/href">https://medium.com/media/1a9e00cd3bc76361f103f515379727b2/href</a></iframe><p>In the above script we define what a file means: it should have an id, name, url(https://medium.com/path) and a created date.</p><h4>Homepage of Web App : Upload feature and table</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eec2360b1b3d7055a96689e3c4da4a8b/href">https://medium.com/media/eec2360b1b3d7055a96689e3c4da4a8b/href</a></iframe><p>The “/” route referes to the home page of the web app, for which we are building the function <em>upload_file().</em></p><p>On the home page if we get a ‘POST’ which is a click on upload button (see the upload.html file under templates folder on github link), we do some basic checks to see what has been passed after which we do a db.session command to add the file path to the database and then commit this to the database.</p><p>Each time a file is uploaded to the database, the files class is initalised (child of db.Model), so we the date created, id is assigned.</p><h4>Search and Delete Features</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a303a59804e55af2d7fc01ede4fa3568/href">https://medium.com/media/a303a59804e55af2d7fc01ede4fa3568/href</a></iframe><p>Now the simple stuff is to build a search and delete feature for our application, which is quite simple since the data is already there in the database.</p><p>Search is pretty much just a sql query of the sqlite database, while delete is based on a db.delete command.</p><p>Tying this to the the html files gives a better idea on how the web app will be displayed.</p><p>All we have to do finally to launch the web app is to run main.py</p><p><em>If something is not working please comment below and I will get back to you ;)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25b286332e80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/flask-for-data-scientists-building-a-searchable-file-directory-25b286332e80">Flask For Data Scientists : Building a Searchable File Directory</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Analyze A/B Test Results?]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-analyze-a-b-test-results-b93c9eafbc61?source=rss-4bb0bf33aed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b93c9eafbc61</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Shabeel Kandi]]></dc:creator>
            <pubDate>Wed, 15 Jul 2020 23:08:04 GMT</pubDate>
            <atom:updated>2023-04-24T18:34:41.040Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8P307ToX4T1fvVujMbxRvA.png" /><figcaption>A/B Test Diagram from <a href="https://www.vippng.com/maxp/hxxximo/">https://www.vippng.com/maxp/hxxximo/</a></figcaption></figure><p>In this article, I will take you through analysing a hypothetical A/B Test conducted by a mobile game developer.</p><h4><strong>Game Overview</strong></h4><ul><li>The game is called Math Puzzle Game. Like other puzzles, players need to beat levels to progress.</li><li>The game is Free to Play meaning that players can download and play the game for free but can optionally buy hints on the levels.</li></ul><h4><strong>Description of Experiment</strong></h4><ul><li>An experiment (A/B Test) was run on the game, offering two different game experiences that we call A and B, group A being the control group where the experience is kept as is, and group B being the experiment group that is exposed to the new experience.</li><li>We set the assignment process to randomly distribute players among the groups: 80% to group A (control) and 20% to group B (test).</li><li>The experiment ran from <em>2017–05–04</em> to <em>2017–05–22.</em></li></ul><h4><strong>Experiment Hypothesis</strong></h4><ul><li>The hypothesis was that the new change to the game would improve the user experience making Math Puzzle Game a better game.</li></ul><h3>Data</h3><p>We have the 2 tables</p><ol><li>Assignment table:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*E50cdb0A8jq-sHq5J39kkQ.png" /></figure><p>2. Activity table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*2XmdyL7v7GGr5sZTNIPjrw.png" /></figure><h3>Analysis</h3><p>The unit of diversion is a player_id. We assume that each player_id is independent. We assume α=0.05 (significance level ).</p><h4>1. Choosing analysis metrics</h4><p>We use two types of metrics: Invariate and evaluation metrics.</p><ul><li>Invariate metrics are used for “sanity checks”, to be confident in the validity of the test, as these metrics are not to be affected because of the experiment by making sure these metrics don’t change drastically between control and experiment groups.</li><li>Evaluation metrics are metrics used where a change is expected that are relevant to the business goal we aim to achieve.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*L2kbMzkW1fVQ46F1hHF8eA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*X2X3MF-YlY1NoRPeiJvK5A.png" /></figure><h4>2. Make sure the A/B test was run properly</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*fzG_R4pQfvyg8wS93yQwXA.png" /></figure><p>This looks similar to the original design of 80–20 split.To make sure difference in player distribution is not significant and is random and even like we expected. We can model this diversion in the following way:<br>We expect the number of players in the control group to be about a 20% of the total number of players.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLwfVqw3vbaYRF-rEZQGcQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*R9IZ3zhhO3pzefxP1-dp0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*YMElkdcD1Yn37-VuR5S9mw.png" /></figure><p>From the above charts we see that most of the players were assigned on the first assignment day, and then the remaining were added subsequently on each assignment day. But the player distribution follows the 80–20 split into groups A and B based on the above chart. This is consistent with our experiment design, thus concluding our sanity checks</p><h4>3. Effect on evaluation metrics</h4><p>For each evaluation metric, the difference between the values from both groups is measured. Then, we compute the confidence interval for that difference and test whether or not this confidence interval is both statistically significant.</p><p><strong>3.1 Conversion</strong></p><p>Conversion rate tells us the rate at which players convert to paying members. We look at its confidence intervals comparision and cumulative conversion rates. Be careful that some of the players have converted before their assignment date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*etW5A-_ct5lwfnvRLPRnRA.png" /></figure><p>From above stats we can see that the control group and experiment group amount of entrants are about 80% and 20%, respectively. The overall conversion rate of the B group is slightly better than A group (3% relative increase). To check weather this increase is statistically significant we will calculate the confidence intervals of the difference of the conversion rate for the two groups and test whether or not this confidence interval is statistically significant.</p><p>We can make the null hypothesis as: there is no differece of the conversion rate between A and B. The test statistic we use is difference between conversion rates of the 2 groups. A ztest will be conducted to calculate the p-value. Let’s check the statistic significance.</p><p><em>Hypothesis:</em></p><ul><li><em>H0: conversion rate is no difference between control group and test group</em></li><li><em>H1: conversion rate in treatment group is different than that in control group</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*PUxNg3zDTwH2h24SBvfTNQ.png" /></figure><p>Our significance level of 0.05. Based on the above p-value test we cannot reject the null hypothesis. Based on the confidence interval of the difference, we again observe that the conversion rate is not significantly different between the two</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*Ru3Oq9ZZmUEsQJ7Rhe-8rw.png" /></figure><p><strong>3.2 Retention</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*fIBD76VU2dYP6lvcahrUIQ.png" /></figure><blockquote>p-value is 0.20587705835186143<br>The change due to the experiment in engagement is -0.02 %<br>Confidence Interval of difference: [ -0.0006000000000000001 , 0.0002 ]<br>The change is statistically significant if the CI doesn&#39;t include 0.</blockquote><p><strong>3.3 Engagement</strong></p><p>The engagement of the playeres are very important to determining the improvement in user experience of the game. We calculate engagement as the Average game rounds played per player</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*V8Y35YuyyBn1wIsOwkbZQA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*SPlWQi3FsLHtY_dG11xCZg.png" /><figcaption>The distributions of gameends per player per day have right skewed distribution with a long tail. However we can use the z-test, since we will be comparing the means of gameends per player per day for each of the two group, and since the the sample is large enough, by Central Limit Theorem, we can assume normal distribution of the means.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ElZ9IKo8ZXB13HAiPG088A.png" /><figcaption>We can see that after the experiment starts on 2017–05–04 , group B has been dropping in avg game rounds per player since the experiment starts, while the control group has stayed stable, which is a indication of the treatment has lowered the engagement.</figcaption></figure><h4>3.4 Average Revenue Per Daily Active User (ARPDAU)</h4><p>The Average Revenue Per Daily Active User of the players is an important metric in determining the commercial success of the game. We calculate engagement as the Average purchases per player</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*pRCqp5NLNG3Oc9DJk_RFtA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*fAIyUVSo-3K8HW1Hzx1gUQ.png" /><figcaption>The distribution has right skewed distribution with a long tail, where the average purchase has a vast majority of people with 0 purchase, indicating a very small percentage of people making a relatively large impact.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lhWnoZuZ0vXi2EI2Ng687A.png" /><figcaption>We initially see an increase in Average Revenue per Daily Active user in the treatment group, but this seems to be fading as we move towards the later days. To understand what could be happening we can try and understand how different users are behaving to the treatment.</figcaption></figure><h3>4. Do different types of players react differently to the treatment?</h3><p>Novelty effect is a common issue during AB test. The most common way to check for novelty effect is by segmenting players in new vs old. New players are defined when assginment date and install date are same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oWw7lWp3sM-L7WKyFwErhQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Qwq1mCL6nAsbsnmKACwdw.png" /></figure><h3>5 Recommendation and Conclusion</h3><p>The AB test for Math Puzzle Game was run successfully according to design specifications.</p><p>From our anlysis of the metrics we can conclude the following:</p><ul><li>The treatment did not show any statistically significant change in conversion.</li><li>The treatment did not show any statistically significant change in retention.</li><li>The treatment has a negative effect on engagement by decreasing the average game rounds played per player.</li><li>The treatment increased the average revenue per daily active user, but this seems to be due to a novely effect, and infact fades over time. Both new players and old players has less engagement towards the treatment. While, old players are more likely to have a spend than new players under the treatment.</li></ul><p>Due to a decrease in engagement and since the increase in Average Revenue per daily active user is caused due to a novelty effect, and finally no significant change in conversions is observed under the treatment, we can conclude that the we should not implement the treatment, as the treatment has negative impact on the user experience based on the metrics.</p><p>The introduced treatment is most likely:</p><ul><li>An increase in difficulty of the levels, due to which number of gamerounds that a player could play reduced, hence decreasing the engagement. The old paying players ended up purchasing hints due to a combination of novelty and due to the increased difficulty of the level.</li></ul><p><strong>Reccomendation</strong> In this AB Test, we observed an interesting phenomena where the old players actaully ended up spending more due to the introduction of a new feature. We can infer from this observation that by periodically releasing new experiences that do not negatively impact engagement, we can increase in-game revenue.</p><p>For the code visit my github page link here</p><p><a href="https://github.com/shabeelkandi/AB_Test_analysis">shabeelkandi/AB_Test_analysis</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b93c9eafbc61" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-analyze-a-b-test-results-b93c9eafbc61">How to Analyze A/B Test Results?</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How do I learn difficult subjects?]]></title>
            <link>https://shabeelkandi.medium.com/how-do-i-learn-difficult-subjects-94a30ded6e96?source=rss-4bb0bf33aed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/94a30ded6e96</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[university]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[Shabeel Kandi]]></dc:creator>
            <pubDate>Thu, 09 Jan 2020 23:37:23 GMT</pubDate>
            <atom:updated>2023-04-24T18:35:04.631Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ndf1ujD_NPzAz40PcDLqA.jpeg" /></figure><blockquote>If you strive to be the best version of yourself, this post is for you.</blockquote><p>There’s a concept known as financial compounding, but most people don’t know about intellectual compounding. Buffett and Munger employed this to great effect and to accumulate mental models such that they can make large decisions quickly. Intuition is simply reading a lot.This allows people to convert typically slow thinking and bad fast thinking (bad intuition) into good intuition. In academic literature this is known as Type 1 and Type 2 thinking.</p><p>One of the common patterns for a self made billionaires is their ability to self study, self reason and accumulate a set of their own mental model. Intellectual capital compounds at a hidden rate and most people use tangible badges and net worth as measures. Intellectual capital is filling out the decision tree and is a forward looking view while net worth is backward looking. Just like how the wrong way to value tech companies is revenue but should rather be by their retention rate and network effects.</p><p>Most people value themselves on their individual badges, which is what society labels for you. But most of what is predictive in life, is how you make decisions. Being effective and investing your time in the right area at the right time is a skill, not purely luck. Allowing readings to fill your mind takes advantage of your diffuse/focused mode of thinking and type 1 and type 2 decision making. Meaning you can focus and use your creative brain (diffuse mode) to wander and make associations. This is the key toward STEM education.</p><p>These ways of thinking and compounding, Buffett, Munger, and polymaths have already used to a large degree and have been confirmed by academia. As the world get faster and faster with AR/VR, AI, crypto. The ability to invest in your own intellectual capital is a crucial prerequisite to maintaining and succeeding in this world and also for your children.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*dbMBnFo04wxROrotBtGm0Q.jpeg" /><figcaption>from dilbert.com</figcaption></figure><p>While politicians say it’s education that’s important, it’s only partially true, it’s the ability to assimilate knowledge trees and compound knowledge that leads to satisfaction, mental stimulation, and long term wealth.What politicians are doing now is simply aiming backwards, but how can you scale this to everyone? It turns out that charter school have been doing a grand experiment.</p><p>Taking children from disadvantaged backgrounds and making them fit for college. This has worked and three of the results are the following.</p><blockquote>“Growth mindset” not “Fixed mindset”<br>“Motivation mindset” not “Fixed mindset”<br>“Student directed AND teacher directed education and projects”</blockquote><ol><li><strong>Growth Mindset</strong></li></ol><p>It turns out statistically those three items are the most predictive. What are they? I’m glad you asked. <strong><em>It turns out that if you just tell students that their mind is like a “muscle” and spend just 10 minutes explaining that concept they will improve their grades dramatically</em></strong><em>. </em>This concept is for a person’s mind and for each skill set. People can have verbal “fixed” mindsets, humor “fixed” mindsets, math “fixed” mindsets. Almost everything. So, you have to consciously unlearn this and apply it consciously even if you know it. Why is this? It turns out people are criticized by society and labeled. So even if you have a “growth” mindset for physical items, you don’t have it for mental items. This applies not only for students but also for adults. You always hear “That’s not me.” It’s a label.</p><p><strong>2. Motivational Mindset</strong></p><p>There’s another concept called “motivational mindset” which teaches the person “what good looks like” which simply teaches the kid to follow “go do the extra problems” “go to office hours” “do more problems” Follow the process of the “motivated student” and it will work. These two concepts turn someone that’s socio economically disadvantaged similar to the education status of someone who grew up upper middle class. This is not a panacea as a lot of people have so much stressors in their lives that they can’t study.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*L5OxzH755UUNtnx2FqRn0w.jpeg" /></figure><p><strong>3. Student directed AND teacher directed education and projects</strong></p><p>The last concept is student self directed project plus teacher lectures is the best. This surprised me, but conceptually, it gives the student agency and motivation. Most students have been so battered down by the system that they can’t do this, but that’s another story. These concepts have to be mind beliefs. Just like in Dune how they cite “fear is the mindkiller” These concepts have to be constantly applied to adults and children as they opposite tends to be pervasive and insidious.</p><p>You thought just because you own a brain you knew how to operate it right? Why do you think the drop out rates for STEM is so high. Most people attribute it to pipeline or professor,<strong> but perhaps it’s because people don’t know how to learn difficult subjects.</strong></p><p>Then you have Cal Newport’s books (with links to amazon, if you want to check them out), and these three resources are simply the best that I know of to hack your own brain.</p><ol><li><a href="https://amzn.to/39RG8Iy">Deep Work: Rules for Focused Success in a Distracted World</a></li><li><a href="https://amzn.to/2sUHj9E">Digital Minimalism: Choosing a Focused Life in a Noisy World</a></li><li><a href="https://amzn.to/2tI9J6E">So Good They Can’t Ignore You: Why Skills Trump Passion in the Quest for Work You Love</a></li></ol><p>Do you know anymore? Would love to know more resources and techniques.</p><p>These are mindsets and as for strategies. People need to take concepts from <a href="https://amzn.to/2T8myBZ">Art Of Learning by Josh Waitzki</a> and Newport as they are are a manual for your brain.</p><p>In conclusion, as the world become more and more technical and complex, most people don’t have the mindset nor tactical skills. In short, people have to re-learn the manual to their own brain. Just because you have a computer, it doesn’t mean you know everything about it. <a href="https://www.nytimes.com/2014/05/18/magazine/who-gets-to-graduate.html">This is a link to school that have tried this</a>.</p><p>References:</p><p>Josh Waitzki</p><p>Shizao Yang</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94a30ded6e96" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prediction Intervals in Forecasting: Quantile Loss Function]]></title>
            <link>https://medium.com/analytics-vidhya/prediction-intervals-in-forecasting-quantile-loss-function-18f72501586f?source=rss-4bb0bf33aed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/18f72501586f</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regression-analysis]]></category>
            <dc:creator><![CDATA[Shabeel Kandi]]></dc:creator>
            <pubDate>Thu, 05 Sep 2019 12:26:53 GMT</pubDate>
            <atom:updated>2023-04-24T18:35:29.924Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X5GFQ8CRyL6BaLrY4VSl3w.png" /></figure><p>In most real world prediction problems, the uncertainty in our predictions provides significant value. Knowing about the range of predictions as opposed to only point estimates can significantly improve decision making processes for many commercial applications.</p><h3>What is a prediction interval?</h3><p>A prediction interval is a quantification of the uncertainty on a prediction. It provides a probabilistic upper and lower bounds on the estimate of an outcome variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*epNbbq2VJbIUn_LB.png" /><figcaption>Relationship between prediction, actual value and prediction interval.<br>Taken from “Machine learning approaches for estimation of prediction interval for the model output”, 2006.</figcaption></figure><p>Although, most model outputs are accurate and close to the observed value, the outputs are themselves random variables, and thus have a distribution. Prediction intervals are necessary to get an idea about the likeliness of the correctness of our results. This likeliness determines an interval of possible values.</p><h3>Quantile Regression Loss function</h3><p>Machine learning models work by minimizing (or maximizing) an objective function. An objective function translates the problem we are trying to solve into a mathematical formula to be minimized by the model. As the name suggests, the quantile regression loss function is applied to predict quantiles. A quantile is the value below which a fraction of observations in a group falls. For example, a prediction for quantile 0.9 should over-predict 90% of the times.</p><blockquote>Given a prediction <em>yi^p</em> and outcome <em>yi</em>, the mean regression loss for a quantile <em>q</em> is</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*fMTBPWvbyk9dG2oh8cnnTA.png" /></figure><p>For a set of predictions, the loss will be its average.</p><p><strong>Intuitive Understanding</strong></p><p>In the regression loss equation above, as <em>q</em> has a value between 0 and 1, the first term will be positive and dominate when over predicting, <em>yip</em> &gt; <em>yi</em>, and the second term will dominate when under-predicting, <em>yip</em> &lt; <em>yi</em>. For <em>q</em> equal to 0.5, under-prediction and over-prediction will be penalized by the same factor, and the median is obtained. The larger the value of <em>q</em>, the more over-predictions are penalized compared to under-predictions. For <em>q</em> equal to 0.75, over-predictions will be penalized by a factor of 0.75, and under-predictions by a factor of 0.25. The model will then try to avoid over-predictions approximately three times as hard as under-predictions, and the 0.75 quantile will be obtained.</p><p><strong>Why use Quantile Loss?</strong></p><p>Prediction interval from least square regression is based on an assumption that residuals (y — y_hat) have constant variance across values of independent variables. We can not trust linear regression models which violate this assumption. We can not also just throw away the idea of fitting linear regression model as baseline by saying that such situations would always be better modeled using non-linear functions or tree based models. This is where quantile loss and quantile regression come to rescue as regression based on quantile loss provides sensible prediction intervals even for residuals with non-constant variance or non-normal distribution.</p><h3>Example: Prediction Intervals for Gradient Boosting Regression</h3><p>This loss function can be used to calculate prediction intervals in neural nets or tree based models.</p><p>We have an example below that shows how quantile regression can be used to create prediction intervals using the scikit-learn implementation of GradientBoostingRegressor. In the following example we forecast for the function f(x)=xcos(x) after adding noise. This example heavily borrows from <a href="http://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_quantile.html">http://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_quantile.html</a> and thus, I only show the implementation of the quantile loss for upper and lower quantiles of y.</p><pre>alpha = 0.95</pre><pre>clf = GradientBoostingRegressor(loss=&#39;quantile&#39;, alpha=alpha,<br>                                n_estimators=250, max_depth=3,<br>                                learning_rate=.1, min_samples_leaf=9,<br>                                min_samples_split=9)</pre><pre>clf.fit(X, y)</pre><pre># Make the prediction on the meshed x-axis<br>y_upper = clf.predict(xx)</pre><pre>clf.set_params(alpha=1.0 - alpha)<br>clf.fit(X, y)</pre><pre># Make the prediction on the meshed x-axis<br>y_lower = clf.predict(xx)</pre><pre>clf.set_params(loss=&#39;ls&#39;)<br>clf.fit(X, y)</pre><pre># Make the prediction on the meshed x-axis<br>y_pred = clf.predict(xx)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y71z4Oe2fjHvFpp1Q6Kzug.png" /><figcaption>Prediction Intervals using Quantile loss (Gradient Boosting Regressor) for f(x)=xcos(x)</figcaption></figure><p>Above figure shows 90% prediction interval calculated using quantile loss where upper bound is constructed at q = 0.95 and lower bound using q = 0.05.</p><p><strong>References</strong></p><ol><li><a href="https://towardsdatascience.com/regression-prediction-intervals-with-xgboost-428e0a018b">https://towardsdatascience.com/regression-prediction-intervals-with-xgboost-428e0a018b</a></li><li><a href="https://heartbeat.fritz.ai/5-regression-loss-functions-all-machine-learners-should-know-4fb140e9d4b0">https://heartbeat.fritz.ai/5-regression-loss-functions-all-machine-learners-should-know-4fb140e9d4b0</a></li><li><a href="https://www.evergreeninnovations.co/blog-quantile-loss-function-for-machine-learning/">https://www.evergreeninnovations.co/blog-quantile-loss-function-for-machine-learning/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18f72501586f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/prediction-intervals-in-forecasting-quantile-loss-function-18f72501586f">Prediction Intervals in Forecasting: Quantile Loss Function</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Scikit-Learn Models using Google ML Engine]]></title>
            <link>https://shabeelkandi.medium.com/deploying-scikit-learn-models-using-google-ml-engine-8dde703ed57?source=rss-4bb0bf33aed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8dde703ed57</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deployment-pipelines]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Shabeel Kandi]]></dc:creator>
            <pubDate>Fri, 22 Mar 2019 17:44:17 GMT</pubDate>
            <atom:updated>2023-04-24T18:36:48.718Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pVn-7T6l1tILDG50" /><figcaption>ML Workflow</figcaption></figure><p>In this tutorial, we will go through the above ML Workflow end-to-end, resulting in a working model deployed on the cloud.</p><h3>1. Code your Model</h3><p>For this tutorial, we will build a model to classify small business stores as either credit-worthy <em>[0] </em>or credit-risky <em>[1]</em>. Thus, a binary classification problem.</p><p>We are not going to focus too much on the model itself, all we have to know is that the each store have the following features: ‘Avg ClosingBalance in Past 30 days’, ‘Avg OutStandingBalance in Past 30 days’,<br> ‘Total Credit NIV in Past 30 days’ and ‘CreditLimit&#39;. And using these features, we train and build a model to classify them.</p><p><strong>Importing libraries</strong></p><pre># [START setup]<br>import datetime<br>import os<br>import subprocess<br>import sys<br>from sklearn.pipeline import Pipeline<br>from sklearn.compose import ColumnTransformer<br>from sklearn.preprocessing import OneHotEncoder<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.model_selection import train_test_split<br>from sklearn.externals import joblib<br>from sklearn.ensemble import RandomForestClassifier<br>#from xgboost import XGBClassifier<br>import pandas as pd<br>import numpy as np</pre><p><strong>Configuring your Google Cloud Set-up</strong></p><pre># Fill in your Cloud Storage bucket name<br>BUCKET_NAME = &#39;test_bucket_name&#39;<br><br>data_filename = &#39;store_data.csv&#39;<br>data_dir = &#39;gs://test_bucket_name/ml-engine-test&#39;</pre><pre># gsutil outputs everything to stderr so we need to divert it to stdout.<br>subprocess.check_call([&#39;gsutil&#39;, &#39;cp&#39;,os.path.join(data_dir,data_filename),data_filename], stderr=sys.stdout)</pre><p><strong>Building Model and exporting model as a <em>Joblib </em>file</strong></p><pre># Load data into pandas, then use `.values` to get NumPy arrays</pre><pre>df=pd.read_csv(data_filename)<br>df=df.replace([np.inf, -np.inf], np.nan)<br>df=df.dropna(axis=&#39;index&#39;)</pre><pre><br>X=df[[&#39;Avg ClosingBalance in Past 30 days&#39;,<br>       &#39;Avg OutStandingBalance in Past 30 days&#39;,<br>       &#39;Total Credit NIV in Past 30 days&#39;, &#39;CreditLimit&#39;]].as_matrix().tolist()</pre><pre>y = df[&#39;wasRestricted&#39;].as_matrix().tolist()</pre><pre>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2, random_state=10)<br>model = RandomForestClassifier(class_weight=&#39;balanced&#39;)<br>model.fit(X_train, y_train)</pre><pre># Export the classifier to a file<br>model_filename = &#39;model.joblib&#39;<br>joblib.dump(model, model_filename)</pre><p><strong>Moving Model to <em>gcs_model_path</em></strong></p><pre>gcs_model_path = os.path.join(data_dir,<br>    datetime.datetime.now().strftime(&#39;store_%Y%m%d_%H%M%S&#39;), model_filename)<br>subprocess.check_call([&#39;gsutil&#39;, &#39;cp&#39;, model_filename, gcs_model_path],<br>    stderr=sys.stdout)</pre><p>Save the above code as store_training.py</p><h3>2. Training Model On Google Cloud</h3><p>Open google cloud shell and from the terminal create a training application package for the model to appear similar to the following:</p><pre>store_sklearn_trainer/<br>    __init__.py<br>    store_training.py</pre><p>The __init__.py file can be created using the following command:</p><p>touch store_sklearn_trainer/__init__.py</p><p><strong>Submit Training Job</strong></p><p>Now we submit the training job. Set the following environment variables for each parameter in your training job request. Replace [VALUES-IN-BRACKETS] with the appropriate values:</p><pre>BUCKET_NAME=[YOUR-BUCKET-NAME]<br>    JOB_NAME=&quot;store_scikit_learn_$(date +&quot;%Y%m%d_%H%M%S&quot;)&quot;<br>    JOB_DIR=gs://$BUCKET_NAME/ml-engine-test/store_learn_job_dir<br>    TRAINING_PACKAGE_PATH=&quot;./store_sklearn_trainer/&quot;<br>    MAIN_TRAINER_MODULE=&quot;store_sklearn_trainer.store_training&quot;<br>    REGION=us-central1<br>    RUNTIME_VERSION=1.13<br>    PYTHON_VERSION=2.7<br>    SCALE_TIER=BASIC</pre><p>Submit the training job request:</p><pre>gcloud ml-engine jobs submit training $JOB_NAME \<br>  --job-dir $JOB_DIR \<br>  --package-path $TRAINING_PACKAGE_PATH \<br>  --module-name $MAIN_TRAINER_MODULE \<br>  --region $REGION \<br>  --runtime-version=$RUNTIME_VERSION \<br>  --python-version=$PYTHON_VERSION \<br>  --scale-tier $SCALE_TIER</pre><p>You should see output similar to the following:</p><pre>Job [store_scikit_learn_[DATE]_[TIME]] submitted successfully.<br>Your job is still active. You may view the status of your job with the command<br><br>  $ gcloud ml-engine jobs describe store_scikit_learn_[DATE]_[TIME]<br><br>or continue streaming the logs with the command<br><br>  $ gcloud ml-engine jobs stream-logs store_scikit_learn_[DATE]_[TIME]<br><br>jobId: store_scikit_learn_[DATE]_[TIME]<br>state: QUEUED</pre><p>We can see the logs for the training using the GCP Console under ML Engine&gt;Jobs</p><p>The model.joblib file can be found in the <strong><em>gcs_model_path </em></strong>specified in the code above.</p><h3>3. Deploy Model</h3><p>Cloud ML Engine organizes your trained models using <em>model</em> and <em>version</em> resources. A Cloud ML Engine model is a container for the versions of your machine learning model. Use the following steps to create model and its version from the ML Engine console itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*9TIz5I8Q-DiVokxZp2DvLg.png" /><figcaption><strong>Create Model</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*drRBcIgkRwf4IgsCndTVeQ.png" /><figcaption><strong>Create Model Version</strong></figcaption></figure><p>When creating the model version, make sure you specify the directory in your GCP Bucket, where your model.joblib file is in, and not the file itself.</p><p>After you have successfully created a model version, Cloud ML Engine starts a new server that is ready to serve prediction requests.</p><p>HOORRAAYY!!</p><h3>4. Sending Prediction Requests</h3><p>Now for the fun part. We can now call on this model from anywhere, using the following method:</p><pre>import googleapiclient.discovery<br>import os</pre><pre>def predict_json(project, model, instances, version=None):<br>    &quot;&quot;&quot;Send json data to a deployed model for prediction.<br>    Args:<br>        project (str): project where the Cloud ML Engine Model is deployed.<br>        model (str): model name.<br>        instances ([[float]]): List of input instances, where each input<br>           instance is a list of floats.<br>        version: str, version of the model to target.<br>    Returns:<br>        Mapping[str: any]: dictionary of prediction results defined by the<br>            model.<br>    &quot;&quot;&quot;<br>    # Create the ML Engine service object.<br>    # To authenticate set the environment variable</pre><pre>service = googleapiclient.discovery.build(&#39;ml&#39;, &#39;v1&#39;)<br>    name = &#39;projects/{}/models/{}&#39;.format(project, model)</pre><pre>if version is not None:<br>        name += &#39;/versions/{}&#39;.format(version)</pre><pre>response = service.projects().predict(<br>        name=name,<br>        body={&#39;instances&#39;: instances}<br>    ).execute()</pre><pre>if &#39;error&#39; in response:<br>        raise RuntimeError(response[&#39;error&#39;])<br>    <br>    if response[&#39;predictions&#39;]==1:<br>        output = &#39;Credit Risky&#39;<br>    else:<br>        output = &#39;Credit Worthy&#39;<br>    return output</pre><p>Make sure that your your google application credentials are set using the following:</p><pre>os.environ[&quot;GOOGLE_APPLICATION_CREDENTIALS&quot;]=&quot;xxxxxxxxx.json&quot;</pre><p>This .json file is the credential file to give authentication to Google Cloud Client library for Python. If you aren’t familiar with it, see <a href="https://cloud.google.com/ml-engine/docs/python-guide"><em>Using the Python Client Library</em></a><em>.</em></p><p>BOOM! Now every time you call the function with input data passed into ‘<em>instances’ </em>you get a prediction whether the store is <strong><em>credit-worthy</em></strong> or <strong><em>credit-risky.</em></strong></p><p><em>This essay is referred from the Google documentation for Cloud ML Engine.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8dde703ed57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handling Out-of-Vocabulary Words in Natural Language Processing based on Context]]></title>
            <link>https://medium.com/analytics-vidhya/handling-out-of-vocabulary-words-in-natural-language-processing-based-on-context-4bbba16214d5?source=rss-4bb0bf33aed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bbba16214d5</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[spacy]]></category>
            <dc:creator><![CDATA[Shabeel Kandi]]></dc:creator>
            <pubDate>Sun, 28 Oct 2018 13:15:29 GMT</pubDate>
            <atom:updated>2023-04-24T18:36:29.180Z</atom:updated>
            <content:encoded><![CDATA[<p>Word Embeddings encode the relationships between words through vector representations of the words. These word vectors are analogous to the meaning of the word. A limitation of word embeddings are that, they are learned by the Natural Language Model (word2vec, GloVe and the like) and therefore words must have been seen in the training data before, in order to have an embedding.</p><p>This articles provides an approach that can be used to handle out-of-vocabulary(OOV) words in natural language processing. Given an OOV word and the sentence it is in, language modelling is used to sequence words in the sentence and predict the meaning of the word by comparison with similar sentences. This is an elegant way of learning word meanings on the fly.</p><h3><strong>Overview of Model</strong></h3><p>The model is built to produce embeddings for Out-of-Vocabulary(OOV) words depending on the OOV word’s context. This is done using language model built using a Bi-directional Recurrent Neural Network with a Long-Short Term Memory cell. This language model is used for predicting the most probable word embedding for the OOV word based on its context, by predicting words in place of the OOV word and then taking a weighted average of their mapped word embeddings. This gives a word embedding for the OOV word which is reliable in terms of usability for entity recognition tasks and a meaningful representation for it in the vector space.</p><p>The Model for predicting OOV word embeddings consist of two segments: The first step is where the Model is prepared by tokenising, training and saving a model based on a training corpus, and the second step, which consists of the Prepared model being used to predict embeddings. The first step is called the Preparation Step and the second step is called the Embedding Prediction step.</p><h3><strong>Pseudo-Code of the Model</strong></h3><h4><strong>Model Preparation</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xGyYJdWbmRTG5Y18fJ8P-g.png" /><figcaption>Model Preparation Step</figcaption></figure><p>In Step 1 of the model preparation, shown above, a large corpus is pre-processed to be used for Step 2, where this corpus is tokenised, and the text is encoded as integers.</p><p>The tokenizer in Step 2 is used to fit the source text to develop the mapping from words to unique integers. Then these integers can be used to sequence lines of text. The size of the vocabulary of the text corpus is retrieved using the tokenizer to define the word embedding layer of the model inn Step 3.1.1. In Step 2.4, we prepare forward and backward sequences of the text data, where both the directions of the sequences are used for predicting embedding based on the previous and later words from the OOV word. The sequences are then split into input (X) and output elements (y) for each forward and reverse sequences to be used for training them on the RNN LSTM model prediction model. The forward and backward sequences are padded to have all sequences of the same length.</p><p>In Step 3, we define a model for each forward and backward sequences with 3 layers. The first embedding layer creates a real valued vector for each sequence. The second layer is the Bi-directional LSTM layer with a set number of units, which can be pruned to best fit the training corpus. The output layer is a dense layer comprised of one neuron for each word in the vocabulary. This layer uses a SoftMax function to ensure that the output is normalised to return a probability.</p><p>In Step 4, the encoded text for forward and backward sequences are compiled fit on respective RNNs. Since the Network is technically used to predict the probability distribution of the vocabulary based on the sequence provided, we use a sparse categorical cross entropy loss function to update weights on the network. The Adam optimiser is an efficient implementation of the gradient decent algorithm that is used to track the accuracy of each epoch of the training. The models is then saved to be used to return probability distributions of the vocabulary depending on forward or backward sequenced models.</p><h4>OOV Embedding Prediction Step</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2XPDV_kU64cevY53sAnAGg.png" /><figcaption>Predict Word Embedding Step</figcaption></figure><p>The OOV Word Embedding Prediction step is shorter than the Model preparation step. Step 1, consists of loading all the models and parameters required to run the Embedding Prediction functions. In Step 2, the Generate Sequence function is used as a function that is called by Step 4’s Set Embedding function to be able to predict the most probable words that occur in place of the OOV word in the Sample text. These predictions are used to map to GloVe vectors of the predicted words and the Step 4 function defines a weighted average of the predicted words’ embeddings. This Embedding is assigned to the vocabulary of our pre-trained model. This method of assigning embeddings is used so that, the OOV words will have a reasonable position in the vector space based on its context, even though it was initially not assigned an embedding.</p><h3>Results and Analysis</h3><p>We can analyse our results based on the cosine similarity of the word embedding of the word before and after being processed my the model.</p><p>For example, given an input sentence:</p><p><em>‘I play fidditch at school’</em></p><p>Now, just by looking at this sentence we can see that <em>fidditch </em>is probably a game. Let’s see how the model performs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*xl2xH7IH8Y17x9XQuJ_Dcw.png" /><figcaption>Original embedding of OOV word — fidditch</figcaption></figure><p>Doing a quick lookup to check if the word exists in the vocabulary, we see that the word has no representation (obviously, since Ijust made that word up). After passing it through the OOV model, we get the following word embedding:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*iOogqQubmIOnTBZixC5vLg.png" /><figcaption>300-dimensional word embedding of OOV word — fidditch</figcaption></figure><p>We obtain the most similar words (using cosine similarity) to <em>fidditch</em> based on the new embeddings produced by our model and get the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*Wu2bz6Y6erac7zCD_32Qxg.png" /><figcaption>Most similar words to produced embedding of OOV word ‘fidditch’</figcaption></figure><p>Wow! The model recognizes that the word is probably a sport. Thus, the model detected the unknown word, and then compared the sequence with similar sentences in its training set and then assigned a meaning to the word. Quite impressive!</p><h3><strong>Further Reading and Code</strong></h3><p>I have given a brief overview of the model in the article above. The full code along with the full documentation of the work can be found on my GitHub page below:</p><p><a href="https://github.com/shabeelkandi/Handling-Out-of-Vocabulary-Words-in-Natural-Language-Processing-using-Language-Modelling.git">https://github.com/shabeelkandi/Handling-Out-of-Vocabulary-Words-in-Natural-Language-Processing-using-Language-Modelling</a></p><p>or on ResearchGate:</p><p><a href="https://www.researchgate.net/publication/335757797_Language_Modelling_for_Handling_Out-of-Vocabulary_Words_in_Natural_Language_Processing?showFulltext=1&amp;linkId=5d7a26a04585151ee4afb0c5">https://www.researchgate.net/publication/335757797_Language_Modelling_for_Handling_Out-of-Vocabulary_Words_in_Natural_Language_Processing?showFulltext=1&amp;linkId=5d7a26a04585151ee4afb0c5</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bbba16214d5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/handling-out-of-vocabulary-words-in-natural-language-processing-based-on-context-4bbba16214d5">Handling Out-of-Vocabulary Words in Natural Language Processing based on Context</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>