<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abid Ali Awan on Medium]]></title>
        <description><![CDATA[Stories by Abid Ali Awan on Medium]]></description>
        <link>https://medium.com/@abidaliawan?source=rss-ebd29761458a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DdQTuZaZoyLFi6e_23FxCw.jpeg</url>
            <title>Stories by Abid Ali Awan on Medium</title>
            <link>https://medium.com/@abidaliawan?source=rss-ebd29761458a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:26:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abidaliawan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Evolution of Tabular Data: From Analysis to AI]]></title>
            <link>https://pub.towardsai.net/the-evolution-of-tabular-data-from-analysis-to-ai-8f016f448acc?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f016f448acc</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[tabular-data]]></category>
            <category><![CDATA[kaggle-competition]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Fri, 11 Aug 2023 23:27:41 GMT</pubDate>
            <atom:updated>2023-08-15T13:30:05.646Z</atom:updated>
            <content:encoded><![CDATA[<h4>Discover how tabular data space is being transformed by Kaggle competitions, the open-source community, and Generative AI.</h4><figure><img alt="The Evolution of Tabular Data: From Analysis to AI cover photo." src="https://cdn-images-1.medium.com/max/1000/1*By8082KmC4YGr0ZkEuvxaA.png" /><figcaption>Image by Author</figcaption></figure><h3>Introduction</h3><p>Tabular data refers to data organized into rows and columns. It encompasses everything from CSV files and spreadsheets to relational databases. Tabular data has been around for decades and is one of the most common data types used in data analysis and machine learning.</p><p>Traditionally, tabular data has been used for simply organizing and reporting information. However, over the past decade, its usage has evolved significantly due to several key factors:</p><ul><li><strong>Kaggle Competitions</strong>: Kaggle emerged in 2010 [1] and popularized data science and machine learning competitions using real-world tabular datasets. This exposed many data scientists and machine learning engineers to the power of analyzing and building models on tabular data.</li><li><strong>Open-source Contributions</strong>: Thanks to major open-source libraries like Pandas, DuckDB, SDV, and Scikit-learn, manipulating, preprocessing, and building predictive models on tabular data is now incredibly easy. Additionally, open-source datasets provide beginners with easy access to practice on real-world datasets.</li><li><strong>Generative AI</strong>: Recent advances in generative AI, especially large language models, now enable the generation of realistic tabular data and make it easy for basically anyone to conduct Data analysis and build machine learning applications.</li></ul><p>In the essay, we will discuss each of these factors in more detail and look at examples of how companies and researchers are using tabular data in innovative ways today. The main takeaway will be the importance of analyzing and preparing tabular data in the right way to reap the benefits of machine learning and AI.</p><blockquote>This essay is a part of the <a href="https://www.kaggle.com/competitions/2023-kaggle-ai-report">2023 Kaggle AI Report</a>, a competition where participants write an essay on one of seven topics. The prompt asks them to describe what the community has learned over the past two years of working and experimenting.</blockquote><h3>Tabular Data Kaggle Competitions</h3><p>The Kaggle Competition has profoundly impacted the field of data science and machine learning engineering. Additionally, Tabular competitions have introduced new techniques, tools, and various tabular tasks.</p><p>In addition to learning and knowledge development, winning competitions often come with cash prizes, providing further motivation for participation. For example:</p><ul><li>On average, Kaggle competitions offer prize money of around $21,246 and have approximately 1,498 participating teams.</li><li>The largest cash prizes have been as high as $125,000, giving the winners a significant incentive to go the extra mile and push the boundaries of what’s possible with tabular data.</li></ul><blockquote><strong><em>Note: </em></strong><em>We will be using </em><a href="https://www.kaggle.com/datasets/kaggle/meta-kaggle"><em>Meta Kaggle</em></a><em> dataset for our analysis and code examples. The dataset is under </em><a href="https://www.apache.org/licenses/LICENSE-2.0"><em>Apache 2.0</em></a><em>, and it is updated daily.</em></blockquote><pre>import pandas as pd<br><br>comptags = pd.read_csv(&quot;/kaggle/input/meta-kaggle/CompetitionTags.csv&quot;)<br>tags = pd.read_csv(&quot;/kaggle/input/meta-kaggle/Tags.csv&quot;)<br>comps = pd.read_csv(&quot;/kaggle/input/meta-kaggle/Competitions.csv&quot;)<br><br><br>tabular_competition_ids = comptags.query(&quot;TagId == 14101&quot;)[&#39;CompetitionId&#39;]<br>tabular_competitions = comps.set_index(&#39;Id&#39;).loc[tabular_competition_ids]<br>tabular_competitions.describe()[[&quot;RewardQuantity&quot;,&quot;TotalTeams&quot;]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*wW_AX257vQ_3Idp1LiNSHQ.png" /><figcaption>Code output</figcaption></figure><p>Over the last decade, Kaggle has hosted numerous competitions centered around tabular data, with several since 2015 offering cash prizes of up to $100,000 for the winning team.</p><pre>import plotly.express as px<br><br>tabular_competitions[&quot;EnabledDate&quot;] = pd.to_datetime(<br>    tabular_competitions[&quot;EnabledDate&quot;], format=&quot;%m/%d/%Y %H:%M:%S&quot;<br>)<br>tabular_competitions[&quot;EnabledDate&quot;] = tabular_competitions[&quot;EnabledDate&quot;].dt.year<br>tabular_competitions.sort_values(by=&quot;EnabledDate&quot;, inplace=True)<br><br>fig = px.bar(<br>    tabular_competitions,<br>    x=&quot;EnabledDate&quot;,<br>    y=&quot;RewardQuantity&quot;,<br>    title=&quot;Reward Distribution of Tabular Competitions over the Years&quot;,<br>    labels={&quot;RewardQuantity&quot;: &quot;Prize Money($)&quot;, &quot;EnabledDate&quot;: &quot;Year&quot;},<br>)<br><br>fig.show()</pre><figure><img alt="Reward Distribution of Tabular Competitions over the Years" src="https://cdn-images-1.medium.com/max/630/1*j5VOwPOTqgXcL5G30L-HqA.png" /><figcaption>Reward Distribution of Tabular Competitions over the Years</figcaption></figure><p>The number of tabular data competitions has grown significantly over this period, with particularly high activity in 2015 and 2022.</p><pre>fig = px.histogram(<br>    tabular_competitions,<br>    x=&quot;EnabledDate&quot;,<br>    nbins=20,<br>    title=&quot;Number of Tabular Competitions over the Years&quot;,<br>    labels={&quot;EnabledDate&quot;: &quot;Year&quot;},<br>)<br>fig.show()</pre><figure><img alt="Number of Tabular Competitions over the Years" src="https://cdn-images-1.medium.com/max/630/1*72L2bkkCNK-DFnnh2lHtyQ.png" /><figcaption>Number of Tabular Competitions over the Years</figcaption></figure><h3>Tabular Playground Series</h3><p>Due to the large demand for tabular data problems, the Kaggle staff started an experiment in 2021 [2] by launching a monthly contest called the Tabular Playground Series. These competitions aimed to provide a consistent platform for competitors to hone their skills on tabular data.</p><p>The Tabular Playground Series contests were based on synthetic datasets that replicated the structure of public data or data from previous Kaggle competitions. The synthetic datasets were created using a deep-learning generative network called CTGAN.[3]</p><ul><li><strong>Exposure:</strong> Many machine learning practitioners got their first exposure to working with tabular data through Tabular Playground Series. This helped familiarize them with concepts like data loading, feature engineering, and model tuning.</li><li><strong>Techniques:</strong> Kaggle competitions showcased techniques like feature engineering, data augmentation, and ensemble modeling that are particularly useful for tabular data. Competitors used these techniques to achieve higher scores, setting examples for others.</li><li><strong>Community:</strong> The discussions within Kaggle competitions provided a fertile ground for sharing techniques and ideas on how to best handle tabular data. This helped form a community of practice around tabular data.</li><li><strong>Democratization:</strong> Kaggle competitions have made machine learning on tabular data more accessible to a wider audience, not just data experts. Participants get free access to both CPU and GPU, as well as to large datasets, and anyone is welcome to participate in the competition.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tbeNc-QqHQRt0y9X.png" /><figcaption>Image from <a href="https://www.kaggle.com/competitions/playground-series-s3e18">playground-series-s3e18</a></figcaption></figure><p>The Tabular Playground Series is still ongoing, currently in Season 3 with Episode 18. This demonstrates that cash prizes are not the only motivation for participants, as these competitions do not offer any monetary prizes or points systems. Rather, the series caters to data enthusiasts who want to hone their skills by practicing various types of tabular data.</p><h3>Competition Solutions</h3><p>Examination of winning solutions has revealed that fancy tools or deep learning models are not necessary to place highly. Even simpler models like linear regression, with careful feature engineering, can win prizes. The key is finding simple yet effective techniques for solving the given problem.</p><p>For example, the winner [4] of the GoDaddy — Microbusiness Density Forecasting competition [5] used Linear Regression. This is unsurprising as winning solutions are often based on simple models but involve extensive feature selection, cross-validation, data augmentation, and ensemble techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JqqDMGmtIwR8HV30.png" /><figcaption>Image from <a href="https://farid.one/kaggle-solutions/">Farid</a></figcaption></figure><h3>Tabular Data Open-source Contributions</h3><p>Open-source contributions related to tabular data have been invaluable for advancing the field and enabling real-world applications. Contributions fall into two main categories:</p><ol><li>Open-source Datasets</li><li>Open-source Tools</li></ol><h3>Open-source Datasets</h3><p>Kaggle owes its success to the generous contributions of open-source contributors who share real-world tabular datasets for machine learning problems. These datasets, covering various domains and use cases, provide valuable training and benchmarking data for the machine learning community. Numerous companies and organizations have openly contributed their proprietary tabular data to advance the field. The remarkable number and diversity of datasets available on Kaggle have been an essential driver for innovation in working with tabular data.</p><p>Kaggle dataset [6] is the go-to place for beginners and experts alike who are looking for specific datasets. Its vast collection of tabular datasets is helping hundreds of community members daily to practice new techniques and handle new types of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6_Ak1q6nGnZFvWtS.png" /><figcaption>Image from <a href="https://www.kaggle.com/datasets">Kaggle</a></figcaption></figure><h3>Open-Source Tools</h3><p>Several major open-source tools for analyzing, manipulating and modeling tabular data have been made possible by the contributions of developer communities. Tools like Pandas, Numpy, scikit-learn, TensorFlow, XGBoost, and many others have been crucial enablers for working with tabular data at scale. These libraries provide a comprehensive set of functionalities that have made tabular data machine learning accessible to a wide audience. Ongoing community contributions ensure the tools continue to improve and keep pace with new requirements.</p><p>Additionally, there are now efficient tools available such as DuckDB and PySpark, that offer a user-friendly yet powerful way to analyze and process large tabular datasets.</p><pre>%pip install duckdb -q</pre><p>With DuckDB, you can easily import a CSV file and run SQL queries in just seconds.</p><pre>import duckdb<br><br>duckdb.sql(&#39;SELECT * FROM &quot;/kaggle/input/meta-kaggle/Competitions.csv&quot; LIMIT 5&#39;)</pre><pre>┌───────┬────────────────┬──────────────────────┬───┬──────────────────────┬──────────┬───────────────────┐<br>│  Id   │      Slug      │        Title         │ … │ EnableSubmissionMo…  │ HostName │ CompetitionTypeId │<br>│ int64 │    varchar     │       varchar        │   │       boolean        │ varchar  │       int64       │<br>├───────┼────────────────┼──────────────────────┼───┼──────────────────────┼──────────┼───────────────────┤<br>│  2408 │ Eurovision2010 │ Forecast Eurovisio…  │ … │ false                │ NULL     │                 1 │<br>│  2435 │ hivprogression │ Predict HIV Progre…  │ … │ false                │ NULL     │                 1 │<br>│  2438 │ worldcup2010   │ World Cup 2010 - T…  │ … │ false                │ NULL     │                 1 │<br>│  2439 │ informs2010    │ INFORMS Data Minin…  │ … │ false                │ NULL     │                 1 │<br>│  2442 │ worldcupconf   │ World Cup 2010 - C…  │ … │ false                │ NULL     │                 1 │<br>├───────┴────────────────┴──────────────────────┴───┴──────────────────────┴──────────┴───────────────────┤<br>│ 5 rows                                                                             42 columns (6 shown) │<br>└─────────────────────────────────────────────────────────────────────────────────────────────────────────┘</pre><p>Perform quick and multiple actions on tabular data using Python Relational API. Its syntax is similar to pandas, making it easy to use.</p><pre>rel = duckdb.read_csv(&#39;/kaggle/input/meta-kaggle/Competitions.csv&#39;)<br>rel.filter(&quot;RewardQuantity &gt; 100000&quot;).project(<br>    &quot;EnabledDate,RewardQuantity&quot;<br>).order(&quot;RewardQuantity&quot;).limit(5)</pre><pre>┌─────────────────────┬────────────────┐<br>│     EnabledDate     │ RewardQuantity │<br>│       varchar       │     double     │<br>├─────────────────────┼────────────────┤<br>│ 07/25/2019 21:10:14 │       120000.0 │<br>│ 11/02/2021 16:00:27 │       125000.0 │<br>│ 11/14/2016 08:02:32 │       150000.0 │<br>│ 11/22/2021 18:53:57 │       150000.0 │<br>│ 05/11/2022 18:46:43 │       150000.0 │<br>└─────────────────────┴────────────────┘</pre><h3>Generative AI for Tabular Data</h3><p>Generative AI is a subfield of artificial intelligence powered by neural networks like Variational Autoencoders and Generative Adversarial Networks (GANs) that can generate photorealistic images, compose original pieces of music, write news articles and stories, and even design objects. It is trained on large datasets, allowing Generative AI models to discover the underlying patterns, structures, and statistical distributions present in the data.</p><p>Generative AI models have significantly progressed the field of working with tabular data. Capabilities like data augmentation, anomaly detection and synthetic data generation have helped tackle issues like data scarcity, privacy and bias.</p><p>However, recent advances like ChatGPT and other large language models (LLMs) are now also being used as assistants for tabular data tasks. Some of the ways generative AI is transforming our workflows include:</p><ul><li><strong>Code assistance:</strong> LLMs like ChatGPT can help with coding tasks like feature engineering, preprocessing, modeling and evaluating machine learning pipelines for tabular data. They can suggest code snippets, functions and entire scripts.</li><li><strong>Data understanding:</strong> Generative AI can provide insights into the data distributions, correlations, missing values, outliers, target variables and more.</li><li><strong>Deep analysis:</strong> It performs statistical tests, creates visualizations and derives other summary metrics that give practitioners a thorough analysis of tabular datasets to inform modeling decisions.</li><li><strong>Web scraping:</strong> Generative AI tools can help you scrape new tabular data from websites/application, assisting with data acquisition tasks.</li></ul><p>While issues like safety, bias, and narrow capability remain, large language models are beginning to transform how data scientists and machine learning engineers work with tabular data on a day-to-day basis. They are increasingly becoming assistants that handle various analytical and coding tasks, freeing up practitioners to focus on higher-level work.</p><h3>ChatGPT for Tabular Data</h3><p>ChatGPT [7] has rapidly become an invaluable assistant for nearly every stage of working with tabular data, from helping with data cleaning and feature engineering to generating complex model code, interpreting metrics, producing data analysis reports, and even aiding in synthetic data generation for tasks like data augmentation and anomaly detection.</p><p>With ChatGPT, you can easily build and train a machine-learning model by simply typing a detailed prompt. Additionally, you can utilize multiple plugins to automate complex tasks such as running the code and getting internet access.</p><p>Check out “A Guide to Using ChatGPT For Data Science Projects”[8] to learn how to use ChatGPT in a real-life end-to-end data science project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/0*wrtH2QDI4aRvBoJJ.png" /><figcaption>Image by Author</figcaption></figure><h3>Generative AI Tools for Tabular Data</h3><p>Generative AI tools, such as PandasAI [9], have made data analysis, dataset cleaning, and data visualization extremely easy for anyone. These tools use large language models like gpt-3.5-turbo [10] to generate insightful results. Additionally, you can also connect with open-source models hosted on Hugging Face to perform AI analysis.</p><pre>%pip install pandasai -q</pre><pre>from kaggle_secrets import UserSecretsClient<br>from pandasai import PandasAI<br>from pandasai.llm.openai import OpenAI<br><br>user_secrets = UserSecretsClient()<br>secret_value_0 = user_secrets.get_secret(&quot;OPENAI_API_KEY&quot;)<br><br>llm = OpenAI(api_token=secret_value_0)<br><br>pandas_ai = PandasAI(llm)</pre><p>To see the top five competitions with the highest RewardQuantity, we asked ChatGPT to display them by typing a prompt.</p><pre>pandas_ai.run(tabular_competitions, prompt=&#39;Can you provide a list of the top five competitions with the highest RewardQuantity? Please only show the name of the competition,Date,and the corresponding reward.&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*C-dVgFjBWdCnIDokWWWhXw.png" /><figcaption>Code output</figcaption></figure><p>You can even ask it to perform complex tasks or generate visualizations.</p><pre>pandas_ai.run(tabular_competitions, prompt=&#39;Please list all the competition which has &quot;Market&quot; in it.&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*Q5P194PlqDymL-bUmycwhw.png" /><figcaption>Code output</figcaption></figure><p>This is just the beginning, as we will see many new AI tools that make data scientists&#39; and developers lives easier by automating tasks and providing assistance.</p><h3>Conclusion</h3><p>While significant progress has been made in leveraging tabular data for machine learning and AI applications, we have likely only seen the beginning. In the future, we can expect new powerful tools driven by advanced AI agents [11] that will automate the entire workflow for tabular machine learning tasks — from data ingestion and cleaning to feature engineering, model training, evaluation, and deployment in web applications. With continued advances in generative AI and natural language processing, these agents will be able to take high-level prompts to complete entire tabular data science projects, from data to insights.</p><p>This essay highlights the significant impact of Kaggle competitions, open-source communities, and generative AI on our approach to working with tabular data for tasks such as data analysis and machine learning. To delve deeper into the topic, you can read the winning essays from the 2023 Kaggle AI Report competition.[12]</p><h3>Reference</h3><p>[1] Wikipedia contributors, “Kaggle,” Wikipedia, Jun. 2023, [Online]. Available: <a href="https://en.wikipedia.org/wiki/Kaggle">https://en.wikipedia.org/wiki/Kaggle</a></p><p>[2] “Tabular Playground Series — Jan 2021 | Kaggle.” <a href="https://www.kaggle.com/competitions/tabular-playground-series-jan-2021">https://www.kaggle.com/competitions/tabular-playground-series-jan-2021</a></p><p>[3] Sdv-Dev, “GitHub — sdv-dev/CTGAN: Conditional GAN for generating synthetic tabular data.,” GitHub. <a href="https://github.com/sdv-dev/CTGAN">https://github.com/sdv-dev/CTGAN</a></p><p>[4] KAGGLEQRDL, “#1 solution — generalization with linear regression,” Mar. 16, 2023. <a href="https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/395131">https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/395131</a></p><p>[5] “GoDaddy — Microbusiness Density Forecasting | Kaggle,” Dec. 15, 2022. <a href="https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/overview">https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/overview</a></p><p>[6] “Find Open Datasets and Machine Learning Projects | Kaggle.” <a href="https://www.kaggle.com/datasets">https://www.kaggle.com/datasets</a></p><p>[7] “Introducing ChatGPT,” OpenAI, Nov. 30, 2022. <a href="https://openai.com/blog/chatgpt">https://openai.com/blog/chatgpt</a></p><p>[8] A. A. Awan, “A Guide to Using ChatGPT For Data Science Projects,” Mar. 2023, [Online]. Available: <a href="https://www.datacamp.com/tutorial/chatgpt-data-science-projects">https://www.datacamp.com/tutorial/chatgpt-data-science-projects</a></p><p>[9] Gventuri, “GitHub — gventuri/pandas-ai: Pandas AI is a Python library that integrates generative artificial intelligence capabilities into Pandas, making dataframes conversational,” GitHub. <a href="https://github.com/gventuri/pandas-ai">https://github.com/gventuri/pandas-ai</a></p><p>[10] “GPT-3.5,” OpenAI. <a href="https://platform.openai.com/docs/models/gpt-3-5">https://platform.openai.com/docs/models/gpt-3-5</a></p><p>[11] R. Cotton, “Introduction to AI Agents: Getting Started With Auto-GPT, AgentGPT, and BabyAGI,” May 2023, [Online]. Available: <a href="https://www.datacamp.com/tutorial/introduction-to-ai-agents-autogpt-agentgpt-babyagi">https://www.datacamp.com/tutorial/introduction-to-ai-agents-autogpt-agentgpt-babyagi</a></p><p>[12] “2023 Kaggle AI Report.” May 2023, [Online]. Available: <a href="https://www.kaggle.com/competitions/2023-kaggle-ai-report/leaderboard">https://www.kaggle.com/competitions/2023-kaggle-ai-report/leaderboard</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f016f448acc" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/the-evolution-of-tabular-data-from-analysis-to-ai-8f016f448acc">The Evolution of Tabular Data: From Analysis to AI</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build Strong Data Science Portfolio as a Beginner]]></title>
            <link>https://medium.com/geekculture/how-to-build-strong-data-science-portfolio-as-a-beginner-5540dd84aff1?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5540dd84aff1</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[portfolio]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Mon, 18 Jul 2022 07:01:33 GMT</pubDate>
            <atom:updated>2022-07-18T07:01:33.215Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Build the Strong Data Science Portfolio as a Beginner</h3><p>Learn a unique way to build a data science portfolio that gets you hired.</p><figure><img alt="How to Build the Strong Data Science Portfolio as a Beginner" src="https://cdn-images-1.medium.com/max/1024/0*ahWtGckXL-2sL0FA" /><figcaption>Photo by <a href="https://unsplash.com/@halacious?utm_source=medium&amp;utm_medium=referral">Hal Gatewood</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a beginner, I had many questions about how do I start? how do I learn or where do I get ideas to work on projects? So, after a long search, I found a project on data analysis. It took me 3 days just to write code and I was happy with my first try, but then there was this big question of how do I share it with the world? I simply did not have good coding skills or documentation skills to showcase my work, so I stored it in the cloud and forget about it. After a month, I was randomly looking for more projects on GitHub and found this amazing <a href="https://github.com/subhamrex">profile</a> that motivated me to create my portfolio. That was the best decision I made as it put me on the map of the developer community and soon after I started to get emails from the recruiters and beginners about my projects.</p><figure><img alt="Platinum Award — KDnuggets" src="https://cdn-images-1.medium.com/max/120/0*scThhgsFjbZ2PNUO.png" /><figcaption><a href="https://www.kdnuggets.com/2021/11/top-stories-2021-oct.html">Platinum Award — KDnuggets</a></figcaption></figure><p><strong>Getting a job</strong> is usually the main reason for building a portfolio. Sometimes, it’s necessary if we don’t have the relevant education or experience <a href="https://eugeneyan.com/writing/data-science-portfolio-how-why-what/">(eugeneyan.com)</a>. In this modern world, the employers are skeptical about hiring new graduates so how do you convince them that you are best for the job? You display your skills by showing the work you have done in a previous project. The stronger your online portfolio means the higher chance you have of getting hired for your dream job.</p><blockquote>“The portfolios are extremely critical to have because when you’re in the interview it shows your real-world experience, so you can explain to an employer from A to Z the entire data science workflow.” — <a href="https://www.springboard.com/blog/data-science/springboard-mentor-david-yakobovitch/">David Yakobovitch</a>.</blockquote><p>The other motivation is to create your <strong>personal project</strong> that satisfies your curiosity about learning new things. When we learn a new skill, we want to experiment and eventually build a working product that can be used in the real world.</p><p>In this article, we will learn the ways you can showcase your work as a data science beginner. You will learn about some new platform that makes your life easy and tips on building strong portfolios.</p><h3>GitHub</h3><p>Let me just clear the misconception among data scientists, yes <a href="https://github.com/">GitHub</a> is necessary, and we all should learn <a href="https://git-scm.com/">git</a>. As a data scientist, I use Github daily where I look for interesting data sets and projects. This is the most popular platform among developers and to be honest, the recruiter does check your GitHub profile before calling you for interview.</p><figure><img alt="Getting started with Julia Machine Learning Library with FastAI.jl" src="https://cdn-images-1.medium.com/max/1024/1*mWN3wcMrugLveT4wN8C1LA.png" /><figcaption>Image from <a href="https://github.com/kingabzpro/Image-Classification-FastAI.jl">GitHub</a></figcaption></figure><p>GitHub is a global collaborative platform where people share and collaborate on projects. As you can see in my profile below how I have contributed to other people’s projects and also worked on my own projects too.</p><figure><img alt="Abid’s GitHub portfolio" src="https://cdn-images-1.medium.com/max/913/1*Cl9z38ZD8SGsjh9RhrMuiw.png" /><figcaption>Image by Author | <a href="https://github.com/kingabzpro">kingabzpro</a></figcaption></figure><p><strong>Tips to create a solid profile:</strong></p><ol><li>Create your profile page, for a complete tutorial check <a href="https://sarah-hart-landolt.medium.com/6-easy-steps-to-create-a-beautiful-github-profile-readme-edc7840b2c7">Sarah Hart’s</a> blog.</li><li>Document every project with links, cover images, and detailed descriptions.</li><li>Fork the project that you like the most and send your first pull request <a href="https://www.freecodecamp.org/news/how-to-make-your-first-pull-request-on-github-3/">(freecodecamp.org)</a>.</li><li>Be active on this platform by contributing, bug reporting, and pushing your current projects.</li></ol><h3>Deepnote</h3><p><a href="https://deepnote.com/">Deepnote</a> is much simpler the GitHub and its beginner friendly too. If you are familiar with the <a href="https://jupyter.org/">Jupyter notebook</a> then it will be piece of cake for you to publish your first project. My experience with Deepnote is absolutely amazing as the platform provides you all the qualities of GitHub but is much simpler and focused on the data scientist’s community.</p><figure><img alt="Deepnote notebook" src="https://cdn-images-1.medium.com/max/1024/1*Y7rrDtbCcqfKM1EEc1SkpA.png" /><figcaption>Image by Author | <a href="https://deepnote.com/@abid/Pakistan-Vaccination-Progress-sgGprLfPSHuLyI5XTrYiqQ">Pakistan Vaccination Progress</a></figcaption></figure><p>Recently, they have introduced a Deepnote profile which will show case all the notebooks you publish with your information and profile picture.</p><figure><img alt="Abid’s Deepnote profile" src="https://cdn-images-1.medium.com/max/1024/1*0ckPz7KDe_6BXjqW9zq31g.png" /><figcaption>Image by Author | <a href="https://deepnote.com/@abid">Abid Ali Awan</a></figcaption></figure><p>Just like <a href="https://gist.github.com/">GitHub Gist</a>, you can share a snippet of your code with your team or the public in general. I have used Deepnote cell on all the Medium Publication and social media platforms. You can check my previous <a href="https://towardsdatascience.com/deploying-your-first-machine-learning-api-1649236c695e">article</a> to understand how to implement a Deepnote cell. Using snippets of code with output gives you the ability to share your projects on multiple platforms.</p><p>The reason I prefer Deepnote embedded cell over GitHub Gist is that it comes with output, not just static output but with interactive features.</p><blockquote>You can use Plotly and display your chart in a Medium article.</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F9de2b3ca-41e5-405a-ac21-02f994f69997%2Fb0f91434-4c37-44d5-ada3-1ff8c4a35841%2F00037-70a25987-0643-4e49-b645-74500f77990b%3Fheight%3D219&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F9de2b3ca-41e5-405a-ac21-02f994f69997%2Fb0f91434-4c37-44d5-ada3-1ff8c4a35841%2F00037-70a25987-0643-4e49-b645-74500f77990b%3Fheight%3D219&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="219" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c1969593c73f05e49e1346e4abc2e17/href">https://medium.com/media/6c1969593c73f05e49e1346e4abc2e17/href</a></iframe><p><strong>Tips to create a solid profile:</strong></p><ol><li>Update your bio, profile photo, and contact information.</li><li>Always add detailed descriptions about your project by using markdown cell.</li><li>Use the cover photo to make your project stand out.</li><li>Use App features in Deepnote to create Interactive webapp.</li><li>Keep posting your old project or even reposting notebooks from GitHub.</li></ol><h3>DagsHub</h3><p><a href="https://dagshub.com/">DagsHub</a> is new to this world and it’s making its name quickly by providing one stop solution for machine learning practitioners and data engineers. DagsHub comes with a <a href="https://dvc.org/">DVC</a> server, <a href="https://mlflow.org/">MLflow</a>, Visualizing pipeline, and GitHub Synchronization. We won’t be going deep into features rather we will be focusing on the features that make it stands out.</p><p>DagsHub allows you to share your GitHub repository and create your data science project with the ability to visualize machine learning and data pipelines. It also has hidden feature README.ipynb as your project description file, which is best for beginners who are not used to markdown and data scientist who love working on Jupyter Notebook. It is similar to GitHub which means you need to learn both Git and DVC to use this platform properly.</p><blockquote>What I’ve seen other users enjoy is the ability to visualize their project structure, via the pipeline, as well as the ability to see their data and models as an integral part of the project. Also, the fact that we are based on open-source tools instead of reinventing existing solutions is something people like. — <a href="https://www.linkedin.com/in/deanpleban/">Dean</a></blockquote><figure><img alt="DagsHub pipelines" src="https://cdn-images-1.medium.com/max/1024/1*IZn6R3RxiG0srHXLEk_qGw.png" /><figcaption>Image by Dean | <a href="https://dagshub.com/Dean/DHEERAJKU">dagshub</a></figcaption></figure><p>My profile is quite new, but I am loving this platform as they provide me with a complete machine learning ecosystem. I think I prefer it more than GitHub in terms of features and UI simplicity.</p><figure><img alt="Abid Ali Awan’s DagsHub profile" src="https://cdn-images-1.medium.com/max/1024/1*Lx_6E2VWdRMtPUTItxqxUA.png" /><figcaption>Image by Author | <a href="https://dagshub.com/kingabzpro">Abid Ali Awan</a></figcaption></figure><p><strong>Tips to create a solid profile:</strong></p><ol><li>Learn <a href="https://dvc.org/">DVC</a>, <a href="https://git-scm.com/">Git</a>, and <a href="https://mlflow.org/">mlflow</a> to take full advantage.</li><li>Add project description to your notebook and README.</li><li>Update your profile by adding bio, avatar, and contact information.</li><li>Try to add dvc.yaml and dvc.lock in your project to display data pipelines, for more information check <a href="https://dagshub.com/docs/tutorial/pipeline/index.html">Defining the Pipeline</a>.</li><li>Keep an active profile by contributing to open-source projects and by pushing your personal project. You can use <a href="https://github.com/DAGsHub/fds">fds</a> cli to make your life easy and avoid mistakes.</li><li>Takes full use of DVC by uploading your data and model on remote server. Recruiters are interested in candidates that know the complete data science cycle from data ingestion to dashboards.</li></ol><h3>Kaggle</h3><p>If you want to get noticed faster in the world of data science you should create a <a href="https://www.kaggle.com/">Kaggle</a> account and start contributing to <strong>competitions, datasets, notebooks,</strong> and <strong>discussions</strong>. When you become grandmaster people respect you and offer you better career opportunities. If you ask me, I will suggest you create a Kaggle profile while learning basics. Learn from experts and discover your niche. I am a huge fan of this platform as it provides support for a beginner to compete and develop innovative solutions for various industries. It is the backbone of AI research.</p><figure><img alt="Kaggle notebook" src="https://cdn-images-1.medium.com/max/1024/1*PQp6QMGQLTaBKnQynUB6ig.png" /><figcaption>Image by Author | <a href="https://www.kaggle.com/kingabzpro/evolution-of-digital-learning-during-covid19">Kaggle</a></figcaption></figure><p>You can check out my profile below as from the start I have been contributing in various categories to gain ranks. Currently, I am an <strong>Expert </strong>but with one gold and silver medal in the competition, I will become a <strong>Master </strong>which is not easy and honestly, I respect <strong>Grandmasters </strong>as they have proven that they are the best among other data practitioners.</p><figure><img alt="Abid’s Kaggle Profile" src="https://cdn-images-1.medium.com/max/977/1*uqsJhdKT1mqyCY40B8TD8Q.png" /><figcaption>Image by Author | <a href="https://www.kaggle.com/kingabzpro">Kaggle</a></figcaption></figure><p><strong>Tips to create a solid profile:</strong></p><ol><li>Be active on the platform by using new datasets and creating data analysis or machine learning models.</li><li>Participate in discussion, learn from experts, and ask for help.</li><li>Use web scraping to publish a new dataset.</li><li>Participate in most competitions to learn several types of machine learning problems and to earn badges.</li><li>Focus on publishing your best work with detailed descriptions and high-quality code.</li><li>Write about yourself in bio and add contact details.</li></ol><h3>Blog</h3><p>Writing Blogs are the next step after creating your project on the above platforms. If you want to expand your audience, I will highly suggest you start with <a href="https://medium.com/">Medium</a>. Writing a blog is not necessary but you get more traction from various fields. The Medium allows you to create your profile and let you publish your articles under various publications such as <a href="https://towardsdatascience.com/">Towards Data Science</a> and <a href="https://pub.towardsai.net/">Towards AI</a>. You can develop your blogging site or use another similar platform such as <a href="https://www.analyticsvidhya.com/">Analytics Vidhya</a>.</p><figure><img alt="Abid Ali Awan Medium Blog" src="https://cdn-images-1.medium.com/max/1024/1*NXqwbUVyZNL3aWgfHgjHVQ.png" /><figcaption>Image by Author | <a href="https://medium.com/@kingabzpro">Medium</a></figcaption></figure><p><strong>Tips to create a solid profile:</strong></p><ol><li>Write blogs about the project you personally worked on.</li><li>Create blogs on Emerging technology or on new data science applications.</li><li>Do proper research while writing blogs and add citations to avoid platform rules violation.</li><li>Use attractive cover photos for every blog.</li><li>Always write about what you learn from your experience while developing certain data science project.</li><li>Don’t follow the trend, focus on the things you are good at.</li></ol><h3>Portfolio Website</h3><p>You can also display your project on the personal website and if you are not a web developer there are some simple tools available to make the process quite easy. You can check out <a href="https://www.youtube.com/watch?v=mEZ1Hj5yQ-8&amp;t=235s">How to Build a Data Science Portfolio Website with Hugo &amp; GitHub Pages</a> and <a href="https://gohugo.io/">Hugo</a> for various templates.</p><p>My portfolio website has a project from all the platforms with short descriptions and subcategories. It took me three days to create the entire website and deploy it on GitHub pages.</p><figure><img alt="Abid’s Portfolio" src="https://cdn-images-1.medium.com/max/1024/1*iHrMOKqIavNd0dp_eMiAzQ.png" /><figcaption>Image by Author | <a href="https://abid.ninja/">Abid’s Portfolio</a></figcaption></figure><p><strong>Tips to create a solid portfolio website:</strong></p><ol><li>Add your skill, bio, and CV.</li><li>Display your Experience and Awards.</li><li>Showcase your projects with links to your GitHub or Deepnote projects.</li><li>Make your website minimal and interactive so that the recruiter has an easy time scrolling through your entire portfolio.</li><li>Keep your portfolio website up to date with the latest project you are working on.</li></ol><h3>Weight &amp; Biases</h3><p>I usually use <a href="https://wandb.ai/">Weight &amp; Biases</a> for machine learning experimentation and logging performance metrics of my models, but that changed with the introduction of W&amp;B profile. You can write a blog about your current project by using embedded links and graph integration. It is quite similar to other portfolio platforms I mentioned but it comes with the peak of direct integration with python libraries.</p><p><a href="https://wandb.ai/ayush-thakur">Ayush</a> profile has impressed me the most as he has been contributing to other organizations while writing blogs about machine learning.</p><figure><img alt="Ayush’s wandb profile" src="https://cdn-images-1.medium.com/max/1024/1*-TWuQZERutUuLHtKe0ennQ.png" /><figcaption>Image by Ayush | <a href="https://wandb.ai/ayush-thakur">Weights &amp; Biases</a></figcaption></figure><p>The W&amp;B project has model performance metrics as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y9sidmTs2pKUL6_54-KE7A.png" /><figcaption>Image by Author | <a href="https://wandb.ai/ayush-thakur/kaggle-seti-exp2?workspace=user-kingabzpro">kaggle-seti</a></figcaption></figure><p><strong>Tips to create a solid profile:</strong></p><ol><li>Join other data science organizations and participate in group projects.</li><li>Use W&amp;B API to display your machine learning project results.</li><li>Write blog using W&amp;B metrics integration.</li><li>Add a bio, profile picture, contact information.</li><li>Try to engage in community discussion and always look for a new interesting project.</li></ol><h3>Conclusion</h3><p>W&amp;B is a wildcard as it is famous for logging the experiments and not for portfolios but with introduction of interactive blogs has given us the unique advantage of displaying your project and create a strong portfolio. <br>If you are a beginner, I will suggest you start with Deepnote as it’s free for teams and give your beginner-friendly tools to get started. If you are looking to get noticed by the data science community, try creating your profile on GitHub and Kaggle. If you are into creating your brand, then start with blogging sites or create your website. <br>In the end, I want you all to create your profile on all the platforms I mentioned above as they all come with unique advantages in impressing your potential employer. I know it’s quite overwhelming at the start but once you get used to documenting and showcasing your projects it will get easy.</p><figure><a href="https://www.kdnuggets.com/2021/11/top-blogs-rewards-oct.html"><img alt="" src="https://cdn-images-1.medium.com/max/120/0*YD9oK3t4sAM_Ex-t.png" /></a><figcaption><a href="https://www.kdnuggets.com/2021/11/top-blogs-rewards-oct.html">Top Blogs Rewards for October 2021 — KDnuggets</a></figcaption></figure><p>The <a href="https://www.kdnuggets.com/2021/10/strong-data-science-portfolio-as-beginner.html">Original</a> Blog is posted on KDnuggets.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5540dd84aff1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/how-to-build-strong-data-science-portfolio-as-a-beginner-5540dd84aff1">How to Build Strong Data Science Portfolio as a Beginner</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Guide To Data Analysis With Duck DB]]></title>
            <link>https://pub.towardsai.net/the-guide-to-data-analysis-with-duck-db-93a79950a08f?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/93a79950a08f</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Wed, 01 Jun 2022 00:02:34 GMT</pubDate>
            <atom:updated>2022-06-01T00:02:34.609Z</atom:updated>
            <content:encoded><![CDATA[<h4>Duck DB will help you reach the next level in your data analysis career. The guide will help you to understand Python API and various ways to read CSV files with SQL script</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFBJcFL355Fccyb3LaA0Dw.jpeg" /><figcaption>Image by Author</figcaption></figure><p>The life of data analysts revolves around loading data through SQL serve to analyze it using various Python packages and finally creating the technical report. Sometimes these SQL queries can take a longer time to process as your data is in terabytes. In this fast-paced world, this strategy fails miserably and most analytics are moving away from traditional ways of doing data analytics. <a href="https://duckdb.org/">DuckDB</a> solves all the issues mentioned above, it is integrated with Python &amp; R. This Database works similarly to SQLite and focuses on proving faster analytical queries.</p><p>In this guide, we are going to dive deep into learning various ways of running DuckDB SQL queries and also learn about Deepnote integration.</p><h3>DuckDB</h3><p>DuckDB is a relational table-oriented database management system that supports SQL queries for producing analytical results. It also comes with various features that are useful for data analytics.</p><h4>Fast Analytical Queries</h4><p>DuckDB is designed to run faster analytical query workloads. It runs on a columnar-vectorized query execution engine that runs a large batch of processes in one go. This makes it faster to run <a href="https://en.wikipedia.org/wiki/Online_analytical_processing">Online analytical processing (OLAP)</a> as compared to traditional systems such as PostgreSQL which process each row sequentially.</p><h4>Simple Operation</h4><p>DuckDB adopts simplicity and embedded operation.</p><ul><li>Serverless database with no external dependencies</li><li>Doesn’t import or copy data while processing queries</li><li>Embedded within a host process</li><li>high-speed data transfer</li></ul><h4>Feature-Rich</h4><p>DuckDB allows users to run complex SQL queries smoothly. It also supports secondary indexing to provide fast query time within the single-file database. DuckDB provides full integration for Python and R so that the queries could be executed within the same file.</p><h4>Free &amp; Open Source</h4><p>DuckDB is free to use and the entire code is available to the public on <a href="https://github.com/duckdb/duckdb">GitHub</a>. It comes with an <a href="https://opensource.org/licenses/MIT">MIT license</a> which means you can use it for commercial purposes.</p><h3>Creating Database</h3><p>DuckDB provides of box experience for you to connect with a database or create a new one with a similar command duckdb.connect(&lt;db name&gt;,read_only=False) . As you can see in the image below <strong>SampleDB</strong> files have been created in your directory.</p><pre>import duckdb<br>conn = duckdb.connect(“SampleDB”,read_only=False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*OxXl2ueTbWyfJXU4yrXHqg.png" /><figcaption>Image by Author</figcaption></figure><p>OR</p><p>DuckDB also allows you to create an in-memory temporary database by using duckdb.connect(). The conn.execute() run all the query requests in the database. In this example, we are going to create a temporary table called <em>test_table</em> which contains <strong>i</strong> as an integer and <strong>j</strong> as a string. If you are familiar with <a href="https://en.wikipedia.org/wiki/SQL">SQL</a> you won’t have a problem understanding query requests, but if you are new to SQL, I will suggest you look at this amazing <a href="https://learnsql.com/blog/sql-basics-cheat-sheet/">cheat sheet</a>.</p><pre>conn = duckdb.connect()</pre><pre># run arbitrary SQL commands<br>conn.execute(“CREATE TABLE test_table (i INTEGER, j STRING)”)</pre><h4>Inserting values</h4><p>You can insert single or multiple values by using SQL commands. In this part, we are adding two values to <em>test_table</em> using execute.</p><pre>conn.execute(“INSERT INTO test_table VALUES (1, ‘one’),(9,’nine’)”)</pre><p>To check whether we have successfully added values we are going to run SQL to read all values of <strong>i. </strong>To display the result of queries in <a href="https://pandas.pydata.org/">Pandas</a> dataframe we will add .fetchdf() as shown below.<strong> Good Job,</strong> We have successfully added our filet two values.</p><pre>conn.execute(“SELECT i from test_table “).fetchdf()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/141/1*TBnkszt25Cow6USN7JT1rA.png" /></figure><p>We can also use placeholders for parameters and then add an array to the <em>test_table</em>. If you are familiar with the Python framework, you will find this method easy to execute. We can also execute multiple values at once by using .executemany(). As you can see below how we have successfully added values into the <em>test_table</em>.</p><pre>conn.execute(“INSERT INTO test_table VALUES (?, ?)”, [2, ‘two’])</pre><pre>conn.executemany(“INSERT INTO test_table VALUES (?, ?)”, [[3, ‘three’], [4, ‘four’]])</pre><pre>conn.execute(“SELECT * from test_table “).fetchdf()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/185/1*60AcJskB7xE3aChLy9_pjQ.png" /></figure><p>Use .fetchnumpy() to display your results in form of a <a href="https://numpy.org/">NumPy</a> array. Things are going to get quite interesting as we will be learning various ways to execute complex SQL queries for data analysis</p><pre><strong>conn.execute(“SELECT * FROM test_table”).fetchnumpy()</strong></pre><pre>{&#39;i&#39;: array([1, 9, 2, 3, 4, 1, 9, 2, 3, 4, 1, 9, 2, 3, 4, 1, 9, 2, 3, 4, 1, 9,<br>        2, 3, 4, 1, 9, 2, 3, 4], dtype=int32),<br> &#39;j&#39;: array([&#39;one&#39;, &#39;nine&#39;, &#39;two&#39;, &#39;three&#39;, &#39;four&#39;, &#39;one&#39;, &#39;nine&#39;, &#39;two&#39;,<br>        &#39;three&#39;, &#39;four&#39;, &#39;one&#39;, &#39;nine&#39;, &#39;two&#39;, &#39;three&#39;, &#39;four&#39;, &#39;one&#39;,<br>        &#39;nine&#39;, &#39;two&#39;, &#39;three&#39;, &#39;four&#39;, &#39;one&#39;, &#39;nine&#39;, &#39;two&#39;, &#39;three&#39;,<br>        &#39;four&#39;, &#39;one&#39;, &#39;nine&#39;, &#39;two&#39;, &#39;three&#39;, &#39;four&#39;], dtype=object)}</pre><h3>Pandas Dataframe and SQL</h3><p>In this section, we are going to play around with pandas dataframe and learn various ways to read .csv files. First, we are going to create a simple Pandas dataframe using a dictionary and then we are going to add it to a new table called <em>test_df.</em></p><p>Using SQL script to find all the values of <strong>j </strong>where <strong>i</strong> is greater than one. The query results show three samples matching our conditions.</p><pre>import pandas as pd</pre><pre>test_df = pd.DataFrame.from_dict({“i”:[1, 2, 3, 4], “j”:[“one”, “two”, “three”, “four”]})</pre><pre>conn.register(“test_df”, test_df)</pre><pre>conn.execute(“SELECT j FROM test_df WHERE i &gt; 1”).fetchdf()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/1*h_6NnRcBqoDhtBVX4cUSBg.png" /></figure><p>In the next step, we are going to import .csv to pandas dataframe and then add it to the new table <em>bank_df</em>. The data set used in this example is from <a href="https://www.kaggle.com/kingabzpro/bank-debt-data">Kaggle</a> under the <a href="http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html">GPL2</a> license. The results show the actual recovery amount where the age is greater than 27. With each step, our complex queries are getting simple to execute.</p><pre>df = pd.read_csv(“bank_data.csv”)</pre><pre>conn.register(“bank_df”, df)</pre><pre>conn.execute(“SELECT actual_recovery_amount FROM bank_df WHERE age &gt; 27”).fetchdf()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/1*Utvze2lX6QrdHSCjTuLoBg.png" /></figure><h3>Relations</h3><p>Relation API uses programmatic queries to evaluate chains f relations commands. In short, you can run python functions on relations and display the results. The results contain an experienced Tree, results in columns and results in a preview. If you are feeling confused right now then it’s ok because we are going to go deep into relations and how to use them to get analytical results.</p><h4>Loading Relations</h4><p>Creating relation from existing Pandas dataframe <em>test_df </em>using conn.from_df(test_df) . By printing rel we can visualize the entire dataframe.</p><pre><strong>rel = conn.from_df(test_df)</strong></pre><pre><strong>rel</strong></pre><pre>---------------------<br>-- Expression Tree --<br>---------------------<br>pandas_scan(140194817412592)<br><br>---------------------<br>-- Result Columns  --<br>---------------------<br>- i (BIGINT)<br>- j (VARCHAR)<br><br>---------------------<br>-- Result Preview  --<br>---------------------<br>i	j	<br>BIGINT	VARCHAR	<br>[ Rows: 4]<br>1	one	<br>2	two	<br>3	three	<br>4	four</pre><p>You can also use duckdb.df() for similar results.</p><pre>rel = duckdb.df(test_df)</pre><p>Use conn.table to create a relation from the existing table. In our case, we are using <em>test_table</em>.</p><pre><strong>rel = conn.table(“test_table”)</strong></pre><pre><strong>rel</strong></pre><pre>---------------------<br>-- Expression Tree --<br>---------------------<br>Scan Table [test_table]<br><br>---------------------<br>-- Result Columns  --<br>---------------------<br>- i (INTEGER)<br>- j (VARCHAR)<br><br>---------------------<br>-- Result Preview  --<br>---------------------<br>i	j	<br>INTEGER	VARCHAR	<br>[ Rows: 5]<br>1	one	<br>9	nine	<br>2	two	<br>3	three	<br>4	four</pre><p>For creating the relation directly from .csv file use .from_csv_auto .</p><pre><strong>rel = duckdb.from_csv_auto(“bank_data.csv”)</strong></pre><pre><strong>rel</strong></pre><pre>---------------------<br>-- Expression Tree --<br>---------------------<br>read_csv_auto(bank_data.csv)<br><br>---------------------<br>-- Result Columns  --<br>---------------------<br>- id (INTEGER)<br>- expected_recovery_amount (INTEGER)<br>- actual_recovery_amount (DOUBLE)<br>- recovery_strategy (VARCHAR)<br>- age (INTEGER)<br>- sex (VARCHAR)<br><br>---------------------<br>-- Result Preview  --<br>---------------------<br>id	expected_recovery_amount	actual_recovery_amount	recovery_strategy	age	sex	<br>INTEGER	INTEGER	DOUBLE	VARCHAR	INTEGER	VARCHAR	<br>[ Rows: 10]<br>2030	194	263.540000	Level 0 Recovery	19	Male	<br>1150	486	416.090000	Level 0 Recovery	25	Female	<br>380	527	429.350000	Level 0 Recovery	27	Male	<br>1838	536	296.990000	Level 0 Recovery	25	Male	<br>1995	541	346.385000	Level 0 Recovery	34	Male	<br>731	548	520.710000	Level 0 Recovery	35	Male	<br>221	549	221.585000	Level 0 Recovery	33	Male	<br>1932	560	373.720000	Level 0 Recovery	19	Female	<br>1828	562	201.865000	Level 0 Recovery	22	Female	<br>2001	565	504.885000	Level 0 Recovery	27	Male</pre><h4>Relational Information</h4><p>Use rel.alias to check the name given to the relation. In our case it&#39;s <em>bank_data.csv.</em></p><pre><strong>rel.alias</strong></pre><pre>&#39;bank_data.csv&#39;</pre><p>We can change the current alias by using .set_alias . This might get useful for joining similar relations.</p><pre><strong>rel2 = rel.set_alias(‘bank_data’)</strong></pre><pre><strong>rel2.alias</strong></pre><pre>&#39;bank_data&#39;</pre><p>The type of our relations in <em>Subquery_Relation.</em></p><pre><strong>rel.type</strong></pre><pre>&#39;SUBQUERY_RELATION&#39;</pre><p>To inspect the columns name of relation, just use .columns .</p><pre><strong>rel.columns</strong></pre><pre>[&#39;id&#39;,<br> &#39;expected_recovery_amount&#39;,<br> &#39;actual_recovery_amount&#39;,<br> &#39;recovery_strategy&#39;,<br> &#39;age&#39;,<br> &#39;sex&#39;]</pre><p>Use .types to inspect column types.</p><pre><strong>rel.types</strong></pre><pre>[&#39;INTEGER&#39;, &#39;INTEGER&#39;, &#39;DOUBLE&#39;, &#39;VARCHAR&#39;, &#39;INTEGER&#39;, &#39;VARCHAR&#39;]</pre><h4>Applying python like functions (Single function)</h4><p>Now comes the fun part. You can add any function with the relation and it will display augmented results. In our case, we have used rel.filter and it displays results of age greater than 18. It’s also displaying additional information about the expression tree, which can be quite useful if you have a long complex query to run.</p><pre><strong>rel.filter(‘age &gt; 18’)</strong></pre><pre>---------------------<br>-- Expression Tree --<br>---------------------<br>Filter [age&gt;18]<br>  read_csv_auto(bank_data.csv)<br><br>---------------------<br>-- Result Columns  --<br>---------------------<br>- id (INTEGER)<br>- expected_recovery_amount (INTEGER)<br>- actual_recovery_amount (DOUBLE)<br>- recovery_strategy (VARCHAR)<br>- age (INTEGER)<br>- sex (VARCHAR)<br><br>---------------------<br>-- Result Preview  --<br>---------------------<br>id	expected_recovery_amount	actual_recovery_amount	recovery_strategy	age	sex	<br>INTEGER	INTEGER	DOUBLE	VARCHAR	INTEGER	VARCHAR	<br>[ Rows: 10]<br>2030	194	263.540000	Level 0 Recovery	19	Male	<br>1150	486	416.090000	Level 0 Recovery	25	Female	<br>380	527	429.350000	Level 0 Recovery	27	Male	<br>1838	536	296.990000	Level 0 Recovery	25	Male	<br>1995	541	346.385000	Level 0 Recovery	34	Male	<br>731	548	520.710000	Level 0 Recovery	35	Male	<br>221	549	221.585000	Level 0 Recovery	33	Male	<br>1932	560	373.720000	Level 0 Recovery	19	Female	<br>1828	562	201.865000	Level 0 Recovery	22	Female	<br>2001	565	504.885000	Level 0 Recovery	27	Male</pre><p>Using .project will display mentioned columns and in our case is displaying <strong>id </strong>and<strong> age.</strong></p><pre># project the relation, get some columns</pre><pre><strong>rel.project(‘id, age’)</strong></pre><pre>---------------------<br>-- Expression Tree --<br>---------------------<br>Projection [id, age]<br>  read_csv_auto(bank_data.csv)<br><br>---------------------<br>-- Result Columns  --<br>---------------------<br>- id (INTEGER)<br>- age (INTEGER)<br><br>---------------------<br>-- Result Preview  --<br>---------------------<br>id	age	<br>INTEGER	INTEGER	<br>[ Rows: 10]<br>2030	19	<br>1150	25	<br>380	27	<br>1838	25	<br>1995	34	<br>731	35	<br>221	33	<br>1932	19	<br>1828	22	<br>2001	27</pre><p>You can transform your columns values by adding a number or using any arithmetic function. In our case, it’s displaying age with +1.</p><pre><strong>rel.project(‘age + 1’)</strong></pre><pre>---------------------<br>-- Expression Tree --<br>---------------------<br>Projection [age + 1]<br>  read_csv_auto(bank_data.csv)<br><br>---------------------<br>-- Result Columns  --<br>---------------------<br>- age + 1 (INTEGER)<br><br>---------------------<br>-- Result Preview  --<br>---------------------<br>age + 1	<br>INTEGER	<br>[ Rows: 10]<br>20	<br>26	<br>28	<br>26	<br>35	<br>36	<br>34	<br>20	<br>23	<br>28</pre><p>Order is similar to SQL script ORDER .</p><pre>rel.order(‘sex’)</pre><p>The .limit shows top samples in a table. In our case, it will only display the top 2 values.</p><pre>rel.limit(2)</pre><h4>Stacking all the functions in a chain</h4><p>Just like in R, you can stack all of your functions to get the SQL output. In our case, it’s displaying the top two <em>actual_recovery_amount</em>, order by <em>sex</em> of the people with <em>age</em> greater than 27. We are now realizing the full advantage of the expression tree.</p><pre><strong>rel.filter(‘age &gt; 27’).project(‘actual_recovery_amount’).order(‘sex’).limit(2)<br></strong>---------------------<br>-- Expression Tree --<br>---------------------<br>Limit 2<br>  Order [sex DESC]<br>    Projection [actual_recovery_amount]<br>      Filter [age&gt;27]<br>        read_csv_auto(bank_data.csv)<br><br>---------------------<br>-- Result Columns  --<br>---------------------<br>- actual_recovery_amount (DOUBLE)<br><br>---------------------<br>-- Result Preview  --<br>---------------------<br>actual_recovery_amount	<br>DOUBLE	<br>[ Rows: 2]<br>278.720000	<br>245.000000</pre><h4>Aggregate functions</h4><p>The aggregate function can perform multiple group tasks. In this case, it’s summing all the actual recovery amount.</p><pre><strong>rel.aggregate(“sum(actual_recovery_amount)”)</strong></pre><pre>---------------------<br>-- Result Preview  --<br>---------------------<br>sum(actual_recovery_amount)	<br>DOUBLE	<br>[ Rows: 1]<br>7529821.469511</pre><p>The function below will display the sum of <em>actual_recovery_amount</em> per age group. We have age columns and sum columns. This is super cool as we have reduced two functions into one.</p><pre><strong>rel.aggregate(“age, sum(actual_recovery_amount)”)</strong></pre><pre>---------------------<br>-- Result Preview  --<br>---------------------<br>age	sum(actual_recovery_amount)	<br>INTEGER	DOUBLE	<br>[ Rows: 10]<br>19	52787.712089	<br>25	72769.342330	<br>27	67569.292950	<br>34	109902.427032	<br>35	115424.466724	<br>33	138755.807230	<br>22	46662.153746	<br>31	92225.534688	<br>18	39969.573274	<br>32	110627.466806</pre><p>If you want to only show the sum of actual _recovery amount then add group by columns as a second input. In our case, it’s only showing the sum of actual recovery amounts per age.</p><pre><strong>rel.aggregate(“sum(actual_recovery_amount)”, “age”)</strong></pre><pre>---------------------<br>-- Result Preview  --<br>---------------------<br>sum(actual_recovery_amount)	<br>DOUBLE	<br>[ Rows: 10]<br>52787.712089	<br>72769.342330	<br>67569.292950	<br>109902.427032	<br>115424.466724	<br>138755.807230	<br>46662.153746	<br>92225.534688	<br>39969.573274	<br>110627.466806</pre><p>To display unique values use distinct()</p><pre><strong>rel.distinct()</strong></pre><pre>---------------------<br>-- Result Preview  --<br>---------------------<br>id	expected_recovery_amount	actual_recovery_amount	recovery_strategy	age	sex	<br>INTEGER	INTEGER	DOUBLE	VARCHAR	INTEGER	VARCHAR	<br>[ Rows: 10]<br>2030	194	263.540000	Level 0 Recovery	19	Male	<br>1150	486	416.090000	Level 0 Recovery	25	Female	<br>380	527	429.350000	Level 0 Recovery	27	Male	<br>1838	536	296.990000	Level 0 Recovery	25	Male	<br>1995	541	346.385000	Level 0 Recovery	34	Male	<br>731	548	520.710000	Level 0 Recovery	35	Male	<br>221	549	221.585000	Level 0 Recovery	33	Male	<br>1932	560	373.720000	Level 0 Recovery	19	Female	<br>1828	562	201.865000	Level 0 Recovery	22	Female	<br>2001	565	504.885000	Level 0 Recovery	27	Male</pre><h4>Multi-relation operators</h4><p>We can create UNION between two relations by using .union This combine the results of both relations.</p><pre><strong>rel.union(rel)<br></strong>---------------------<br>-- Expression Tree --<br>---------------------<br>Union<br>  read_csv_auto(bank_data.csv)  read_csv_auto(bank_data.csv)</pre><p>Joining two relations on <em>id </em>column. We have created <em>rel2 </em>and joined it to<em> rel </em>based on<em> id.</em></p><pre><strong>rel2 = duckdb.df(df)</strong></pre><pre><strong>rel.join(rel2, ‘id’)</strong></pre><pre>---------------------<br>-- Expression Tree --<br>---------------------<br>Join<br>  read_csv_auto(bank_data.csv)  pandas_scan(139890483423984)</pre><p>To join similar relation we will be using alias() to change the alias and then joining them as shown below.</p><pre>rel.set_alias(‘a’).join(rel.set_alias(‘b’), ‘a.id=b.id’)</pre><pre> — — — — — — — — — — — — Expression Tree — — — — — — — — — — — — Join read_csv_auto(bank_data.csv) read_csv_auto(bank_data.csv)</pre><h3>DuckDB functions on dataframe.</h3><p>We can skip creating relations and dive right into filtering and sorting by using duckdb.&lt;fucntion&gt;(&lt;dataframe&gt;,&lt;script&gt;). The examples show how you can create results using Pandas dataframe directly.</p><pre>print(duckdb.filter(df, ‘age &gt; 1’))</pre><pre>print(duckdb.project(df, ‘age +1’))</pre><pre>print(duckdb.order(df, ‘sex’))</pre><pre>print(duckdb.limit(df, 2))</pre><p>Similarly, we can create a chain of functions and display the results as shown below.</p><pre><strong>duckdb.filter(df, ‘age &gt; 1’).project(‘age + 1’).order(‘sex’).limit(2)</strong></pre><pre>---------------------<br>-- Expression Tree --<br>---------------------<br>Limit 2<br>  Order [sex DESC]<br>    Projection [age + 1]<br>      Filter [age&gt;1]<br>        pandas_scan(139890483423984)<br><br>---------------------<br>-- Result Columns  --<br>---------------------<br>- age + 1 (BIGINT)<br><br>---------------------<br>-- Result Preview  --<br>---------------------<br>age + 1	<br>BIGINT	<br>[ Rows: 2]<br>26	<br>20</pre><h3>Relation Query Results</h3><p>To display the results of the relation use:</p><ul><li><strong>fetchone()</strong>: to display only the first row and you can repeat this command until you have cycled through all.</li><li><strong>fetchall()</strong>: to display all the values from the results.</li><li><strong>fetchdf()</strong>: to display results in form of Pandas Dataframe.</li><li><strong>fethnumpy()</strong>: to display results in form of a Numpy array.</li></ul><pre><strong>res = rel.execute()</strong></pre><pre><strong># res is a query result, you can call fetchdf() or fetchnumpy() or fetchone() on it</strong></pre><pre><strong>print(res.fetchone())<br></strong>(2030, 194, 263.54, ‘Level 0 Recovery’, 19, ‘Male’)</pre><pre><strong>print(res.fetchall())</strong></pre><pre>[(1150, 486, 416.09, ‘Level 0 Recovery’, 25, ‘Female’), (380, 527, 429.35, ‘Level 0 Recovery’, 27, ‘Male’).....</pre><ul><li>We can also convert the results into Pandas dataframe by using rel.df() or rel.to_df() as shown below.</li></ul><pre>rel.to_df()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*ZU-KIDryHLMp9sFN8qhEpA.png" /></figure><h3>Tables and DuckDB</h3><p>We can also create tables in relation by using .create</p><pre>rel.create(“test_table2”)</pre><p>In the next part, we are going to learn various ways to insert values into relation.</p><ul><li>Create a new table <em>test_table3.</em></li><li><em>Insert values using </em><em>values() and </em>insert_into(&lt;tablename&gt;)</li><li>create a relationship by using test_table3.</li><li>Insert values into relation using simple .insert function.</li></ul><p>As we can see we have successfully added a single value to our relationship.</p><pre><strong># Inserting elements into table_3</strong></pre><pre><strong>conn.execute(“CREATE TABLE test_table3 (i INTEGER, j STRING)”)</strong></pre><pre><strong>print(conn.values([5, ‘five’]).insert_into(“test_table3”))</strong></pre><pre><strong>rel_3 = conn.table(“test_table3”)</strong></pre><pre><strong>rel_3.insert([6,’six’])</strong></pre><pre><strong>rel_3</strong></pre><pre>---------------------<br>-- Expression Tree --<br>---------------------<br>Scan Table [test_table3]<br><br>---------------------<br>-- Result Columns  --<br>---------------------<br>- i (INTEGER)<br>- j (VARCHAR)<br><br>---------------------<br>-- Result Preview  --<br>---------------------<br>i	j	<br>INTEGER	VARCHAR	<br>[ Rows: 2]<br>5	five	<br>6	six</pre><h3>DuckDB Query function</h3><p>To run SQL queries directly you can use .query with your relations. Then add the first input as the view name of the query and the second input as SQL script using the view name as shown below.</p><pre>res = rel.query(‘my_name_for_rel’, ‘SELECT * FROM my_name_for_rel LIMIT 5’)</pre><p>In order to see the results, let’s use fetchall() to display all 5 values.</p><pre><strong>res.fetchall()</strong></pre><pre>[(2030, 194, 263.54, &#39;Level 0 Recovery&#39;, 19, &#39;Male&#39;),<br> (1150, 486, 416.09, &#39;Level 0 Recovery&#39;, 25, &#39;Female&#39;),<br> (380, 527, 429.35, &#39;Level 0 Recovery&#39;, 27, &#39;Male&#39;),<br> (1838, 536, 296.99, &#39;Level 0 Recovery&#39;, 25, &#39;Male&#39;),<br> (1995, 541, 346.385, &#39;Level 0 Recovery&#39;, 34, &#39;Male&#39;)]</pre><p>We can directly use .query() and run a query on <em>test_df </em>table. The results show all four values from the table. The query function can perform all the functions on your table, dataframe, and even on .csv files.</p><pre>res = duckdb.query(‘SELECT * FROM test_df’)</pre><pre>res.df()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/163/1*qLgzwEWGoVMTp07M4BuZQQ.png" /></figure><p>In order to run query on .csv file we will be using read_csv_auto() in SQL script. In our case, we are reading the entire data directly from a CSV file and displaying it in Pandas dataframe.</p><pre>res = duckdb.query(“SELECT * FROM read_csv_auto(‘bank_data.csv’)”)</pre><pre>res.df()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*bFH8UYuYHjWcx_CyBN6ORw.png" /></figure><p>Let’s run a complex script so that we can realize the full potential of DuckDB. The result below shows that more amount was recovered the expected from a female using the level 4 strategy. Using SQL queries for data analysis is fast and fun, instead of loading your data into pandas and writing a bunch of code to get similar results.</p><pre>res = duckdb.query(“SELECT sex, SUM(expected_recovery_amount) as Expected,\<br> SUM(actual_recovery_amount) as Actual \<br> FROM ‘bank_data.csv’\<br> WHERE recovery_strategy LIKE ‘Level 4 Recovery’\<br> GROUP BY sex\<br> HAVING sex = ‘Female’”)<br>res.df()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*aBLQxyveMuJf251NY5c4hA.png" /></figure><h3>Deepnote SQL Cell</h3><p><a href="https://deepnote.com/">Deepnote</a>’s <a href="https://docs.deepnote.com/features/sql-cells/dataframe-sql">Dataframe SQL cell</a> is implemented using DuckDB, so if you want an easy way to use all the functions of DuckDB, I recommend you to use Deepnote SQL cell. To run your SQL query first all add an SQL cell from the block option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*rHhj1QiUyVDQ9FDzU6GqqQ.png" /></figure><p>Try writing a simple SQL script. As we can see we are reading data directly from <em>bank_data.csv </em>and the results are stored in <em>df_1</em> in form of Pandas dataframe<em>. </em>I just love this feature, it&#39;s even better than loading your data through Pandas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*EwJQ8UGejE2YdSvdhbCQGg.png" /></figure><p>Let’s try a more complex SQL query on a CSV file. We will be displaying three columns <em>sex, expected</em>, and <em>actual</em>. Then, we will select values where <em>recovery_stretagy</em> is ‘Level 4 Recovery’ and group by <em>sex</em>. Finally, we will only display <em>female values</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*1CEYvpdJFE4x-u_QG17KFQ.png" /></figure><h3>Final thoughts</h3><p>DuckDB can solve most of the issues present in traditional SQL databases. It is a serverless database management system with faster analytics queries and it’s completely integrated with Python and R. If you are going to deploy a data science or machine learning application I suggest you add DuckDB to your stack.</p><p>In this guide, we have learned various ways to create a database, add tables, and run queries. We have also learned about relations and we can use them to create complex functions. Finally, we have used various ways to directly interact with CSV files and ran multiple complex scripts. This guide includes all kinds of examples so if you get stuck in your project, you can also come back and learn some tricks.</p><blockquote>In the end, I will always suggest you to learn the basics by practicing and look for other resources to improve your knowledge about various database systems.</blockquote><h3>Learning Resources</h3><ul><li><a href="https://dev.to/volkmarr/duckdb-an-embedded-db-for-data-wrangling-4hfm">DuckDB: an embedded DB for data wrangling — DEV Community 👩‍💻👨‍💻</a></li><li><a href="https://www.w3schools.com/sql/default.asp">SQL Tutorial (w3schools.com)</a></li><li><a href="https://duckdb.org/docs/sql/introduction">DuckDB — SQL Introduction</a></li><li><a href="https://deepnote.com/@allan-campopiano/Intro-to-SQL-Cells-HxtuNpQrSrOwmEkwb8qB8w">Intro to SQL Cells (deepnote.com)</a></li><li><a href="https://docs.deepnote.com/features/sql-cells/dataframe-sql">DataFrame SQL — Deepnote docs</a></li></ul><p><strong>Repost:</strong> The <a href="https://www.analyticsvidhya.com/blog/2021/12/the-guide-to-data-analysis-with-duckdb/">original</a> post was published on Analytics Vidhya.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93a79950a08f" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/the-guide-to-data-analysis-with-duck-db-93a79950a08f">The Guide To Data Analysis With Duck DB</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Find The Best Data Science Remote Jobs]]></title>
            <link>https://abidaliawan.medium.com/how-to-find-the-best-data-science-remote-jobs-a122f22f2f97?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a122f22f2f97</guid>
            <category><![CDATA[job-hunting]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[freelancing]]></category>
            <category><![CDATA[remote-job]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Fri, 27 May 2022 14:55:10 GMT</pubDate>
            <atom:updated>2022-05-28T13:46:49.244Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn Why, What, and How about the best data science remote jobs</h4><figure><img alt="How To Find The Best Data Science Remote Jobs? | Cover by Abid Ali Awan" src="https://cdn-images-1.medium.com/max/1024/1*oonFBJrsSMEn2czzcMAHQQ.png" /><figcaption>Image by Author | Canva Pro</figcaption></figure><h3>Why Should You Apply for Remote Data Science Jobs?</h3><p>Remote jobs provide a work-life balance and the opportunity to learn versatile skills. You get to enjoy the freedom that you always dream of. Furthermore, remote data science jobs offer fabulous salaries and competitive benefits. There is a high demand for tech workers, so you won’t have to look hard to find the best job. According to the <a href="https://www.bls.gov/ooh/computer-and-information-technology/computer-and-information-research-scientists.htm">US Bureau of Labor Statistics</a>, the number of jobs will increase by 20 percent for Computer and Information Research Scientists.</p><p>The educational requirement for these jobs are not that high, but a bachelor’s degree in data science, computer science, statistics, or any related field is desirable. Getting a master’s or Ph.D. will increase the chance of hiring. Candidates without a formal degree can also apply if they can showcase the skills to work on a professional data science project.</p><p>I have a bachelors in Telecommunication Engineering and Masters in Technology Management which is quite different from data science, so how did I get my first remote job? I learned essential skills, took multiple courses, and got professional certification. In data science, your skills and experience matter more than education.</p><figure><img alt="Data Professional using working in a desert Freepik Photo" src="https://cdn-images-1.medium.com/max/1024/1*Y8ryCaZJreNtXE6NxIhOhg.jpeg" /><figcaption>Photo by <a href="https://www.freepik.com/free-photo/businessman-using-laptop-desert_8924077.htm#query=remote%20jobs&amp;position=0&amp;from_view=search">master1305</a></figcaption></figure><h3>What Are Data Science Remote Jobs?</h3><p>You can apply for Data Analyst, Data Scientist, Data Engineer, Machine Learning Engineer, MLOps Engineer, Database Manager, Business Intelligence (BI) Developer, and Data Architect. These remote data jobs require you to have:</p><ul><li>Best internet connection</li><li>Ability to use multiple communication and productivity tools</li><li>Ability to work alone and meet tasks deadline</li><li>Understanding of technical terms and business use case</li><li>Ability to switch from one task to another based on priority</li><li>Ability to understand and solve data science problems using modern data stack</li></ul><p>The list is long, but we have covered the most vital skills.</p><p>At the start, I only knew one or two things, mostly related to technical knowledge, and with time I got used to modern tools such as <strong>Slack</strong>, <strong>Monday</strong>, <strong>Zoom</strong>, <strong>Google Calendar</strong> and <strong>Docs</strong>, and <strong>VS Code</strong> extensions. Apart from productivity tools, you will learn to manage cloud applications, retrieve data using SQL, create data analysis reports, and learn the entire data ecosystem from data ingestion to production.</p><p>In day to day job, you will be assigned a task or multiple tasks, and by the end of the day, you need to provide a summary of success and failure. Most of the time, you are working alone and meet with the team on Zoom call after a day or a week. Your manager will not micromanage you, and you have freedom, and you will be also saving money and time on traveling.</p><figure><img alt="Hispanic Data Scientist using a digital tablet" src="https://cdn-images-1.medium.com/max/1024/1*z93zyQ7dtg3i-IS5dJxzeQ.jpeg" /><figcaption>Photo by <a href="https://www.freepik.com/free-photo/hispanic-woman-using-digital-tablet_16250072.htm">rawpixel.com</a></figcaption></figure><h3>5 In-Demand Data Science Remote Jobs</h3><p>In this section, we will cover five in-demand data science fields and describe the job role in a few lines. The average salary and top companies’ information is extracted from <a href="https://www.indeed.com/career">Indeed</a> on 25th May 2022.</p><h4>Data Scientist</h4><ul><li><strong>Average remote salary: </strong>$102,365 /year</li><li><strong>Top companies offering remote jobs:</strong> Selby Jennings, Airbnb, eBay, and Apple</li></ul><p>The remote data scientist solves business problems using statistics and technical tools. The data scientist must have experience in accessing data using SQL, data analysis using Python or R, and building predictive models. This job is flexible and mostly depended on company’s requirements. Sometimes you might have to do data engineering, machine learning, and web developing tasks.</p><h4>Business Analyst</h4><ul><li><strong>Average remote salary:</strong> $90,510 /year</li><li><strong>Top companies offering remote jobs:</strong> Uline, Twitter, and Wells Fargo</li></ul><p>Business analysts work closely with stakeholders to find solutions to a business problems. They conduct research and analysis using business intelligence tools (Power BI, Tableau). In rear case, a company want you to have experience in web scraping, data cleaning, SQL, Dashboarding, web applications, and understanding business terminologies (KPIs, ROI).</p><h4>Data Analyst</h4><ul><li><strong>Average remote salary:</strong> $65,319 /year</li><li><strong>Top companies offering remote jobs:</strong> Citizens, Meta, and USAA</li></ul><p>Data analysis is high in demand, and you need a basic understanding of statistics and analytical tools to get in. Your job will be running analysis on various data sources, creating a report, communicating critical information, and creating predictive models.</p><h4>Machine learning Engineer</h4><ul><li><strong>Average remote salary:</strong> $111,385 /year</li><li><strong>Top companies offering remote jobs:</strong> eBay, Snap, and Capital One</li></ul><p>It is the highly lucrative job in the tech industry, and some companies are offering a $300K+ salary with benefits. You get to test, train, build, and deploy machine learning models. You will also require to create a machine learning pipeline and automate the deployment system. The work you do will have a direct impact on the company’s growth.</p><h4>Data Engineer</h4><ul><li><strong>Average remote salary:</strong> $115,602 /year</li><li><strong>Top companies offering remote jobs:</strong> Netflix, Meta, and LinkedIn</li></ul><p>Data engineers can earn up to $300K per year by building and maintaining data pipelines. You get to create and manage databases, create ETL/ELT pipelines, distributed computing, automate pipelines, and monitor data integrity. The whole data team is depend on you to provide them with accurate and clean datasets.</p><figure><img alt="Data Scince remote job search" src="https://cdn-images-1.medium.com/max/1024/1*q_X95bd0-TXNsbzsH9f9oA.jpeg" /><figcaption>Photo by <a href="https://www.freepik.com/free-photo/job-career-hiring-recruitment-qualification-graphic_17096429.htm">rawpixel.com</a></figcaption></figure><h3>Where to Find Remote Data Science Jobs?</h3><p>You can find full-time remote work on Job boards, the company’s website, social media, and data science communities. Remote work also gives you the ability to get part-time or freelancing jobs that pays the same. In this section, we will learn about platforms to find full-time and freelancing jobs.</p><h4>Full-Time Jobs</h4><p>You can create profiles and search for full-time work on job boards, such as <a href="https://www.glassdoor.com/index.htm">Glassdoor</a>, <a href="https://www.ziprecruiter.com/">ZipRecruiter</a>, <a href="https://www.coderemote.com/jobs">CodeRemote</a>, <a href="https://www.linkedin.com/">LinkedIn</a>, and <a href="https://pk.indeed.com/?r=us">Indeed</a>. You can also join data science communities on Facebook, LinkedIn, Discord, and Slack to improve your chance of getting hired faster.</p><p>There are some hidden gems such as <a href="https://www.datacamp.com/data-jobs"><strong>DataCamp Jobs</strong></a>, that helps data professionals find the best job. Instead of showing random jobs to you, it will select a few jobs tailored to you. You just need to create an account and fill in your preference, and within no time, you will start to get job offers from top companies such as Discord, Greenbox Capital, General Motors, and many more.</p><h4>Freelance Work</h4><p>You can find freelance work on <a href="https://www.upwork.com/">Upwork</a>, <a href="https://angel.co/">AngelList</a>, <a href="https://www.kolabtree.com/">Kolabtree</a>, and <a href="https://www.fiverr.com/">Fiverr</a>. Moreover, you can also find contract freelance work on the job boards by changing job types to part-time, contract, or temporary.</p><p>Similarly, we have a freelance hidden gem called <a href="https://www.toptal.com/"><strong>Toptal</strong></a><strong>. </strong>The platform has higher standards of only hiring the top 3% of freelance talent. It also requires you to have 2+ years of experience in data science. For experienced data professionals, this platform is a haven for providing high-paying jobs with benefits and flexible working contracts.</p><figure><img alt="Getting a Job offer in Data Science profession" src="https://cdn-images-1.medium.com/max/1024/1*SGYJ6a9ewDFcFsIVuJ2DAg.jpeg" /><figcaption>Photo by <a href="https://www.freepik.com/free-photo/excited-young-woman-hold-paper-letter-feel-euphoric-receiving-job-promotion-tax-refund-from-bank-happy-woman-reading-paperwork-document-smiling-good-pleasant-news-getting-student-scholarship_27158628.htm#query=getting%20a%20job&amp;position=8&amp;from_view=search">stefamerpik</a></figcaption></figure><h3>Final Thoughts</h3><p>Before you apply for remote jobs or freelance work, please make sure you have the required skills and experience.</p><p>To learn more skills, you can take online courses and tutorials, read books, start a pet project, and work on research.</p><p>To gain more experience, you can take part in Kaggle competitions, collaborate on open source projects, work for NGOs, apply for internships and apprenticeships, and temporary jobs.</p><blockquote>If you have questions about a data science career, do ask me in the comments.</blockquote><p><strong>Best of luck with finding your next remote job.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a122f22f2f97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Tips for Acing Your First Data Science Interview]]></title>
            <link>https://medium.com/data-science/5-tips-for-acing-your-first-data-science-interview-dbad5b5959d7?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbad5b5959d7</guid>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[cheatsheet]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Mon, 16 May 2022 18:18:40 GMT</pubDate>
            <atom:updated>2022-05-16T18:18:40.750Z</atom:updated>
            <content:encoded><![CDATA[<h4>If you are looking for a full-time, part-time, or even summer internship opportunity in the field of data science, then check out 5 simple tips on clearing behavioral and technical interview stage.</h4><figure><img alt="Cover image for blog on “5 Tips for Acing Your First Data Science Interview”" src="https://cdn-images-1.medium.com/max/1024/1*WLR0xBq9D_K7fHUbOqRozg.png" /><figcaption>Image by Author using Canva Pro</figcaption></figure><p>If you are a fresh graduate looking to start a career, an expert switching jobs for better opportunities, or even a student applying for a summer internship, then you must know that a job interview is a tricky part of the recruitment process. To clear multiple rounds of interviews, an applicant must showcase expertise in domain knowledge, technical experience, interpersonal skills, and understanding of the job and company. You need to know the value you bring to the company and decipher hiring manger expectations.</p><p>When I started applying for jobs, I had no idea how data science interview works. I was switching from the management field, so I thought maybe they will ask me simple questions about data science and how to manage a project. I was wrong about most of the things. The data science interviews are quite different. You need to have the knowledge of technical terms, modern frameworks, coding, business use case, and statistics.</p><p>After learning from my failure, I have curated five tips that worked for me, and I am sure they will work for you. Apart from these tips, you just need self-confidence and practice to ace any interview.</p><h3>Review the Job and Company Profile</h3><p>Forgetting about the specifics of the job description and vision of a company is the most common mistake that an applicant makes. The hiring manager will identify these mistakes in the first few minutes, and by that time he/she had made the mind to reject you. Why? Because they think that you are not serious about the job.</p><p>If you have applied through LinkedIn or any recruitment platform, always go back and put the time to research the company and job description. Read the job requirement, day-to-day tasks, and what companies are expecting from you. Make sure you understand the organization’s motto and work environment. Visit the official website, and look for the recent projects. This information will help you impress the interviewer, and to be honest, that is your initial goal.</p><p>The most common question asked by the recruiter is <strong>“Why are you the right person for this job?”</strong> or <strong>“What do you know about our company?”.</strong> If you haven’t prepared for the interview, you will get stuck in these questions, and it will look bad on your job application.</p><figure><img alt="Research the Job and Company Profile Ilustration" src="https://cdn-images-1.medium.com/max/700/1*PmErJHQeuy2cYhlrgjnLyg.png" /><figcaption>Image by Author using Canva Pro</figcaption></figure><h3>Review Your Resume and Projects</h3><p>Nowadays, recruiters focus on portfolio projects, and they will look for <a href="https://github.com/kingabzpro">GitHub</a> or <a href="https://www.kaggle.com/kingabzpro">Kaggle</a> profiles before scheduling a job interview. It helps them assess your technical skills and prepare for the questions related to your recent projects. The questions are not limited to the projects. The interviewer might ask you questions about your previous jobs and the skill mentioned in your resume.</p><p>During a technical interview, I was asked to describe everything about my recent project on Natural Language Processing. They asked me about the tools, technical concepts, and web frameworks. In the second round of interviews, I was asked about the biography on the <a href="https://deepnote.com/@abid">Deepnote profile</a> and resume. So, it is better to go back and review both resume and portfolio projects. Make sure you can answer all of the questions, even the project resulted in failure.</p><figure><img alt="Abid’s Data Science projects." src="https://cdn-images-1.medium.com/max/933/1*Fm9sTaGG-zG9RU7MZo0ycg.png" /><figcaption>Image from Author’s <a href="https://abidaliawan.me/">Website</a></figcaption></figure><h3>Revises Data Science Terms</h3><p>In one instance, I completely forget about how <strong>“stochastic gradient descent”</strong> works, and in another instance, I forgot about the term <strong>“bag of words”</strong>. These situations were quite embarrassing as I have been working on the data science projects for couple of years and still couldn’t answer basic questions. To avoid this happening ever again, I made the habit of revising the technical terms using cheat sheets. Preparing in advance also made me confident about answering programming-related questions. Check out the complete collection of data science cheat sheets <a href="https://www.kdnuggets.com/2022/02/complete-collection-data-science-cheat-sheets-part-1.html">Part 1</a> and <a href="https://www.kdnuggets.com/2022/02/complete-collection-data-science-cheat-sheets-part-2.html">Part 2</a>.</p><figure><img alt="Cover Image “Complete Collection of Data Science Cheat Sheets Part 1 and Part 2”" src="https://cdn-images-1.medium.com/max/1024/1*ksPMEZfG5Rm4m_Y-yxu-Ng.png" /><figcaption>Image by Author | Complete Collection of Data Science Cheat Sheets <a href="https://www.kdnuggets.com/2022/02/complete-collection-data-science-cheat-sheets-part-1.html">Part 1</a> and <a href="https://www.kdnuggets.com/2022/02/complete-collection-data-science-cheat-sheets-part-2.html">Part 2</a></figcaption></figure><p>The cheat sheets provide byte size information about technical terms and if you want to learn more, then check out DataCamp&#39;s resource on data science with <a href="https://www.datacamp.com/learn/python">Python</a>, <a href="https://www.datacamp.com/learn/r">R</a>, and <a href="https://www.datacamp.com/learn/sql">SQL</a>. The resources include courses, blogs, tutorials, cheat sheets, assessment tests, and a guide on certifications.</p><h3>Read Most Asked Interview Questions</h3><p>There are three sessions of interview. The first is conducted by HR manager and it is non-technical focused on your strength and weakness. The second part is more technical where team manager and chief data scientist assess your technical knowledge. The last part is conducted by company’s executives and it is focused on company culture, work life balance, and growth potential.</p><p>I will highly suggest you to go read frequency asked questions in data science interview. These question are both technical and non-technical. You will be surprised to see how you were unprepared after reading <a href="https://www.springboard.com/blog/data-science/data-science-interview-questions/">87 Commonly Asked Data Science Interview Questions</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*-uMJDF78PU4LS5oowfb3EA.png" /><figcaption>Image by Author using Canva Pro</figcaption></figure><p>I will highly recommend you to read <a href="https://www.amazon.com/Ace-Data-Science-Interview-Questions/dp/0578973839/ref=pd_sbs_sccl_2_1/137-5550312-9155521?pd_rd_w=F9JAI&amp;pf_rd_p=3676f086-9496-4fd7-8490-77cf7f43f846&amp;pf_rd_r=GSCE0TAY50BX0WEYR4B2&amp;pd_rd_r=d16b6cae-ed4b-45da-b06e-9a8ae4925c7e&amp;pd_rd_wg=D6aXS&amp;pd_rd_i=0578973839&amp;psc=1">Ace the Data Science Interview</a> by <a href="https://www.amazon.com/Nick-Singh/e/B09DRSNL2Y/ref=dp_byline_cont_book_1">Nick Singh</a> and <a href="https://www.amazon.com/Kevin-Huo/e/B09DTKK4Y7/ref=dp_byline_cont_book_2">Kevin Huo</a>. The book explains how the job application process works and how you can ace your behavioral interview. Furthermore, it dives deep into the technical side of data science. Focusing on probability, statistics, SQL, machine learning, coding, product development, and case studies.</p><h3>Take Few Mock Interviews</h3><p>Mock interviews are more like practice interviews that will improve your body language, talking points, and anxiety. I had performance anxiety where I was unable to respond to the question even though I knew the answer. I even got stuck on simple probability questions.</p><p>To improve my body language and overcome anxiety issues, I started taking mock interviews on <a href="https://www.skilledinc.com/">Skilled</a>. The platform connects you with the top industry HR manager, software engineers, and data scientists. In my case, I was matched with the hiring manager and chief data scientist. The interview has also made me aware of new opportunities in the field and how to approach recruiters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*V7WkZi1e3tG-Oiok8H_bLQ.png" /><figcaption>Image by Author using Canva Pro</figcaption></figure><p>If you don’t want to pay a fee for a mock interview, you can ask your friend or family member to conduct an interview. The primary<strong> </strong>purpose of a mock interview is to get honest feedback, and use it to prepare for your actual<strong> </strong>in-person/online interview.</p><h3><strong>Additional Resources</strong></h3><ul><li><a href="https://www.dataquest.io/blog/data-science-glossary/">Data Science Terms and Jargon</a></li><li><a href="https://www.freecodecamp.org/news/statistics-for-data-science/">Statistics for Data Science</a></li><li><a href="https://www.jigsawacademy.com/blogs/data-science/data-science-framework/">Top 10 Data Science Frameworks</a></li><li><a href="https://appsilon.com/data-science-sql-interview-questions/">Top 10 Data Science SQL Interview Questions and Answers</a></li><li><a href="https://www.datacamp.com/cheat-sheet#!">Data Science Cheat Sheets</a></li><li><a href="https://insightsoftware.com/ultimate-glossary-bi-terms/">The Ultimate Glossary of BI Terms</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbad5b5959d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/5-tips-for-acing-your-first-data-science-interview-dbad5b5959d7">5 Tips for Acing Your First Data Science Interview</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Gradio App on Spaces Using DagsHub: A Beginner’s Tutorial]]></title>
            <link>https://medium.com/data-science/deploying-gradio-app-on-spaces-using-dagshub-a-beginners-tutorial-a42664abcc14?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a42664abcc14</guid>
            <category><![CDATA[gradio]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Tue, 01 Mar 2022 08:25:44 GMT</pubDate>
            <atom:updated>2022-03-01T08:25:44.479Z</atom:updated>
            <content:encoded><![CDATA[<h4>The project based tutorial introduces MLOps integrations such as DVC, DagsHub, Gradio web framework, and Hugging Face Spaces</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ttWU99MdhcyfJ9r3pU7Jxw.png" /><figcaption>Cover by author</figcaption></figure><p>The demand for MLOps tools is high as companies are looking for easy-to-deploy solutions for their complex machine learning (ML) models — <a href="http://v/">AIM</a>. To make things simple and effective, we will be integrating DVC, DagsHub, Gradio, and Hugging Face Spaces into our collaborative ML project.</p><p>In this tutorial, we will learn about end-to-end machine learning integrations and use them in creating image generation web applications. We will be using Gradio to develop model inference on open-source project SavtaDepth which will provide a user-friendly web interface. SavtaDepth converts 2D images into 3D by predicting their depth dimension. This is a beginner-friendly tutorial which means we will learn tricks to simplify the deployment process for complex ML models.</p><h3>SavtaDepth Project</h3><p><a href="https://dagshub.com/OperationSavta/SavtaDepth">SavtaDepth</a> is a collaborative open-source data science project for monocular depth estimation. It takes a 2D image and estimates the depth of objects. The depth estimation is used for creating 3D images, 3D mapping, security surveillance, and self-driving cars. The goal in monocular depth estimation is to predict the depth value of each pixel or infer depth information, given only a single RGB image as input — <a href="https://keras.io/examples/vision/depth_estimation/">(keras.io)</a>.</p><p>The project uses the <a href="https://paperswithcode.com/method/u-net">U-Net</a> model and the author has changed the last layer from object segmentation to depth estimation. The model was trained on <a href="https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html">NYU Depth Dataset V2</a> under the <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a> license. The dataset contains video sequences from a variety of indoor scenes as recorded by both the RGB and Depth cameras from the Microsoft <a href="http://www.xbox.com/kinect">Kinect</a>. The project is still active, so contributors can help in improving the results. If you are interested in the project, please read the contribution <a href="https://dagshub.com/OperationSavta/SavtaDepth#contributing-guide">guide</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TUyxCvtadYnbfOik" /><figcaption>Image by author | Monocular Depth Estimation</figcaption></figure><h3>Integrations</h3><p>In this section, we will learn about various integrations that will make our project unique. We will also learn how these tools fit into our project ecosystem.</p><ul><li><a href="https://dvc.org/"><strong>DVC</strong></a> is an open-source version control machine learning system. It comes with data and model versioning, model metrics monitoring, and reproducing the experiments. It works as an extension to Git, so learning DVC will come naturally.</li><li><a href="https://dagshub.com/"><strong>DagsHub</strong></a> is a community-first platform like GitHub for machine learning and data science projects. It enables its users to leverage popular open-source tools to version datasets &amp; models, track experiments, label data, and visualize results.</li><li><a href="https://www.gradio.app/"><strong>Gradio</strong></a> is the fastest way to create a user-friendly web interface for your machine learning models. You can build and share your app within minutes. It also comes with FastAPI support which means you can access the model anywhere.</li><li><a href="https://huggingface.co/spaces"><strong>Spaces</strong></a> is a new machine learning application sharing platform by Hugging Face. You can build Streamlit, Gradio, or HTML web apps and deploy them to Spaces with few lines of code.</li></ul><p>In this project, we will use <strong>DVC</strong> for data and model versioning, <strong>DagsHub</strong> for the remote storage, <strong>Gradio</strong> for ML application interface, and <strong>Spaces</strong> for a web server.</p><h3>Gradio WebApp</h3><p><strong>Gradio</strong> is a lightweight powerful web interface for creating machine learning demos. In this section, we are going to learn how to add an image, run model inference using FastAI, and output the generated result. To run the web app without dependency issues, we need to first install <a href="https://pytorch.org/get-started/locally/">PyTorch</a>, <a href="https://pypi.org/project/fastai/">FastAI</a>, and <a href="https://www.gradio.app/">Gradio</a>.</p><h4>Model Inference</h4><p>To make things simple, I have added only the important part and removed the data loader. You can learn more about the image data loader and <em>create_data</em> function by following the SavtaDepth <a href="https://colab.research.google.com/drive/1XU4DgQ217_hUMU1dllppeQNw3pTRlHy1?usp=sharing">notebook</a>.</p><p>We are simply using FastAI’s function <em>unet_learner</em> to create model architecture and then load the latest model checkpoint. Finally, we are going to create a <em>gen</em> function that will intake images in the form of a Numpy array and run a prediction to generate a black and white 3D image.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4041ca3387d5cb73248a56cdef22366d/href">https://medium.com/media/4041ca3387d5cb73248a56cdef22366d/href</a></iframe><h4>Web Interface</h4><p>Creating a Gradio web interface is easy. We just need to customize the <em>gradio.Interface </em>function for our use case and Vallah!!! Your web app is ready.</p><p>Let’s closely look at the parameters used in <em>Interface</em> function:</p><ul><li><strong>First</strong> parameter is <em>fn</em>. We need to provide it with the model inference, which takes input and return output. In our case, it <em>gen</em> function.</li><li><strong>Second</strong> parameter is <em>inputs</em>. We are converting the image shape to a 640X480 Numpy array. This function automatically does most of the image processing jobs for us.</li><li><strong>Third</strong> parameter is <em>outputs</em>. In our case, it’s an image. All of the post-processing is done by Gradio automatically. So we do not have to worry about using the PIL package to display images.</li><li><strong>title</strong> takes a String to display the app name or title.</li><li><strong>description</strong> takes a simple String, markdown, or HTML to display subheading or image under the title.</li><li><strong>article</strong> is the footer of the app where you write about application information such as links to your research and project repository. It also takes in simple text, markdown, or HTML.</li><li><strong>examples</strong> can be used as sample inputs so that we don’t have to find images to run the app. In our case, we have created a folder and copied two images from a test subset of the dataset. The examples take in an array of relative file paths as shown below.</li><li><strong>theme is</strong> for customizing UI, you can find more about themes in <a href="https://www.gradio.app/docs/">docs</a>.</li><li><strong>allow_flagging</strong> is a cool feature that helps you keep track of the model performance. You can flag the wrong prediction so that the developer can improve the model based on the feedback.</li><li><strong>enable_queue</strong> was used to prevent inference from time out.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11902a98bd8c1c897774be162124aaa1/href">https://medium.com/media/11902a98bd8c1c897774be162124aaa1/href</a></iframe><p>In the final version, I have made some changes to <strong>description </strong>and <strong>article </strong>by using HTML script. Feel free to check out my <a href="https://dagshub.com/kingabzpro/SavtaDepth/src/main/app/app_savta.py">code</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-OAa0Cdnl6pfCBVY" /><figcaption>Gif by author</figcaption></figure><h4>Deployment</h4><p>In this section, we are going to learn how to deploy our Gradio app to Hugging Face Spaces. For that we need to create new Spaces using Hugging Face website and then make changes in savta_app.py, README.md, and requirement.txt files.</p><h4>Hugging Face Spaces</h4><p><a href="https://huggingface.co/spaces">Spaces</a> by Hugging Face machine learning app sharing platform where people can create apps on various Python web frameworks and deploy using simple Git functions. You can also view featured Spaces and experience state-of-the-art ML models in action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SXPU3x6enEMUmmc2" /><figcaption>Image from <a href="https://huggingface.co/spaces">Spaces — Hugging Face</a></figcaption></figure><p>Before we start with deployment, we need to first create a <a href="https://huggingface.co/new-space">new Space</a> then we need to add the name of the Space, license, and select SDK. After that, we can clone Space or add a remote to the current Git repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/0*n8buZtX7wTbNW9BH" /><figcaption>Image by author</figcaption></figure><h4>Setting Up DVC</h4><p>Integrating DVC is easy with Spaces. We just need to run a shell script within the main Python file. The code below only pulls a model and a few samples from the train and test dataset. In the end, it removes the <strong>.dvc</strong> folder to optimize the storage. Make sure you are adding this code before the model inference function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5136dbba35159ff720f622ce18dbe2b6/href">https://medium.com/media/5136dbba35159ff720f622ce18dbe2b6/href</a></iframe><h4>Custom Environment</h4><p>For deploying the Gradio app to Spaces, we need to make a few changes to avoid errors and dependency issues.</p><p><strong>Adding Hugging Face Remote</strong></p><p>First, we need to add our Space remote to the current project. The Space remote address should look like</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/998a785a401ba8623314fc9f2e711826/href">https://medium.com/media/998a785a401ba8623314fc9f2e711826/href</a></iframe><p><strong>README.md</strong></p><p>Then go to your <strong>README.md</strong> file and add the metadata in the form of <strong>yml</strong>. This metadata will tell Space the location of the app file, emoji for the cover, color gradient for the app thumbnail, SDK, and license information. You can customize the color and emoji to make your thumbnail stand out.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd660daf571a5a45ba47c83359e4afe1/href">https://medium.com/media/cd660daf571a5a45ba47c83359e4afe1/href</a></iframe><p><strong>requirements.txt</strong></p><p>We are limited to CPU, and to optimize the storage, we will use the PyTorch CPU version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/0*Cgp83bH332rV5cun" /><figcaption>Image from <a href="https://pytorch.org/get-started/locally/">Get Started | PyTorch</a></figcaption></figure><p>We will only include packages to <em>requirements.txt</em> file that are necessary for running model inference and web interface.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44e4129b38ef9f0b2f70718a541211c8/href">https://medium.com/media/44e4129b38ef9f0b2f70718a541211c8/href</a></iframe><h4>Finalizing</h4><p>After making all changes, it’s time to commit and push the code to the Space remote server. The initial remote Space has README.md and .gitattributes, so to avoid conflicts, we will use the<strong> -f</strong> flag. We will send all the files from the local (master branch) to the remote(main branch) server using <strong>master:main</strong>.</p><blockquote><strong>Warning:</strong> Please use <strong>-f</strong> flag only once at the start, avoid using it as it will override other people’s work.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b8588ab246f65922c8fdb20c30c9bfe/href">https://medium.com/media/2b8588ab246f65922c8fdb20c30c9bfe/href</a></iframe><p>After pushing the code, you will see a “<strong>Building” </strong>sign on your app. It will take approximately 3 minutes to build.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/0*hJenmPUQfquFUTy7" /><figcaption>Image by author</figcaption></figure><p>Congratulations, your app is successfully deployed and it can be shared among friends and colleagues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*2oxRPgSNNajXX7BH" /><figcaption>Image by author | <a href="https://huggingface.co/spaces/kingabzpro/savtadepth">Hugging Face Space</a></figcaption></figure><h3>Bonus</h3><p>The bonus part is for MLOps enthusiasts who are always eager to learn more ways to integrate new tools and databases. In this section, we are going to integrate the Hugging Face <a href="https://huggingface.co/datasets">Dataset</a> to collect all the flags. The flags will include input image, output image, and CSV file. The flag option helps us track our model performance and we can later use this dataset to improve model performance.</p><p>We need to add the code below to the <em>savta_app.py</em> file for the integration to work. It will use <em>HuggingFaceDatasetSaver</em> to create and update flag dataset. The function requires two parameters <em>HF_TOKEN</em> that you can find in <a href="https://huggingface.co/settings/tokens">settings</a> and the name of the dataset. Finally, you can add the object to <strong>flagging_callback</strong> in the Gradio Interface function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/36743442502c601c4de677d35088af46/href">https://medium.com/media/36743442502c601c4de677d35088af46/href</a></iframe><p>You also need to set the <em>HF_TOKEN</em> environment variable in Space settings. Just copy and paste the <a href="https://huggingface.co/settings/tokens">user access token</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/0*ZppVNG9I0BIP6BYN" /><figcaption>Image by author</figcaption></figure><p>After that, go to the app and flag the few predictions so that you can see the flagged dataset under your profile. In our case, it’s <a href="https://huggingface.co/datasets/kingabzpro/savtadepth-flags">savtadepth-flags</a> that can be accessed publicly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/0*AWYswgYK_-RUflrS" /><figcaption>Image from <a href="https://huggingface.co/datasets/kingabzpro/savtadepth-flags">savtadepth-flags</a></figcaption></figure><h3>Conclusion</h3><p>Hugging Face Space provides an easy-to-deploy platform where data scientists can test and share machine learning models in the form of a user-friendly interface. If you are a beginner to machine learning and want to experience end-to-end product development, then try to develop your app on Gradio, integrate it with DagsHub, and finally deploy it to Spaces. It will also help you create a strong portfolio where you can mention the experience of deploying machine learning models.</p><p>In this tutorial, we have learned how to use Git, DVC, DagsHub, Gradio, and Hugging Face Space to create and deploy complex machine learning applications. It is your first step toward becoming an MLOps engineer.</p><p><strong>Project Resources:</strong></p><ul><li><a href="https://huggingface.co/spaces/kingabzpro/savtadepth">SavtaDepth WebApp</a></li><li><a href="https://huggingface.co/datasets/kingabzpro/savtadepth-flags">Flagged Database</a></li><li><a href="https://dagshub.com/OperationSavta/SavtaDepth">SavtaDepth Project</a></li><li><a href="https://dagshub.com/kingabzpro/SavtaDepth/src/dagshub">SavtaDepth Fork + WebApp</a></li><li><a href="https://colab.research.google.com/drive/1XU4DgQ217_hUMU1dllppeQNw3pTRlHy1?usp=sharing">Google Colab Demo</a></li><li><a href="https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html">Project Dataset</a></li></ul><p><strong>Reference to dataset:</strong> Nathan Silberman, P. K., Derek Hoiem en Fergus, R. (2012) “Indoor Segmentation and Support Inference from RGBD Images”, in <em>ECCV</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a42664abcc14" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deploying-gradio-app-on-spaces-using-dagshub-a-beginners-tutorial-a42664abcc14">Deploying Gradio App on Spaces Using DagsHub: A Beginner’s Tutorial</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tips & Tricks of Deploying Deep Learning Webapp on Heroku Cloud]]></title>
            <link>https://medium.com/data-science/tips-tricks-of-deploying-deep-learning-webapp-on-heroku-cloud-80ea9063211a?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/80ea9063211a</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Mon, 10 Jan 2022 15:55:05 GMT</pubDate>
            <atom:updated>2022-01-10T15:55:05.891Z</atom:updated>
            <content:encoded><![CDATA[<h4>Things I learned by deploying Tensorflow based image classifier Streamlit app on Heroku server</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z3fH05duIMwB-HjcJem-QQ.png" /><figcaption>Image by author</figcaption></figure><p><a href="https://www.heroku.com/home">Heroku</a> is a famous platform among web developers and machine learning enthusiasts. The platform provides a easy ways to deploy and maintain the web application, but if you are not familiar with deploying deep learning applications, you might struggle with storage and dependence issues.</p><p>This guide will make your deployment process smother so that you can focus on creating amazing web applications. We will be learning about DVC integration, Git &amp; CLI-based deployment, error code H10, playing around with python packages, and optimizing storage.</p><h3>Git &amp; CLI-based Deployment</h3><p>The Streamlit app can be deployed by either Git based, GitHub integration or using Docker. The Git based is by far faster and easier way to deploy any data app on Heroku server.</p><h4><strong>Simple Git-based</strong></h4><p>The streamlit app can be deployed using:</p><pre>git remote add heroku https://heroku:$HEROKU_API_KEY@git.heroku.com/&lt;name of your heroku app&gt;.git<br><br>git push -f heroku HEAD:master</pre><p><strong>For this to work you need:</strong></p><ul><li>Heroku API Key</li><li>Heroku App: either by CLI or using website.</li><li>Git based project</li><li>Procfile</li><li>Requirements.txt</li></ul><h4>CLI-based</h4><p>CLI based deployment is basic and easy to learn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3GCNZ_SMxFXBqTG617zH9A.png" /><figcaption>Image by Author</figcaption></figure><ol><li>Create a <strong>free</strong> Heroku account <a href="https://signup.heroku.com/">here</a>.</li><li>Install Heroku CLI using this <a href="https://devcenter.heroku.com/articles/heroku-cli#download-and-install">link</a>.</li><li>Either clone remote repository or use git init</li><li>Type heroku login and heroku create dagshub-pc-app. This will log you into the server and create an app on a web server.</li><li>Now create <strong>Procfile</strong> containing the commands to run the app: web: streamlit run --server.port $PORT streamlit_app.py</li><li>Finally, commit and push code to heroku server git push heroku master</li></ol><h3>PORT</h3><p>If you are running the app with streamlit run app.py it will produce an error code <strong>H10 </strong>which means <strong>$PORT </strong>assigned by the server was not used by the streamlit app.</p><p><strong>You need to:</strong></p><ul><li>Set PORT by using Heroku CLI</li></ul><pre>heroku config:set PORT=8080</pre><ul><li>Make changes in your <strong>Procfile </strong>and add server port in arguments<strong>.</strong></li></ul><pre>web: streamlit run --server.port $PORT app.py</pre><h3>Tweaking Python Packages</h3><p>This part took me two days to debug as Heroku cloud comes with a <strong>500MB</strong> limitation and the new <a href="https://pypi.org/project/tensorflow/#files">TensorFlow</a> package is <strong>489.6MB</strong>. To avoid dependencies and storage issues we need to make changes in the<strong> requirements.txt</strong> file:</p><ol><li>Add <strong>tensorflow-cpu</strong> instead of tensorflow which will reduce the slug size from 765MB to 400MB.</li><li>Add <strong>opencv-python-headless</strong> instead of opencv-python to avoid installing external dependencies. This will resolve all the cv2 errors.</li><li>Remove all unnecessary packages except <strong>numpy, Pillow </strong>and<strong> streamlit</strong>.</li></ol><h3>DVC Integration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Km6EEXwm0XDMOKC-E8Mhtw.png" /><figcaption>Image by Author</figcaption></figure><p>There are a few steps required for successfully pulling data from the DVC server.</p><ol><li>First, we will install a <a href="https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-apt">buildpack</a> that will allow the installation of apt-files by using Heroku APIheroku buildpacks:<strong>add</strong> --<strong>index</strong> 1 heroku-community/apt</li><li>Create a file name <strong>Aptfile </strong>and add the latest DVC version <a href="https://github.com/iterative/dvc/releases/download/2.8.3/dvc_2.8.3_amd64.deb">https://github.com/iterative/dvc/releases/download/2.8.3/dvc_2.8.3_amd64.deb</a></li><li>In your <strong>app.py</strong> file add extra lines of code to pull data from remote DVC server:</li></ol><pre><strong>import</strong> os<br><br><strong>if</strong> &quot;DYNO&quot; <strong>in</strong> os.environ <strong>and</strong> os.path.isdir(&quot;.dvc&quot;):<br>    os.system(&quot;dvc config core.no_scm true&quot;)<br>    <strong>if</strong> os.system(f&quot;dvc pull&quot;) != 0:<br>        exit(&quot;dvc pull failed&quot;)<br>    os.system(&quot;rm -r .dvc .apt/usr/lib/dvc&quot;)</pre><p>After that, commit and push your code to Heroku server. Upon successful deployment, the app will automatically pull the data from DVC server.</p><h3>Optimizing Storage</h3><p>There are multiple ways to optimize storage and the most common is to use Docker. By using the docker you can bypass the 500MB limit, you also have the freedom to install any third-party integration or packages. To learn more about how to use docker check out this <a href="https://www.r-bloggers.com/2020/12/creating-a-streamlit-web-app-building-with-docker-github-actions-and-hosting-on-heroku/">guide</a>.</p><p><strong>For optimizing storage:</strong></p><ul><li>Only add model inference python libraries in requiremnets.txt</li><li>We can pull selective data from DVC by using dvc pull {model} {sample_data1} {sample_data2}..</li><li>We only need model inference file so add rest of files to.slugignore which works similarly to .gitignore, to learn more check out <a href="https://devcenter.heroku.com/articles/slug-compiler#ignoring-files-with-slugignore">Slug Compiler</a>.</li><li>Remove .dvc and .apt/usr/lib/dvc directory after successfully pulling the data from server.</li></ul><h3>Outcomes</h3><p>The initial slug size was 850MB but with storage and package optimizations the final slug size was reduced to 400MB. We have solved error code H10 with a simple command and added opencv-python-headless package to solve dependency issues. This guide was created to overcome some of the common problems faced by beginners on Heroku servers.</p><p>The <a href="https://devcenter.heroku.com/categories/deploying-with-docker">Docker-based</a> deployment can solve a lot of storage problems but it comes with complexity and a slow deployment process. You can use heroku container:push web but before that, you need to build the docker, test it, and resolve all the issues locally before you can push it. This method is preferred by advance Heroku users.</p><p>The next challenge is to deploy your web application with webhook. This will allow you to automate the entire machine learning ecosystem from any platform. The automation process will involve creating a simple Flask web server that will run shell commands.</p><h3>Additional Resources</h3><ul><li><a href="https://www.r-bloggers.com/2020/12/creating-a-streamlit-web-app-building-with-docker-github-actions-and-hosting-on-heroku/">Creating a Streamlit web app, building with Docker + GitHub Actions, and hosting on Heroku | R-bloggers</a></li><li><a href="https://towardsdatascience.com/deploying-an-image-classification-web-app-with-python-3753c46bb79">Deploying an Image Classification Web App with Python | by Ruben Winastwan | Towards Data Science</a></li><li><a href="https://github.com/GuilhermeBrejeiro/deploy_ML_model_Heroku_FastAPI">GuilhermeBrejeiro/deploy_ML_model_Heroku_FastAPI</a></li><li><a href="https://devcenter.heroku.com/articles/heroku-cli#getting-started">The Heroku CLI | Heroku Dev Center</a></li></ul><p><em>Originally published at </em><a href="https://www.kdnuggets.com/2021/12/tips-tricks-deploying-dl-webapps-heroku.html"><em>K</em>Dnuggets</a><em> on December 24, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80ea9063211a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/tips-tricks-of-deploying-deep-learning-webapp-on-heroku-cloud-80ea9063211a">Tips &amp; Tricks of Deploying Deep Learning Webapp on Heroku Cloud</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Ace Data Science Interview by Working on Portfolio Projects.]]></title>
            <link>https://pub.towardsai.net/how-to-ace-data-science-interview-by-working-on-portfolio-projects-6e5027430565?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e5027430565</guid>
            <category><![CDATA[portfolio]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Mon, 27 Dec 2021 12:02:42 GMT</pubDate>
            <atom:updated>2021-12-27T12:02:42.052Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/careers">Careers</a></h4><h4>Learn how maintaining GitHub, Deepnote, and Kaggle profiles helped me ace multiple data science interviews.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cde6z9M7RLqQRFKU" /><figcaption>Photo by <a href="https://unsplash.com/@mkvandergriff?utm_source=medium&amp;utm_medium=referral">Maranda Vandergriff</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Recruiters are checking your online presence before contacting you about an interview. They will look for your LinkedIn profile, GitHub, and Kaggle to figure out what value will you bring to their company. The hiring manager will also look for the latest blogs or projects to prepare interview questions so that they can test your intelligence — <a href="https://catherinescareercorner.com/2021/09/22/explore-6-sure-ways-to-ace-your-data-science-interview/">catherinescareercorner</a>. Other than that, working on real-world projects will also give you the required experience for the job, and adding projects to your portfolio will make a good impression on the recruiter’s <a href="https://data-flair.training/blogs/data-science-portfolio/">data-flair</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUyMrp53ckb7TOkP7no8tw.jpeg" /><figcaption>Image by Author</figcaption></figure><p>We will be learning new ways to crack your interviews by creating a strong portfolio that will help clear the technical interview stage. I will give you tips on improving your current projects and how recruiters go through your profile. The article is based on my personal experience and the things I learn from career coaches.</p><blockquote>After reading this article you will improve your chance of passing the interview stage.</blockquote><figure><a href="https://www.kdnuggets.com/2021/10/ace-data-science-interview-portfolio-projects.html"><img alt="" src="https://cdn-images-1.medium.com/max/96/0*3jrZgkxC6fh2yNqe.png" /></a><figcaption><a href="https://www.kdnuggets.com/2021/10/ace-data-science-interview-portfolio-projects.html">KDnuggets Silver Award</a></figcaption></figure><h3>Interview # 1</h3><p>A recruiter contacted me after going through my Kaggle <a href="https://www.kaggle.com/kingabzpro">profile</a> and told me how he was impressed by my recent competition performance. He asked me if he will love to talk more on a call. To be honest, I was surprised by how a financial organization this big contacted me through Kaggle and how contributing to competition landed me an interview with a principal data scientist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/233/1*Ir6SQMu74VVSAKrvXmDnyw.png" /><figcaption>Image by Author | <a href="https://www.kaggle.com/kingabzpro">Abid Ali Awan | Expert | Kaggle</a></figcaption></figure><blockquote><strong><em>Insight<br></em></strong>Participating in the Kaggle competition can land you a job.</blockquote><p>During the interview, we have talked about my previous projects and my previous competitions. I explained to him that how I am actively participating in other competitive platforms such as <a href="https://zindi.africa/users/kingabzpro/competitions">Zindi</a>. He asked me about the project I was currently working on and what are my future goals.</p><blockquote><strong><em>Insight<br></em></strong>Creating portfolio projects will defend your skills on your resume.</blockquote><p>At the end of the interview, he was quite impressed with my current performance and told me that I was the best possible candidate for this job.</p><h3>Interview # 2</h3><p>This interview was quite different as the data scientist who took my interview focused more on my GitHub profile rather than my resume. After this interview, I realized how much recruiters are interested in a portfolio rather than a LinkedIn profile or resume.</p><p>So, after applying for the job on LinkedIn, I received an email regarding the interview and during the interview, we talked about my GitHub projects. They even asked me what were the problems I faced during the project and how I solved that problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*ABbXGikqNk4fslDp48loRg.png" /><figcaption>Image by Author | <a href="https://github.com/kingabzpro">kingabzpro (Abid)</a></figcaption></figure><blockquote><strong><em>Insight<br></em></strong>Even data scientists maintain GitHub profiles.</blockquote><p>After the interview, I went back to see my profile and noticed that the interviewer was focusing on the latest projects in my repository with descriptions. He also talked about my pinned projects, and I was about to answer every question because I knew every part from start till the end of these projects.</p><blockquote><strong><em>Insight<br></em></strong>Always pinned/highlight the projects you have worked on 100%.</blockquote><p>Overall, the interview went smoother than I thought, and I received exceptional feedback from the recruiters as they were comparing me with Harvard Graduates.</p><h3>Interview #3</h3><p>The third interview was quite surprising as I was interviewed by five people from different departments, and they all talked about my Deepnote <a href="https://deepnote.com/@abid">profile</a> directly or indirectly. This interview was the reason I begin to draft this article so that other people can take advantage of my experience.</p><p>After applying for a job, I received a call from the top telecommunication company in my country. My first interview was taken by three experts in their field. They asked me about my past work and my goals. After the initial introduction, we dove into detailed discussions about my projects and Deepnote profile. The expert data scientist asked me key questions about NLP and how I was using them in my projects. They went through top projects and asked me about how I solved certain problems.</p><blockquote><strong><em>Insight<br></em></strong>Publish your best work in form of notebooks with detailed descriptions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*RobPVTnFMhjRcYXrLP9dFA.png" /><figcaption>Image by Author | <a href="https://deepnote.com/@abid">Abid</a></figcaption></figure><p>My second interview was conducted by higher management who were interested in my mindset and how I can deal with certain data. They asked me about feature engineering and finally one of the managers asked me about my Deepnote profile bio. He was interested in my vision of building an application that will help students struggling with mental illness. In the end, he asked me how working in the company can help you reach your goal.</p><blockquote><strong><em>Insight<br></em></strong>The biography should be personal, it should describe your life goals.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*2rLQpwGQ2fs45zleloaczA.png" /><figcaption>Image by Author</figcaption></figure><p>Both of my interviews were excellent in terms of feedback, and the company&#39;s higher management was impressed with my answers about my future goals. I have explained how working with them will help me grow and achieve my goals in long run.</p><h3>Tips</h3><ol><li><strong>Tidy up your profile: </strong>If you have an interview related to NLP, make irrelevant repo and notebook private, so that the interviewer can focus on the projects related to the field.</li><li><strong>Project Descriptions: </strong>make sure the first line they read is about the project summary. Writing a description about the GitHub repo or Deepnote notebook will help the recruiter comes up with related questions.</li><li><strong>Revise your Project:</strong> You should go through the project description on GitHub or Deepnote so that during the interview you can answer in detail about what this project was about and how you can overcome some of the critical issues.</li><li><strong>Keep your Portfolio Up to date: </strong>This will make an impression on the recruiter that you are actively contributing to the project. This will also help you improve your skills, with a unique problem comes the unique learning experience.</li><li><strong>Focus on profile bio:</strong> Write about your job experience and your current project. Writing about your goals will help you get noticed. Make sure that it’s your personal story and you care about your vision.</li><li><strong>Participate in the Kaggle competition:</strong> This will improve your chance to get noticed and this will make you an attractive candidate for a job.</li><li><strong>Revises Data Science cheatsheet:</strong> Even though you know everything about your project, you might get stuck in technical knowledge such as topic modeling and how tf-idf works.</li></ol><h3>Author</h3><p><a href="https://www.polywork.com/kingabzpro"><strong>Abid Ali Awan</strong></a> (<a href="https://twitter.com/1abidaliawan">@1abidaliawan</a>) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master’s degree in Technology Management and a bachelor’s degree in Telecommunication Engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.</p><blockquote><strong>You can follow me on </strong><a href="https://www.linkedin.com/in/1abidaliawan/"><strong>LinkedIn</strong></a>, <a href="https://ko-fi.com/kingabzpro"><strong>Ko-fi</strong></a><strong> and </strong><a href="https://www.polywork.com/kingabzpro"><strong>Polywork</strong></a><strong> where I post my article weekly.</strong></blockquote><p>Original blog is published on <a href="https://www.kdnuggets.com/2021/10/ace-data-science-interview-portfolio-projects.html">KDnuggets</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e5027430565" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-to-ace-data-science-interview-by-working-on-portfolio-projects-6e5027430565">How to Ace Data Science Interview by Working on Portfolio Projects.</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Learning During Covid19: A Complete Analysis]]></title>
            <link>https://pub.towardsai.net/digital-learning-during-covid19-a-complete-analysis-45c525b1904?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/45c525b1904</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[kaggle-competition]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[covid19]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Tue, 21 Dec 2021 01:17:24 GMT</pubDate>
            <atom:updated>2021-12-21T01:17:24.427Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/data-analysis">Data Analysis</a></h4><h4>Using data analysis tools to figure out trends in digital learning during Covid19 and how the online platforms are effective towards improvised communities.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P0Vcx1vDatMT5oaQ" /><figcaption>Photo by <a href="https://unsplash.com/@comparefibre?utm_source=medium&amp;utm_medium=referral">Compare Fibre</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>There is an imbalance in the education system during the Covid19 pandemic and most of the students don’t even have access to educational tools and online learning platforms. There is an urgent need to come up with solutions and by using <a href="https://learnplatform.com/">LearnPlatform</a> dataset we will figure some of the common patterns and identify the clusters based on demography, geography, and accessibility.</p><h3>Project goals</h3><p>In this project, we will be using data analysis tools to figure out trends in digital learning and how it is effective towards improvised communities. We will be comparing districts and states on factors like demography, internet access, learning product access, and finance. In the end, we will summarize our report and point towards the areas that need our more attention to make education accessible for all students the United States.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*gPu6KvJO8MemhqAUf1U7hA.jpeg" /><figcaption>Image by Author</figcaption></figure><h3>Data</h3><p>The Dataset Is available at <a href="https://www.kaggle.com/c/learnplatform-covid19-impact-on-digital-learning">Kaggle</a> under noncommercial license.</p><p><strong>Use this command to download the dataset faster</strong></p><pre>!pip install kaggle<br>!kaggle competitions download -c learnplatform-covid19-impact-on-digital-learning</pre><blockquote><em>Don&#39;t forget to add Kaggle API key 👇</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*631tiOGh-l-wE2TB47zfrw.jpeg" /><figcaption>Image by Author</figcaption></figure><p>We have three types of Dataset.</p><ol><li>Products data contains Service names, Companies, and educational sectors.</li><li>District data contains Demography, Locations, and Educational Spending.</li><li>Engagement data contains students’ engagement with different products per day.</li></ol><h3>Product</h3><p>The product file products_info.csv includes information about the characteristics of the top 372 products with most users in 2020. The categories listed in this file are part of Learn Platform&#39;s product taxonomy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*LuS1bmzbSxOp7kDcPwLAKg.png" /><figcaption>Image by Author</figcaption></figure><h3>Loading data educational product data</h3><p>As we can see we have product names, Companies, and Sector data with Primary Functions.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00010-55858daf-b7e2-4226-bc72-b7133b02cce9%3Fheight%3D536&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00010-55858daf-b7e2-4226-bc72-b7133b02cce9%3Fheight%3D536&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="536" frameborder="0" scrolling="no"><a href="https://medium.com/media/623a354965270b308b1e95ec04764357/href">https://medium.com/media/623a354965270b308b1e95ec04764357/href</a></iframe><h3>District</h3><p>The district file districts_info.csv includes information about the characteristics of school districts, including data from</p><ul><li>NCES (2018–19),</li><li>FCC (Dec 2018), and</li><li>Edunomics Lab.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*qK5J2dmXb5rq5PIqSHryNA.png" /><figcaption>Image by Authors</figcaption></figure><h3>Loading data district data</h3><p>We have different states, locale, educational spending, and more importantly we have demographical data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00013-c02e5170-8839-474f-b181-4a91aaea7f8b%3Fheight%3D422&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00013-c02e5170-8839-474f-b181-4a91aaea7f8b%3Fheight%3D422&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="422" frameborder="0" scrolling="no"><a href="https://medium.com/media/414e917bff8245387bb26f6db7a05f04/href">https://medium.com/media/414e917bff8245387bb26f6db7a05f04/href</a></iframe><h3>Engagement data</h3><p>The engagement data are aggregated at the school district level and each file in the folder engagement_data represents data from one school district.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*dIkTBNNuNToMjPqeuzMEdQ.png" /><figcaption>Image by Author</figcaption></figure><h4>Engagement data extraction function</h4><p>Our engagement data is divided into multiple .csv files based on <em>District id</em> so we need to create a Python function to extract data from the individual files and concat them into one.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00016-49afe3c5-23b7-477f-be50-136bd477ae05%3Fheight%3D386&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00016-49afe3c5-23b7-477f-be50-136bd477ae05%3Fheight%3D386&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="386" frameborder="0" scrolling="no"><a href="https://medium.com/media/22086b9f887715354f95dcb5db6e107e/href">https://medium.com/media/22086b9f887715354f95dcb5db6e107e/href</a></iframe><h3>Loading data engagement data</h3><p>It took us 19 seconds to load all files from a folder and now we have DataFrame containing all distract Engagement Data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00018-4e8d0426-e80c-4b45-8e0b-ba635eeb8014%3Fheight%3D489&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00018-4e8d0426-e80c-4b45-8e0b-ba635eeb8014%3Fheight%3D489&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="489" frameborder="0" scrolling="no"><a href="https://medium.com/media/119440f0df7111ec157e97d8e04af348/href">https://medium.com/media/119440f0df7111ec157e97d8e04af348/href</a></iframe><h3>Missing data</h3><p>Let’s look at all three DataFrames for missing values per Column</p><h4>Plot missing function</h4><p>The function helps us plot an Altair bar chart for all data frames containing missing values.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00021-302e989c-f8b8-4e13-8f88-a67898ce1904%3Fheight%3D224&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00021-302e989c-f8b8-4e13-8f88-a67898ce1904%3Fheight%3D224&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="224" frameborder="0" scrolling="no"><a href="https://medium.com/media/31113eb18d1712d614c045df719e7d80/href">https://medium.com/media/31113eb18d1712d614c045df719e7d80/href</a></iframe><p>As we can observe that our district dataset mostly contains missing values in pp_total_raw,pct_free/reduced,county_connections_ratio. Let&#39;s see all the missing values together in the next section.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00023-e63df821-9b96-43c1-9eb0-55b8e4a2bd37%3Fheight%3D421&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00023-e63df821-9b96-43c1-9eb0-55b8e4a2bd37%3Fheight%3D421&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="421" frameborder="0" scrolling="no"><a href="https://medium.com/media/2bd211fd01b8e353041a48c82b69fbf2/href">https://medium.com/media/2bd211fd01b8e353041a48c82b69fbf2/href</a></iframe><p>The subplot below shows missing values in all three datasets we will be removing them in the next section as it is hard to replace them without actual information.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00025-ca4a0cb7-0a8a-42d5-bd4d-fd1c50878f10%3Fheight%3D743&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00025-ca4a0cb7-0a8a-42d5-bd4d-fd1c50878f10%3Fheight%3D743&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="743" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8d623611f9ffe3f9d03574fda10c8df/href">https://medium.com/media/b8d623611f9ffe3f9d03574fda10c8df/href</a></iframe><h3>Cleaning data</h3><p>We will be cleaning district data as it contains an array of values. We will be taking advantage of both limits.</p><ol><li>Converting list pct_black/hispanic and pct_free/reduced into float</li><li>Converting list pp_total_raw into Integers.</li><li>Dropping county_connections_ratio due to static 0.18 to 1 values across all categories.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00027-fcffe7b0-bf95-4f44-b084-fb1718450e93%3Fheight%3D513&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00027-fcffe7b0-bf95-4f44-b084-fb1718450e93%3Fheight%3D513&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="513" frameborder="0" scrolling="no"><a href="https://medium.com/media/d086dbd945716459e32e69422e2f443e/href">https://medium.com/media/d086dbd945716459e32e69422e2f443e/href</a></iframe><h3>Feature distributions visualization</h3><p>In this section, we will discover different features distributions.</p><h4>Districts per states</h4><p>As we can see the dataset contains more districts from Utah and Illinois than any other state.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00030-5a82fac9-6b58-4708-87e6-429ca7cc4f56%3Fheight%3D776&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00030-5a82fac9-6b58-4708-87e6-429ca7cc4f56%3Fheight%3D776&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="776" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d6c219840a895a8654b4aa55e937e81/href">https://medium.com/media/1d6c219840a895a8654b4aa55e937e81/href</a></iframe><h3>Locale distribution</h3><p>The suburbs are dominating with 59 percentage, which also means people from medium to high class have more access to internet.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00032-868e7df9-352e-4042-9571-335e4c66ff55%3Fheight%3D874&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00032-868e7df9-352e-4042-9571-335e4c66ff55%3Fheight%3D874&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="874" frameborder="0" scrolling="no"><a href="https://medium.com/media/f86d357763fc4ad0092139d9c327c7ca/href">https://medium.com/media/f86d357763fc4ad0092139d9c327c7ca/href</a></iframe><h3>Educational product providers</h3><p>Google provides more educational products than anyone close to it.</p><blockquote><em>In short online education is dominated by Google LLC</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00034-20ae6cae-7b1b-4dc3-80d3-1d06ae597c06%3Fheight%3D812&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00034-20ae6cae-7b1b-4dc3-80d3-1d06ae597c06%3Fheight%3D812&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="812" frameborder="0" scrolling="no"><a href="https://medium.com/media/186a03c64fa3325a2313cbc9b6e20de0/href">https://medium.com/media/186a03c64fa3325a2313cbc9b6e20de0/href</a></iframe><p>Let’s see what are the products provided by Google</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00036-4901e7ae-3e84-458b-baf1-98fcd096abdd%3Fheight%3D754&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00036-4901e7ae-3e84-458b-baf1-98fcd096abdd%3Fheight%3D754&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="754" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c64da2fa281d2c1050271f507e8ac49/href">https://medium.com/media/2c64da2fa281d2c1050271f507e8ac49/href</a></iframe><h3>Educational sector distributions</h3><p>Educational sectors are divided into three categories, PreK-12, Higher Education, and Corporate.</p><blockquote><em>Some products are specific to a sector but others are quite general.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00038-fe68ff0a-d98c-4098-9f88-3c85dc45a143%3Fheight%3D309&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00038-fe68ff0a-d98c-4098-9f88-3c85dc45a143%3Fheight%3D309&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="309" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a40f41bf51421065e4766c7ff59d9c8/href">https://medium.com/media/3a40f41bf51421065e4766c7ff59d9c8/href</a></iframe><p>As we can see PreK education is dominating the distribution with 54 percentage.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00040-5a39e261-a419-492b-90a2-ed33a9f0b86d%3Fheight%3D1072&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00040-5a39e261-a419-492b-90a2-ed33a9f0b86d%3Fheight%3D1072&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="1072" frameborder="0" scrolling="no"><a href="https://medium.com/media/c452273727de8ca358b1b9ed36f568f2/href">https://medium.com/media/c452273727de8ca358b1b9ed36f568f2/href</a></iframe><h3>Primary functions with main and subcategories</h3><p>There are three types of main primary functions.</p><ol><li>LC = Learning &amp; Curriculum</li><li>CM = Classroom Management</li><li>SDO = School &amp; District Operations.</li></ol><p>You can interact with the Plotly sunburst plot to explore the distribution of main categories and subcategories.</p><blockquote><em>The Learning products have majority shares in this group and the most common subcategories are digital learning platforms.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00042-c3fb90b6-6e24-4a8b-a56d-44c45e02e16b%3Fheight%3D1181&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00042-c3fb90b6-6e24-4a8b-a56d-44c45e02e16b%3Fheight%3D1181&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="1181" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b24707236429c2f67fb77115c81c689/href">https://medium.com/media/4b24707236429c2f67fb77115c81c689/href</a></iframe><h3>Essential function subcategories distribution</h3><p>We can observe all the subcategories distribution.</p><blockquote><em>Content creation and digital learning are leading in this market.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00044-9484c1ac-ba4b-440d-a4d2-af75c20cafa9%3Fheight%3D821&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00044-9484c1ac-ba4b-440d-a4d2-af75c20cafa9%3Fheight%3D821&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="821" frameborder="0" scrolling="no"><a href="https://medium.com/media/d86cf83fa2c1a896e287a2af6c600d95/href">https://medium.com/media/d86cf83fa2c1a896e287a2af6c600d95/href</a></iframe><h3>Merging three datasets</h3><p>We will be merging all three datasets on LP ID and district id. As we can see the combined dataset has 18 columns that will later help us with both geographical and time series plots.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00047-433aa19c-bd56-401b-938f-1d2b322c7753%3Fheight%3D332&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00047-433aa19c-bd56-401b-938f-1d2b322c7753%3Fheight%3D332&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="332" frameborder="0" scrolling="no"><a href="https://medium.com/media/87fb8c2ded501bcf146519686ae77aee/href">https://medium.com/media/87fb8c2ded501bcf146519686ae77aee/href</a></iframe><h3>Top educational products</h3><p>Google products are dominating with Wikipedia and Netflix as an exception. Netflix provides kids with educational content and tutorials.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00050-9835df7d-f01f-428d-96f2-77cbcfea31d9%3Fheight%3D752&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00050-9835df7d-f01f-428d-96f2-77cbcfea31d9%3Fheight%3D752&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="752" frameborder="0" scrolling="no"><a href="https://medium.com/media/280848f80b801eed80c6f31539421ef3/href">https://medium.com/media/280848f80b801eed80c6f31539421ef3/href</a></iframe><h3>Distribution of race, reduced fee, expenditures, and Internet connection per state</h3><p>We will be used the Pandas function to display a table showing the mean distribution of various features related to demography and expenditures.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00052-5621402b-bb10-4a6f-b75d-1f8ebc1b6efc%3Fheight%3D206&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00052-5621402b-bb10-4a6f-b75d-1f8ebc1b6efc%3Fheight%3D206&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="206" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0d60fd7c67515a215e885d1052cc5d8/href">https://medium.com/media/a0d60fd7c67515a215e885d1052cc5d8/href</a></iframe><h3>Black and Hispanic</h3><p>I seems like Taxes have more Black/Hispanic students than another state followed by Florida, Michigan, and Minnesota.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00054-f40e8aff-ab95-4e64-b3f4-e19b11a99d13%3Fheight%3D552&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00054-f40e8aff-ab95-4e64-b3f4-e19b11a99d13%3Fheight%3D552&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="552" frameborder="0" scrolling="no"><a href="https://medium.com/media/af8ca1ae65ab4812c26e050d703b1365/href">https://medium.com/media/af8ca1ae65ab4812c26e050d703b1365/href</a></iframe><h3>Reduced fee or free education</h3><p>Minnesota provides 70 percent of free or reduced fee education, followed by Michigan and Indiana.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00056-d98332bb-9f99-445b-89fa-9f189a8b781b%3Fheight%3D552&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00056-d98332bb-9f99-445b-89fa-9f189a8b781b%3Fheight%3D552&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="552" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a43122787f01f65c227ee52bad65834/href">https://medium.com/media/1a43122787f01f65c227ee52bad65834/href</a></iframe><h3>Per-pupil total expenditure</h3><p>New York spends more on education than any other State in US. The runner-up States are New Jersey and Minnesota.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00058-dea8af17-bd61-48dd-9644-a4707d50dcf9%3Fheight%3D552&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00058-dea8af17-bd61-48dd-9644-a4707d50dcf9%3Fheight%3D552&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="552" frameborder="0" scrolling="no"><a href="https://medium.com/media/1acac8c3215c8f3ee3a1a961b14de394/href">https://medium.com/media/1acac8c3215c8f3ee3a1a961b14de394/href</a></iframe><h3>Time series distribution of educational product access students</h3><p>We will be observing time-series distributions of Product Access Based on Locale and State.</p><h4>Based on Locale</h4><p>The cities were hit with Covid19 the hardest which affected the students the most as they have to study from home that is why we can see the dip in product access from April to July 2020 as compared to other locales. During Summer vacation students stopped using educational products. In suburbs, students have more access to the tools than another locale. overall, they all follow a common pattern.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00061-fdd63f25-388d-4f64-a64b-1ec1b043880c%3Fheight%3D821&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00061-fdd63f25-388d-4f64-a64b-1ec1b043880c%3Fheight%3D821&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="821" frameborder="0" scrolling="no"><a href="https://medium.com/media/954454722281ee1ce62fa28baedaa74a/href">https://medium.com/media/954454722281ee1ce62fa28baedaa74a/href</a></iframe><h3>Based on the top five states</h3><p>New York was hit hardest with Covid19 and that didn’t affect the online learning of students as you can see graph became more consistent after the lockdown.</p><blockquote><em>New York, Wisconsin, and Indian have higher students using these educational products overall.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00063-8dad9e7c-31bc-4a97-8f58-be4425b2f290%3Fheight%3D929&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00063-8dad9e7c-31bc-4a97-8f58-be4425b2f290%3Fheight%3D929&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="929" frameborder="0" scrolling="no"><a href="https://medium.com/media/d48ebfa4b9850b791fb33f90c469d2c0/href">https://medium.com/media/d48ebfa4b9850b791fb33f90c469d2c0/href</a></iframe><h3>Based on the bottom five states</h3><p>This is odd as Michigan and North Carolina have almost zero product access from March till the end of August. Texas has a similar pattern but they had a peak in June. Overall these bottom states became active after September 2020.</p><blockquote><em>It is strange that some states have a very odd pattern, maybe due to a bad educational system or lack of awareness.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00065-e792932e-ff16-434b-82d4-4e4dd65c721f%3Fheight%3D857&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00065-e792932e-ff16-434b-82d4-4e4dd65c721f%3Fheight%3D857&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="857" frameborder="0" scrolling="no"><a href="https://medium.com/media/67158ffdb6621d958964515b8d159183/href">https://medium.com/media/67158ffdb6621d958964515b8d159183/href</a></iframe><h3>Geographical analysis</h3><p>This section will be looking at different states based on product access and engagement index.</p><blockquote><em>We do not have access to all states data so you might see a lot of gray space on the map.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00067-eea42c95-2d75-4644-a873-b81989558c38%3Fheight%3D495&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00067-eea42c95-2d75-4644-a873-b81989558c38%3Fheight%3D495&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="495" frameborder="0" scrolling="no"><a href="https://medium.com/media/d90457e4183fcb6719bf4a61e44fed4c/href">https://medium.com/media/d90457e4183fcb6719bf4a61e44fed4c/href</a></iframe><h3>Education products access per state</h3><p>New York has a higher product access score than any other state, then comes the Orange States Wisconsin, Illinois, and Indiana. There might be a link between the lower population of Black/Hispanics in these states. Let’s explore that in the next part.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00069-f88e9a49-937d-4409-b4d7-724a1e40bf19%3Fheight%3D601&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00069-f88e9a49-937d-4409-b4d7-724a1e40bf19%3Fheight%3D601&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="601" frameborder="0" scrolling="no"><a href="https://medium.com/media/70d88c60b48876990b18093bc94730ee/href">https://medium.com/media/70d88c60b48876990b18093bc94730ee/href</a></iframe><p>There is a relation between states with a lower population of Black/Hispanic but it is not the stronger. I can see Newyork and Wisconsin with a lower Black/Hispanic population and higher access rate but when it comes to other states with a lower percentage of people of color it’s all over the place. The system is fairly balanced towards other races.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00071-0c57092d-22ab-4e0c-aa4d-28abc4ea76ef%3Fheight%3D764&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00071-0c57092d-22ab-4e0c-aa4d-28abc4ea76ef%3Fheight%3D764&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="764" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8abf906768eea90ded6034b3eb7733a/href">https://medium.com/media/a8abf906768eea90ded6034b3eb7733a/href</a></iframe><h3>Engagement index per state</h3><p>The Engagement distribution per state is quite similar to product access. It’s logical that if students have access to these products there is a chance of an increase in usage of these platforms.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00073-444e3dfc-a8d3-4d10-88fe-49a4afaf7248%3Fheight%3D459&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00073-444e3dfc-a8d3-4d10-88fe-49a4afaf7248%3Fheight%3D459&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="459" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ae98492d0dbda6598cabbdaab000687/href">https://medium.com/media/2ae98492d0dbda6598cabbdaab000687/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00074-f5a1198b-5f0e-4f47-acdc-06855595dfa3%3Fheight%3D893&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00074-f5a1198b-5f0e-4f47-acdc-06855595dfa3%3Fheight%3D893&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="893" frameborder="0" scrolling="no"><a href="https://medium.com/media/67b58a0b858996b79ca846b6a7dfd860/href">https://medium.com/media/67b58a0b858996b79ca846b6a7dfd860/href</a></iframe><h3>Focusing on people from the improvised neighborhood</h3><p>In this section, we will be focusing on the Black/Hispanic community and free or reduced fees.</p><h4>Correlation</h4><p>There is a high correlation between product access and engagement index, which means if the student has access to these products they might use them on daily basis. We can also see another high correlation between Black/Hispanic and Free/ Reduced education. This means the government is doing its job in helping improvised communities to get a proper education.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00077-06daec26-31de-4274-85eb-edf1817acc05%3Fheight%3D544&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00077-06daec26-31de-4274-85eb-edf1817acc05%3Fheight%3D544&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="544" frameborder="0" scrolling="no"><a href="https://medium.com/media/1bf262cd52cb4bcbca2f04884760e415/href">https://medium.com/media/1bf262cd52cb4bcbca2f04884760e415/href</a></iframe><h3>Black/Hispanic products access over the year.</h3><p>The graph below shows that over time the communities with a lesser population of Blacks and Hispanics have a similar pattern to some of the Top states we have discussed earlier. Let’s compare this graph with the mixed communities.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00079-c9dac158-b261-48c3-81f8-1dd123208f7f%3Fheight%3D875&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00079-c9dac158-b261-48c3-81f8-1dd123208f7f%3Fheight%3D875&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="875" frameborder="0" scrolling="no"><a href="https://medium.com/media/666c68a6fb487b44633df3aba658643b/href">https://medium.com/media/666c68a6fb487b44633df3aba658643b/href</a></iframe><p>We can see some changes as the peak has fallen from 1.4 to 1.2. It is a disparity among the different racial groups but its marginal compares to other countries. We can say Black/Hispanic communities have lesser access to educational products.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00081-9e3a62ff-8b90-484e-9baa-8ed005195f57%3Fheight%3D875&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2F10b483d4-cd30-4b17-8915-b61c2790ec6c%2F5966a34d-462e-40e1-b858-4e4d77afe684%2F00081-9e3a62ff-8b90-484e-9baa-8ed005195f57%3Fheight%3D875&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnail-updated.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="875" frameborder="0" scrolling="no"><a href="https://medium.com/media/4abad269845631c8fcd6439fde9575d9/href">https://medium.com/media/4abad269845631c8fcd6439fde9575d9/href</a></iframe><h3>Code</h3><ul><li><a href="https://deepnote.com/@abid/Digital-Learning-During-COVID19-EDA-ELSD1M0wSxeJFbYcJ5DsbA">Digital Learning During COVID19 EDA (deepnote.com)</a></li><li><a href="https://www.kaggle.com/kingabzpro/evolution-of-digital-learning-during-covid19">Evolution of Digital Learning During COVID19 | Kaggle</a></li><li><a href="https://github.com/kingabzpro/Digital-Learning-During-COVID19-EDA">kingabzpro/Digital-Learning-During-COVID19-EDA(github.com)</a></li></ul><h3>Conclusion</h3><p>The data is limited to fewer states and I didn’t find any concrete evidence that Black or Hispanic communities get unfair treatment. Online education is fairly balanced towards all. I did see how some of the states perform worst in terms of product access rating, this might be due to bad policymaking and lack of awareness. We can see that Google is dominating the online education industry by providing a complete ecosystem. The majority of Pre-K students are using these platforms for digital learning. Suburbs have the highest students accessing these products and we can see a clear correlation between product access and engagement index. There is also a high correlation between Black/Hispanic and Free education/ Reduced Fee, which means the government is doing its best to help the poor community by providing free education.</p><blockquote><strong>You can follow me on </strong><a href="https://www.linkedin.com/in/1abidaliawan/"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://twitter.com/1abidaliawan"><strong>Twitter</strong></a><strong>, and </strong><a href="https://www.polywork.com/kingabzpro"><strong>Polywork</strong></a><strong> where I post my article on weekly basis.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45c525b1904" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/digital-learning-during-covid19-a-complete-analysis-45c525b1904">Digital Learning During Covid19: A Complete Analysis</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Discord bot using DailoGPT and HuggingFace API.]]></title>
            <link>https://medium.com/geekculture/discord-bot-using-dailogpt-and-huggingface-api-c71983422701?source=rss-ebd29761458a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c71983422701</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[rick-and-morty]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[discord]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Abid Ali Awan]]></dc:creator>
            <pubDate>Mon, 20 Dec 2021 18:02:41 GMT</pubDate>
            <atom:updated>2025-04-26T07:58:57.337Z</atom:updated>
            <content:encoded><![CDATA[<h4>Design your own Rick bot using Hugging face API and deploy it on Replit server.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9x1EmiAyie6zVAr2" /><figcaption>Photo by <a href="https://unsplash.com/@michael_marais?utm_source=medium&amp;utm_medium=referral">Michael Marais</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>If you ever wonder how Discord bot works and how you can create your own bot that speaks like a certain celebrity or certain character from your favorite cartoon shows that you are at the best place to start. We will be learning how to use <strong>HuggingFace API</strong> and use it as a Discord bot. We will also learn about Replit, Kaggle CLI, and <strong>uptimerobot</strong> to keep your bot running. We won’t be going deep into fine-tuning our <strong>DailoGPT</strong> model for that you can check out this <a href="https://towardsdatascience.com/make-your-own-rick-sanchez-bot-with-transformers-and-dialogpt-fine-tuning-f85e6d1f4e30">blog</a>.</p><h3>Dataset</h3><p>we will use a <strong>Kaggle CLI </strong>to download the <a href="https://www.kaggle.com/andradaolteanu/rickmorty-scripts">Rick&amp;Morty Scripts | Kaggle</a> that is publicly avaible. If you want to learn how to get your Kaggle API and how to use Kaggle CLI please check you this this <a href="https://github.com/Kaggle/kaggle-api">link</a>.</p><pre>!kaggle datasets download andradaolteanu/rickmorty-scripts -f RickAndMortyScripts.csv</pre><p>As you can see, we have columns: <em>index, season no., episode no, episode name, name of the character, and line</em>. We will be focusing on name and line column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*uoEVrC8hWPsZh4Fy5U_W5A.png" /></figure><p>We will convert this dataset that every response row will contain <strong>n</strong> previous responses as a context.</p><pre>CHARACTER_NAME = &#39;Rick&#39;</pre><pre>contexted = []<br><br><em># context window of size 7</em><br>n = 7<br><br><strong>for</strong> i <strong>in</strong> data[data.name == CHARACTER_NAME].index:<br>  <strong>if</strong> i &lt; n:<br>    <strong>continue</strong><br>  row = []<br>  prev = i - 1 - n <em># we additionally substract 1, so row will contain current responce and 7 previous responces  </em><br>  <strong>for</strong> j <strong>in</strong> range(i, prev, -1):<br>    row.append(data.line[j])<br>  contexted.append(row)<br><br>columns = [&#39;response&#39;, &#39;context&#39;] <br>columns = columns + [&#39;context/&#39; + str(i) <strong>for</strong> i <strong>in</strong> range(n - 1)]<br><br>df = pd.DataFrame.from_records(contexted, columns=columns)</pre><p>As you can see below, we have response columns and 6 context columns. This will help us fine tune our generative model. After this we will download model from Hugging face and start train our model. You can check full code <a href="https://deepnote.com/viewer/github/kingabzpro/DailoGPT-RickBot/blob/main/Training%20Model.ipynb">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gtdQzFhtORNMobb1KMn5EA.png" /></figure><h3>Deploy to Hugging Face</h3><p>After fine-tuning your model you can simply deploy your model to hugging face cloud but before that, you need to create a HF (<a href="https://huggingface.co/">Hugging Face</a>) account and create the model as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*GjZ8yHM-MhoC4GpqzEHTUw.png" /></figure><p>You will need HF API to upload your mode and you can get it from settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DLW0B0qSEIED1fF3ucg0Yg.png" /></figure><ul><li>Install git-lfs: git extension to upload larger files</li><li>Configure git email and user mail</li><li>Use your HF API key and model named to upload both tokenizer and model.</li></ul><pre>!sudo apt-get install git-lfs</pre><pre>Reading package lists... Done<br>Building dependency tree       <br>Reading state information... Done<br>git-lfs is already the newest version (2.3.4-1).<br>The following package was automatically installed and is no longer required:<br>  libnvidia-common-460<br>Use &#39;sudo apt autoremove&#39; to remove it.<br>0 upgraded, 0 newly installed, 0 to remove and 40 not upgraded.</pre><pre>!git config --<strong>global</strong> user.email &quot;X@X.com&quot;<br><em># Tip: using the same email as your huggingface.co account will link your commits to your profile</em><br>!git config --<strong>global</strong> user.name &quot;kingabzpro&quot;</pre><pre>MY_MODEL_NAME = &#39;DialoGPT-small-Rick-Bot&#39;<br><strong>with</strong> open(&#39;/content/HuggingFace_API.txt&#39;, &#39;rt&#39;) <strong>as</strong> f:<br>  HUGGINGFACE_API_KEY = f.read().strip()</pre><pre>model.push_to_hub(MY_MODEL_NAME, use_auth_token=HUGGINGFACE_API_KEY)<br>tokenizer.push_to_hub(MY_MODEL_NAME, use_auth_token=HUGGINGFACE_API_KEY)</pre><p>You can add tags such as conversational, gpt2 etc by editing your model card. You can check out my model <a href="https://huggingface.co/kingabzpro/DialoGPT-small-Rick-Bot">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7i1wrQo-7WN6WucQxBCIw.png" /><figcaption>Image by Author</figcaption></figure><h3>Testing your model</h3><p>After deploying let’s test it in python script using transformers.</p><ul><li>download both tokenizer and model using transformer.</li><li>run loop for 4 inputs</li><li>user input is used to generate text</li></ul><pre>tokenizer = AutoTokenizer.from_pretrained(&#39;kingabzpro/DialoGPT-small-Rick-Bot&#39;)<br>model = AutoModelWithLMHead.from_pretrained(&#39;kingabzpro/DialoGPT-small-Rick-Bot&#39;)<br><em># Let&#39;s chat for 4 lines</em><br><strong>for</strong> step <strong>in</strong> range(4):<br>    <em># encode the new user input, add the eos_token and return a tensor in Pytorch</em><br>    new_user_input_ids = tokenizer.encode(input(&quot;&gt;&gt; User:&quot;) + tokenizer.eos_token, return_tensors=&#39;pt&#39;)<br>    <em># print(new_user_input_ids)</em><br><br>    <em># append the new user input tokens to the chat history</em><br>    bot_input_ids = torch.cat([chat_history_ids, new_user_input_ids], dim=-1) <strong>if</strong> step &gt; 0 <strong>else</strong> new_user_input_ids<br><br>    <em># generated a response while limiting the total chat history to 1000 tokens, </em><br>    chat_history_ids = model.generate(<br>        bot_input_ids, max_length=200,<br>        pad_token_id=tokenizer.eos_token_id,  <br>        no_repeat_ngram_size=3,       <br>        do_sample=True, <br>        top_k=100, <br>        top_p=0.7,<br>        temperature=0.8<br>    )<br>    <br>    <em># pretty print last ouput tokens from bot</em><br>    print(&quot;RickBot: {}&quot;.format(tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True)))</pre><p>As you can see our model is working perfectly. For every user input, there is a response by Rick.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*j5hK5XyIPwwd1AT6uFa1ag.png" /></figure><h3>Discord bot</h3><p>In order to create the <a href="https://discord.com/developers/applications">Discord</a> bot, first, you need to get into portal <a href="https://discord.com/developers/applications">Discord Developer Portal</a>. Click on New Application and get started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vKaW8Gqf0Xi6HKlpdzeueQ.png" /></figure><p>Write the application name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fbqDVDUSChtcTidOc_QRHQ.png" /></figure><p>Now click on the Bot tab and then Add Bot. It will promote you to a new window that will ask you to write the bot name and add an image to the bot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eoym_8CdsnAJeU0by9mhLQ.png" /></figure><p>Click on OAuth2 tab and check mark bot and bot permission: Send Message. Later copy the link and paste it into a new tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pjyVytXvn2hQeSwkigW3uA.png" /></figure><p>After pasting the link in the new tab, it will ask you to select the server, and that’s it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*suwu9EOr2l1u4yf-ULc3Eg.png" /></figure><h3>Replit</h3><p>You can create your account on <a href="https://replit.com/">https://replit.com/</a>. We will be using this platform as a bot server for our Discord bot. After creating an account just create a new python repl or import my GitHub <a href="https://github.com/kingabzpro/DailoGPT-RickBot">repo</a>. This platform is beginner-friendly and it won’t take more than 5 minutes to understand how to use the editor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AyGEvHxqSe-r3Hu5W7_O5w.png" /></figure><h3>Testing API</h3><p>Before we need to use replit secret feature to add our HF API and Discord API as show below. Your Discord API key is in developer portal: Bot tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*JRpaG6EXf52ICcCGMNIVmw.png" /></figure><ul><li>create test python file to test HF API</li><li>create function that use requests to send and receive information using model id and HF API token.</li><li>Print generated response.</li></ul><p>The function is working perfectly now we need to add this function into bot.py.</p><pre>import requests</pre><pre>def query(payload, model_id, api_token):</pre><pre> headers = {&quot;Authorization&quot;: f&quot;Bearer {api_token}&quot;}</pre><pre> API_URL = f&quot;https://api-inference.huggingface.co/models/{model_id}&quot;</pre><pre> response = requests.post(API_URL, headers=headers, json=payload)</pre><pre> return response.json()<br></pre><pre>model_id = &quot;kingabzpro/DialoGPT-small-Rick-Bot&quot;</pre><pre>api_token = &quot;api_XXXXXXXXXXXXXXXXXXXXXX&quot; # get yours at hf.co/settings/token</pre><pre>data = query(&#39;What you think about mom&#39;, model_id, api_token)</pre><pre>print(data[&#39;generated_text&#39;])</pre><h3>Uptime for Replit</h3><p>We will be using <a href="https://uptimerobot.com/">https://uptimerobot.com</a> to keep our bot running forever. As you can see below that we can Add Monitor and add the URL that we will be creating using flask. It will ping our bot app after 5 minutes which will trick Replit server into running forever.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YEhV3ROlqVmEJV9XoLYZ5g.png" /></figure><h4>Flask WebApp</h4><p>The code below is a simple Flash web app which will display “I am Alive” while running. The web app will generate the link in Replit, something like this: <a href="https://dailogpt-rickbot.kingabzpro.repl.co/">https://DailoGPT-RickBot.kingabzpro.repl.co</a>. You are going to paste this link into uptimerobot which will trigger this app after 5 minutes.</p><pre>from flask import Flask</pre><pre>from threading import Thread</pre><pre>app = Flask(&#39;&#39;)<br></pre><pre>@app.route(&#39;/&#39;)</pre><pre>def home():</pre><pre> return &quot;I am Alive&quot;</pre><pre><br>def run():</pre><pre> app.run(host=&#39;0.0.0.0&#39;,port=8080)</pre><pre>def keep_alive():</pre><pre> t=Thread(target=run)</pre><pre> t.start()</pre><h3>Bot Code</h3><p>This code contains the backbone of our Discord bot and integration of HF API and Flask web app.</p><ul><li>We are using the keep_alive function from the web_app file that we just create.</li><li>Using discord python client.</li><li>query function sends and receives a response from HF API, which is protected by the API key.</li><li>on_ready warms up HF API by sending a “Hello” message. It takes almost 20 seconds to warm up the server.</li><li>on_message checks if the message is from bot or user, then it takes user message and sends it to HF API and then displays the response on discord chatbot. Extra error checks or not responding serve are added to make sure we can debug easily.</li></ul><p>To learn more, you can look at the code below and read the comments.</p><pre>import os</pre><pre># these modules are for querying the Hugging Face model<br>import json<br>import requests<br>from web_app import keep_alive</pre><pre># the Discord Python API<br>import discord</pre><pre># this is my Hugging Face profile link<br>API_URL = ‘<a href="https://api-inference.huggingface.co/models/&#39;">https://api-inference.huggingface.co/models/&#39;</a></pre><pre>class MyClient(discord.Client):<br> def __init__(self, model_name):<br> super().__init__()<br> self.api_endpoint = API_URL + model_name<br> # retrieve the secret API token from the system environment<br> huggingface_token = os.environ[‘HUGGINGFACE_TOKEN’]<br> # format the header in our request to Hugging Face<br> self.request_headers = {<br> ‘Authorization’: ‘Bearer {}’.format(huggingface_token)<br> }</pre><pre>def query(self, payload):<br> “””<br> make request to the Hugging Face model API<br> “””<br> data = json.dumps(payload)<br> response = requests.post(<br> self.api_endpoint,<br> headers=self.request_headers,<br> data=data)<br> ret = json.loads(response.content.decode(‘utf-8’))<br> return ret</pre><pre>async def on_ready(self):<br> # print out information when the bot wakes up<br> print(‘Logged in as’)<br> print(self.user.name)<br> print(self.user.id)<br> print(‘ — — — ‘)<br> # send a request to the model without caring about the response<br> # just so that the model wakes up and starts loading<br> self.query({‘inputs’: {‘text’: ‘Hello!’}})</pre><pre>async def on_message(self, message):<br> “””<br> this function is called whenever the bot sees a message in a channel<br> “””<br> # ignore the message if it comes from the bot itself<br> if message.author.id == self.user.id:<br>  return</pre><pre> # form query payload with the content of the message<br> payload = message.content</pre><pre> # while the bot is waiting on a response from the model<br> # set the its status as typing for user-friendliness<br> async with message.channel.typing():<br> response = self.query(payload)<br> bot_response = response[‘generated_text’]<br> <br> # we may get ill-formed response if the model hasn’t fully loaded<br> # or has timed out<br> if not bot_response:<br>  if ‘error’ in response:<br>   bot_response = ‘`Error: {}`’.format(response[‘error’])<br>  else:<br>   bot_response = ‘Hmm… something is not right.’</pre><pre> # send the model’s response to the Discord channel<br>  await message.channel.send(bot_response)</pre><pre>def main():<br> # DialoGPT<br> client = MyClient(‘kingabzpro/DialoGPT-small-Rick-Bot’)<br> keep_alive()<br> client.run(os.environ[‘DISCORD_TOKEN’])</pre><pre>if __name__ == ‘__main__’:<br> main()</pre><p>As you can see our bot serve is running perfectly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UhojmlesEWrxKV8KprPUtw.png" /><figcaption>Image by Author</figcaption></figure><p>Now let&#39;s check our discord server. As you can see RickBot is online.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*aN2zL6AdgUWESBJ2P3vpgg.png" /><figcaption>Image by Author</figcaption></figure><p>Have fun playing around with the bot you trained and deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/0*OUll_sFXnClN6sps.png" /><figcaption>Image by Author</figcaption></figure><p>You can also just import my repository on Replit and add your HF API and Discord API key to get started with in a few minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*OXkWc3Aj0b7-o6c1v7UlMA.png" /><figcaption>Image by Author</figcaption></figure><h3>Code</h3><p>You can find code on <a href="https://github.com/kingabzpro/DailoGPT-RickBot">GitHub</a> and <a href="https://dagshub.com/kingabzpro/DailoGPT-RickBot">DAGsHub</a>. You can also check my HuggingFace model <a href="https://huggingface.co/kingabzpro/DialoGPT-small-Rick-Bot">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P6YJtRhkWjuNqQ4CIt0_lw.png" /><figcaption>Image by Author</figcaption></figure><h3>Conclusion</h3><p>My experience of learning from video tutorials and training my own bot was amazing. It got even better when I started playing around with the discord library and server. In this guide, we have learned Hugging Face API, Transformers, Discord server, uptime robot, Kaggle API, and Replit platform. I can’t wait to see what character you chose next and share your experience with me.</p><blockquote>Machine learning is simple done make it hard, always keep looking out for hacks and tips to improve your workspace.</blockquote><h3>Reference</h3><ul><li>The code is highly influenced by freeCodeCamp tutorial: <a href="https://www.youtube.com/watch?v=UjDpW_SOrlw">Code a Discord Chat Bot That Talks Like Your Favorite Character</a></li><li>The finetuning model is from <a href="https://towardsdatascience.com/make-your-own-rick-sanchez-bot-with-transformers-and-dialogpt-fine-tuning-f85e6d1f4e30">medium blog</a>.</li><li>The Kaggle data set is from <a href="https://www.kaggle.com/andradaolteanu/rickmorty-scripts">Rick&amp;Morty Scripts</a>.</li></ul><h3>About Author</h3><p><a href="https://www.polywork.com/kingabzpro">Abid Ali Awan</a> (<a href="https://twitter.com/1abidaliawan">@1abidaliawan</a>) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master’s degree in Technology Management and a bachelor’s degree in Telecommunication Engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2021/11/learn-how-to-build-your-discord-bot/"><em>https://www.analyticsvidhya.com</em></a><em> on November 1, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c71983422701" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/discord-bot-using-dailogpt-and-huggingface-api-c71983422701">Discord bot using DailoGPT and HuggingFace API.</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>