<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nirmal Budhathoki on Medium]]></title>
        <description><![CDATA[Stories by Nirmal Budhathoki on Medium]]></description>
        <link>https://medium.com/@nirmal.budhathoki?source=rss-500fd3d011d3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*x3FNh1qcAxBVSwdv-QP8ug.jpeg</url>
            <title>Stories by Nirmal Budhathoki on Medium</title>
            <link>https://medium.com/@nirmal.budhathoki?source=rss-500fd3d011d3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:38:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nirmal.budhathoki/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Data Science Roles vary across the Tech Companies]]></title>
            <link>https://medium.com/@nirmal.budhathoki/how-data-science-roles-vary-across-the-tech-companies-cef1893a56a5?source=rss-500fd3d011d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/cef1893a56a5</guid>
            <category><![CDATA[data-science-job]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science-interview]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Nirmal Budhathoki]]></dc:creator>
            <pubDate>Thu, 24 Aug 2023 17:23:59 GMT</pubDate>
            <atom:updated>2023-08-24T17:48:49.133Z</atom:updated>
            <content:encoded><![CDATA[<p>More than a decade ago, an article was published in Harvard Business Review (HBR) titled as “<a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century"><strong>Data Scientist: The Sexiest Job of the 21st Century</strong></a>” by DJ Patil and Thomas Davenport. At the time, the title Data Scientist was defined as “a high-ranking professional with the training and curiosity to make discoveries in the world of big data.”</p><p>Fast forward today, the Data Scientist title has evolved in so many directions that it has become one of the most misused and confusing job titles in the data industry. The interview process for Data Science is the messiest of all time. One of the main reasons for not having a clear, concrete or streamlined interview process is the confusion around the job titles vs. responsibilities expected from the role.</p><p>In this article, I will try to unbox what the Data Scientist as a job title means in the top four tech companies- <strong>Amazon, Google, Microsoft and Meta (</strong>formerly known as<strong> Facebook</strong>). I will also highlight the top skills needed for those roles.</p><blockquote>I have interviewed myself with all these companies, therefore I am writing this to help others.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*INHpFlXph39dcYzCxeFfbw.jpeg" /><figcaption>What does Data Scientist Title means in these tech companies ?</figcaption></figure><h3>Data Science Jobs in Google:</h3><p>If you are going for the Data Scientist title, most of them are actually the re-branded titles that used to be called Quantitative Analysts in Google. Most of the data scientist roles are aligned to work with product teams, and it is more into the analytics side than the core ML (Machine Learning) side.</p><p>Google has another job title called ML Engineer, which is more aligned with ML focus, and they fall under SWE (Software Engineering). Most of their data science titles have the product team listed in the job title. For example- Data Scientist, Core Compute or Data Scientist, gTech etc.</p><p>Top skills asked are SQL, Python, Stats and Data Analysis skills. If you are more interested in the core ML work (building and deploying ML models), you want to consider ML Engineer titles instead.</p><h3>Data Science Jobs in Amazon:</h3><p>The data scientist titles in amazon are a little bit more well defined and structured. I want to give some credit to Amazon for that. For the analytics side, Amazon has other job titles like Data Analysts, Business Analysts and Business Intelligence Engineers. For the core ML, they have Applied Scientist titles. Therefore, the Data Scientist is the perfect balance between the above titles who will have analytics as well as an engineering mindset to build models. Top skills asked are Python, Coding (Data Structures &amp; Algorithms, meeting the SWE level-1 bar), SQL, Data Visualization, Machine Learning concepts (more breadth than depth) and how to build the models. Deployment of models can be additional or preferred skills for data scientists, but for the Applied Scientists both build and deploy skills are required.</p><h3>Data Science jobs in Meta:</h3><p>The data scientist job in Meta is similar to Google, which means they are also embedded with product teams rather than engineering teams. Most of the DS (Data Science) roles in Meta are analytics heavy, working along with product teams to help with data analysis, defining, measuring and monitoring KPIs (Key Performance Indicators). Meta also has ML engineer roles which are focused more in the core ML.</p><p>Top skills asked for DS roles in Meta are SQL, Python, Stats, and Data Visualization. One additional skill they usually demand is experimental design (A/B testing). Therefore the DS interviews in Meta always include one analytical case study problem.</p><p>I created the diagram below to distinguish between the two major tracks for <strong><em>Data Science </em></strong>roles- one is Analytics side and the other is Core ML side. When you read the job description- if you see keywords are focusing on data analysis and SQL, statistical modeling, A/B testing, hypothesis testing, experimentation, visualization- then the DS role is analytics based. On the other hand, if you see the keywords are more focused on asking for python, big data tools like spark, machine learning frameworks like sklearn, deep learning frameworks like Pytorch, and Tensorflow, skills for doing ML in cloud, then the DS role is core-ML based. Therefore, regardless of the title, it is important to read and understand the job description before you make a decision to apply.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qH8Ghu52Yo-61gWZfkUp-g.png" /><figcaption>At a high level- Data Science Splits into Analytics and Core ML</figcaption></figure><h3>Data Science jobs in Microsoft:</h3><p>One of the most confusing job titles when it comes to Data Scientist is from Microsoft. Being a part of Microsoft myself, I am not so proud on how they have defined the DS roles. Microsoft has the high-level job title to cover most of the data roles under the same title called ‘Data &amp; Applied Scientist’. The good thing is their DS roles are mostly aligned with engineering, so the parity in pay is more comparable to SWE roles. You have to rely on job descriptions to read well and carefully to conclude if the role is more on the analytics side like Data Analyst, BI engineer, or is it more ML focused like ML engineer or Applied Scientist.</p><p>Top skills asked are Python, SQL, Stats, Big Data query, and Data Visualization. If the role is focused towards the research or deep learning side- they also ask DL (Deep Learning) frameworks like PyTorch, Keras , Tensorflow etc.</p><p><em>Lastly,</em> I would like to wish you all the best if you are currently applying or interviewing for DS roles in any of these companies or others. If you’ve enjoyed reading this article, please subscribe for more contents. You can also follow me on <a href="https://www.linkedin.com/in/nirmal-budhathoki"><strong>linkedin</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cef1893a56a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some of the Best Practices to follow in a Data Science Project]]></title>
            <link>https://pub.aimind.so/some-of-the-best-practices-to-follow-in-a-data-science-project-6c6923f6ad4a?source=rss-500fd3d011d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c6923f6ad4a</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nirmal Budhathoki]]></dc:creator>
            <pubDate>Wed, 23 Aug 2023 06:25:13 GMT</pubDate>
            <atom:updated>2023-08-23T20:43:26.721Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I will talk about some of the best and recommended practices to follow when it comes to a data science project. Before I present the best practices, it is important to understand the general workflow of a data science project. There are many variations to it, but I always prefer to use the traditional (yet the most effective) workflow known as <strong>CRISP-DM</strong>, which stands for <strong>CR</strong>oss <strong>I</strong>ndustry <strong>S</strong>tandard <strong>P</strong>rocess for <strong>D</strong>ata <strong>M</strong>ining.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*dF5w7J1z8PTN2iKB.png" /><figcaption>CRISP-DM framework for a Data Science Project</figcaption></figure><p>The intention of this article is not to explain the <a href="https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining">CRISP-DM</a> process. However, the best practices discussed here are inspired by this process, therefore I will briefly touch upon this framework.</p><p>One of the important things to notice on the CRISP-DM process is the feedback loop. Data preparation to modeling stage has an iterative feedback loop- the bidirectional arrows signify that the feature engineering is an iterative process. The model evaluation stage connects to business understanding before deployment. This is very important that we evaluate and re-evaluate our models, making sure it answers all the right business questions before we deploy the model. Another factor I want to highlight is the data, which is the center of this framework. It signifies that — Everything revolves around the data.</p><p>Now that we have set enough context, below are the five best practices that I recommend you to follow if you are working in a data science or a machine learning project.</p><h4>1. Business understanding or problem statement:</h4><p>Make sure you spend enough time understanding the business questions you are trying to solve with data science. Spending enough time in the early stage of problem formulation will save you time in future stages. As a data scientist, you have to work closely with different functional areas of business, stakeholders, and product managers to understand the business problem from the customer’s perspective.</p><h4>2. Data is the secret sauce of your model:</h4><p>If you have heard the good old saying- ‘Garbage In, Garbage Out’, this statement is very true when it comes to the data you are using for training your model. The quality of data is important. About 80% of the data scientist’s time is spent on data preparation. The model part is important, but the success of a model is highly dependent on the quality of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*CLHr15GDs0T79bEzoufyHw.jpeg" /><figcaption>Garbage In Garbage Out- Data is the driving force behind the ML model</figcaption></figure><h4>3. Communication with the subject matter or domain experts:</h4><p>One of the common mistakes from data scientists is not talking to domain experts frequently enough, during the feature engineering and model building stages. Feature engineering is one of the crucial steps of a data science project. This is the stage where you are deriving or creating features for the ML model. Domain experts are your best friends when it comes to gaining additional insights on what features to create. For example, if I am building a ML model to detect credit card fraud, communicating with the fraud analyst will help me collect some insights on what information they are looking for in the data to make the decision on a fraudulent transaction.</p><h4>4. Start with proof of concept, build MVP and iterate over it:</h4><p>Always start small, and iterate over the simple model, rather than trying to build everything at once. Once you identify some of the key data sources, connect with the data owners to gather a small set of sample data, explore the data, understand the data, and build a simple MVP (minimum viable product). There is a popular saying in the IT world- <em>If at First You Don’t Succeed Call It Version 1.0. </em>This is a very true statement when it comes to data science projects.</p><h4>5. Measure what matters:</h4><p>The model evaluation stage (as shown in above diagram) is very critical because it actually decides whether the business question or the problem statement we started with is answered by the model. Before we can answer if we met the success criteria or not, it is important to define that success criterion first. This stage also decides if the model should be deployed into production, therefore measuring the metrics that matter is very important.</p><p>If we consider the same example of fraud detection model, it is important to decide on the threshold that separates the two classes- fraud vs. not fraud. If we only rely on default PR (precision and recall) metrics, the cutoff threshold is 0.5, however, it is important to decide on what to optimize for.</p><p>If the business thinks having more false positives (detected fraud, but in reality it is not) are more expensive than having false negatives (not able to detect the fraud), then we shall increase the threshold to improve precision. In the contrary, if the business thinks having false negatives are more costly, then we lower the threshold to improve recall. In this example, high false positive rate leads to poor customer experience, while high false negative rate will lead to losing money on the missed fraud cases. Therefore, business has to be careful on making the decision on the cutoff.</p><p><strong>In summary,</strong> if we follow some of the simple standards and practices in a data science project, it can tremendously help us improve our success rate. In future, I will be posting more articles on similar topics. Stay tuned !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c6923f6ad4a" width="1" height="1" alt=""><hr><p><a href="https://pub.aimind.so/some-of-the-best-practices-to-follow-in-a-data-science-project-6c6923f6ad4a">Some of the Best Practices to follow in a Data Science Project</a> was originally published in <a href="https://pub.aimind.so/">AI Mind</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Bias vs. Model Bias in Machine Learning]]></title>
            <link>https://pub.aimind.so/data-bias-vs-model-bias-in-machine-learning-75d62400a89b?source=rss-500fd3d011d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/75d62400a89b</guid>
            <category><![CDATA[data-bias]]></category>
            <category><![CDATA[bias-variance-tradeoff]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nirmal Budhathoki]]></dc:creator>
            <pubDate>Wed, 02 Aug 2023 05:15:45 GMT</pubDate>
            <atom:updated>2023-08-07T15:59:41.842Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Bias and Model Bias are two different things. Both of them can affect the performance of machine learning (ML) models.</p><p>Before we jump into comparing these two biases, first let&#39;s try to understand- what is bias?</p><blockquote>Bias refers to a conscious or unconscious preference towards certain things- it could be certain group of people or events, certain assumptions we make, certain choices we made or certain beliefs we have.</blockquote><p><strong>Data Bias</strong> occurs when the data used for training the model is not the representative samples of the actual population. In other words, the training data is not quite representing the real-world data.</p><p><strong>Model Bias </strong>occurs when the model itself is not able to accurately represent the underlying relationship between the input features and the output variable. In other words, the model is too simple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X4Uw8EH8kKitzRK7vKUNVg.png" /><figcaption>Bias can come in many shapes and sizes.</figcaption></figure><p>We have to understand the difference between these two biases, so we can mitigate each of them. For data bias, it&#39;s important to be aware of common human biases or the biases infused by data collection and sampling process. For model bias, it&#39;s important to choose a right model or algorithm, and tune its hyperparameters to get the optimal results.</p><blockquote>Keep In Mind: The most popular discussion in machine learning regarding the bias vs. variance tradeoff, is mostly talking about the model bias.</blockquote><p>Among these two biases, the first one to understand and tackle would be data bias, because it is already present in the data before we start modeling.</p><p>Example story of data bias: Back in 2015, Amazon realized that its AI enabled hiring tool was not rating candidates for software developer jobs in a gender-neutral way. Upon investigating, the researchers found that their model was trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most of the applications were representing men, which shows a reflection of male dominance across the tech industry. In effect, the model believed that male candidates were preferable.</p><p>As we see in this example, one of the most common types of data bias happens when some variable occurs more frequently than others. Therefore, it is important to analyze the distribution and imbalance in data- not only from the target label’s perspective but also from input attributes.</p><p>One of the common mistakes we make while we are doing correlation analysis is that we are always seeking for input variables that are highly correlated to the target variable. The logic is true- In machine learning, we have learned that the input variables should be independent of each other, but the target variable should be dependent on input variables for high predictive power. However, we have to be very cautious if those input variables are sensitive features like gender, age, location etc.</p><p>I remember when I was working in Wells Fargo Bank, the model risk and governance team were not allowing us to use zip code as a feature for our loan approval model. The reason is to avoid data bias- as they have noticed some zip codes have more approved applications than others. This could potentially create data bias making model to believe applications coming from certain zip codes are preferable. When we looked into the data more carefully, we realized that the location feature was just a casual effect from the status of people living there- whether people with bad credit or people with good credit.</p><p>One more thing to keep in mind- when we impute missing values, we have to be aware if we are infusing biased data. For example, the mean imputation can often skew the distribution.</p><p>Some of the other examples of data bias are reporting bias, selection bias, automation bias, group attribution bias, and implicit bias. The most challenging one in my opinion is the implicit bias because we cannot control other’s belief or stereotype while collecting inputs from them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YwB1fkMm49wjej-jTDBcZw.jpeg" /><figcaption>Implicit Bias is rooted in our thought processes.</figcaption></figure><p>Now let&#39;s come to the model bias. As we defined it earlier- model bias happens when we fail to find that optimal relationship between input features and output variable. The model error can be broken down as:</p><blockquote>Model Error = Variance + Bias² + Noise</blockquote><p>Bias represents the inherent error that we get from our model regardless of training data size. Our model is “biased” to a particular type of solution. For example, we are using simple linear classifier, when there exists non-linear relationship between the inputs and output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*yj722Y7GlzqEcog_L8bEew.png" /><figcaption>Bias- Variance Errors Against Model Complexity</figcaption></figure><p>Model having a high bias is also referred to as under-fitting. It is diagnosed by looking and comparing the training and validation error. If both of them are lower than the acceptable error, then we have high bias in our model.</p><p>Some of the methods to overcome high bias are:</p><ul><li>Use more complex model (e.g., kernelize, use non-linear models)</li><li>Add features: Remember it&#39;s about adding more features NOT more data. If variance is also high, then we can add data, but if variance is low with high bias, then adding data will not help much.</li><li>Boosting: combing weak learners to create a strong learner</li></ul><p>In conclusion, data bias and model bias are different types of biases, both of which should be identified and mitigated. Data bias can lead our ML model to make biased decision because we trained it with biased data, however the model bias can lead our model to make inaccurate predictions because we are using simple model whose parameters are not enough to capture the underlying relationship of input features with desired output.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75d62400a89b" width="1" height="1" alt=""><hr><p><a href="https://pub.aimind.so/data-bias-vs-model-bias-in-machine-learning-75d62400a89b">Data Bias vs. Model Bias in Machine Learning</a> was originally published in <a href="https://pub.aimind.so/">AI Mind</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My 30-days journey towards learning PySpark]]></title>
            <link>https://medium.com/@nirmal.budhathoki/my-30-days-journey-towards-learning-pyspark-b346ad06e10a?source=rss-500fd3d011d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b346ad06e10a</guid>
            <dc:creator><![CDATA[Nirmal Budhathoki]]></dc:creator>
            <pubDate>Thu, 17 Nov 2022 21:12:33 GMT</pubDate>
            <atom:updated>2022-12-08T07:06:11.438Z</atom:updated>
            <content:encoded><![CDATA[<p>I am starting a new learning series on <strong><em>#30daysofPySparkLearning.</em></strong> This will be focused on learning PySpark. I also needed a refresher for myself on this topic, therefore I am doing a series will help me organize the content.</p><p>This blog is to organize all my linkedin posts into one place. I will keep appending on it, as we move along the journey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E27zNNFMwTvXj_XbO2RY-Q.png" /><figcaption>When Python falls in relationship with Spark</figcaption></figure><p>Lets begin with <strong>Day 1 of #3<em>0daysofPySparkLearning:</em></strong></p><p><strong>What is PySpark ? Why it is valuable skill to learn ?</strong></p><p>Before PySpark, let’s define Spark, also known as Apache Spark- is open source big data framework supporting in-memory processing of data to boost the performance when it comes to data pre processing and cleaning, one of the major steps in any machine learning project.</p><p>Spark is actually written in Scala. If you are interested in data engineering career track, and if you will be involved more in building infrastructure for data pipelines/ ETL jobs ; then Scala might be a good skill to learn and add in your resume.</p><p>However from data science and machine learning perspective, I am little more biased towards learning PySpark for its simplicity, and alignment with Python. PySpark is the python API to core Spark, and it also lets us use SQL like scripts wrapped into python functions. Data Scientists will find new spark in their relationship with Python after using PySpark.</p><p>If you are in core machine learning side of Data Science, chances are you will be dealing with big data, and pandas is not always ideal for pre-processing. PySpark will be useful to set up data processing jobs. It will save you time and money :)</p><p><strong>Day 2 of <em>#30daysofpysparklearning</em> : Architecture of Spark</strong></p><p>We will briefly cover the architecture of spark. If there are 3 key features, you want to keep in mind regarding spark then I would say following:</p><blockquote>1. In-memory Processing: speed</blockquote><blockquote>2. Distributed Processing: parallelism</blockquote><blockquote>3. Lazy Execution: optimized computing</blockquote><p>The architecture of spark job execution follows the driver- workers (one to many) relational model. The driver is the main coordinator that accepts and distributes the jobs among workers. Each worker will have one or more executors to execute the assigned jobs (tasks).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NsucgPhWvDyHf9nUY9y1uw.png" /><figcaption>Driver- Worker Relational Model of Spark Architecture</figcaption></figure><p>Cluster manager is the management layer, the platform or framework used to manage the jobs in spark. It can be standalone (from spark), Mesos, YARN or Kubernetes.</p><p>When a spark job is submitted, the driver contacts cluster manager to request resources for worker nodes; the cluster manager launches worker nodes and executors. Once that happens, there could be direct relationship between driver to worker nodes (as shown in the graph).</p><p>Each job has an execution plan created by driver, and it is in the form of DAG (Direct Acyclic Graph). The plan is not executed until there is an action. In other words, all the transformation code we write and chain it, it wont be executed until there is an action followed to the transformation. This is the lazy execution or lazy evaluation part of spark.</p><p><strong>Day 3 of </strong><a href="https://www.linkedin.com/feed/hashtag/?keywords=30daysofpysparklearning&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7001311883790024704"><strong>#30daysofpysparklearning</strong></a><strong>: </strong>We will cover installation of PySpark. I am sharing two options:</p><blockquote>Option 1: On conda virtual environment: This is the preferred and ideal way if you want to set it up on your local machine. On mac OS, the best resource I could find that worked for me is:<br><a href="https://lnkd.in/erxUwMCm">https://lnkd.in/erxUwMCm</a></blockquote><blockquote>*caution with Option 1: you might run into Java gateway error, make sure you set the $JAVA_HOME variable properly.</blockquote><blockquote>I have not tried on windows but I will share the link on comment later.</blockquote><blockquote>Option 2: While the above option is ideal, I am also sharing the link to my google colab notebook. The java and spark packages are not too heavy so even if we have to initiate the installation each time we run the notebook, it’s not too bad. The advantage of this option is not worrying about any environment setup.</blockquote><p>For simplicity of sharing the lessons in a single notebook, I will be using option 2 for our learning journey. I will add different chapters as we move along. The first chapter is just loading the sample data and viewing as dataframe.</p><p>Here is the link to the colab notebook:<br><a href="https://lnkd.in/edJ-eMpB">https://lnkd.in/edJ-eMpB</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NGWOS1tyo6aQg-pVuDSHMA.png" /></figure><p><strong>Day 4 of<em> </em></strong><a href="https://www.linkedin.com/feed/hashtag/?keywords=30daysofpysparklearning&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7003203873641254912"><strong><em>#30daysofpysparklearning</em></strong></a><strong><em>:</em></strong><em> </em>On my last post, I talked about installation of pyspark. As I mentioned I will be using the colab notebook to share all the learnings broken down by small chapters.</p><p>On this post, we are covering chapter 2 on reading csv and checking the datatypes and schema.</p><p>Click here for the <a href="https://colab.research.google.com/drive/11kOFXTuj6tRM3HORpF-Xf3JpD2FyDB-F?usp=sharing">link to the Colab.</a></p><p>Click here for the <a href="https://drive.google.com/drive/folders/1znRv5kexyqR1vJQMIDaRon_ZGjlNz4H_?usp=sharing">link to get data</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EgLyftPZld54WJBCfeAZZQ.png" /></figure><p><strong>Day 5 of </strong><a href="https://www.linkedin.com/feed/hashtag/?keywords=30daysofpysparklearning&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7003203873641254912"><strong><em>#30daysofpysparklearning</em></strong></a><strong><em>: </em></strong>On this post, we are covering Chapter 3 on finding and handling missing values, and encoding a column based on mapping.</p><p>Click here for the<a href="https://colab.research.google.com/drive/11kOFXTuj6tRM3HORpF-Xf3JpD2FyDB-F?usp=sharing"> link to the Colab</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IHeDvUo4LzUV9qyPLGdC9g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b346ad06e10a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[30 Days of Data Science Career Tips- Part 2]]></title>
            <link>https://medium.com/@nirmal.budhathoki/30-days-of-data-science-career-tips-part-2-418199d33536?source=rss-500fd3d011d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/418199d33536</guid>
            <dc:creator><![CDATA[Nirmal Budhathoki]]></dc:creator>
            <pubDate>Tue, 01 Nov 2022 19:49:17 GMT</pubDate>
            <atom:updated>2022-11-01T19:51:04.525Z</atom:updated>
            <content:encoded><![CDATA[<p>This is the part 2 of the 30 days of data science career tips.</p><p><a href="https://medium.com/@nirmal.budhathoki/30-days-of-data-science-career-tips-part-1-a2ab998fbada">Click here for Part 1</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d9EFLiax4AewVtwh" /><figcaption>Figure: Word Cloud created based on all the Tips I shared</figcaption></figure><p><strong>Tip 16: </strong>Let’s talk about Model Explainability and Interpretability, one of the areas that is always overlooked during the learning phase of DS. While the two concepts come together, there is some difference. Explainability deals with how well the model parameters can justify the result, while Interpretability deals with how accurately a model can associate a cause to an effect. For every feature we select, can we form a hypothesis to prove the cause and effect is what interpretability tries to answer, while explainability seeks answers for what features are contributing to the output and by how much. So in a way explainability tries to quantify the interpretability. Model accuracy and interpretability are negatively correlated, and there is a tradeoff. Pick a model based on your problem. If you are working on a loan approval model, you need to explain why you approved vs reject the loan, however if you are classifying an image of dog vs cat, you don’t need much explanation. Always follow the principle of KIS (Keep It Simple). Here is a good read: <a href="https://christophm.github.io/interpretable-ml-book/shap.html">https://christophm.github.io/interpretable-ml-book/shap.html</a></p><p><strong>Tip 17:</strong> Let’s talk about the current data science job market. Based on the most recent data, there seems to be higher demand for data engineers (DE) than data scientists (DS). In 2020, there was 88% increment for DE jobs, and 49% for DS jobs, as compared to previous year. While the DS was topping the list in earlier years, I believe the reason behind this flip is that companies have now realized that they need to set up proper data pipeline before they can build and ship models to production. Of course without the data, what good is model for ? According to one Gartner report, it predicts that 80% of DS projects would fail without the proper support from DE. I suggest spending some time on learning the big data pipeline. DE is a different domain itself, so I am not suggesting to learn everything if you want to be DS focused. But getting some knowledge and skills will make you more marketable. There are so many tools, so I would rather learn in a tool agnostic way by picking some best ones. If you know one, knowledge transfer to other similar tools won’t take long.</p><p><strong>Tip 18: </strong>Let’s try to bust one myth in DS. Sometimes we hear in the DS community that Data Science is all about Machine Learning (ML). It’s true that ML is a part of DS, but it definitely goes beyond ML. I think DS correlates strongly with Data Mining instead- finding patterns in big data using methods of ML, Stats, and/or other analytics techniques. In the real world, the underlying goal of DS is to identify data sources that matter to the problem statement you are trying to solve, and generate the business value out of the data. The initial phase involves: ingest the right data (data contract and service level agreements aka SLA), data validation, data preprocessing and exploration. These steps establish the most heavy lifting phase of DS (more than 80%), and it’s unavoidable. Next phase depends on how you are solving the problem- is it by ML, or by statistical modeling, by deep learning, or sometimes simply doing descriptive analysis and showcasing the valuable metrics to aid decision makers. It does not always necessarily involve ML.</p><p><strong>Tip 19:</strong> Do I need a degree in Data Science ? This is another confusing topic. Having Master’s in DS myself, I actually do NOT think that you need a DS degree. We mostly see the companies mention their minimum requirement of a Bachelor’s degree, with MS/ PhDs as preferred qualification. However your MS will help you compensate for the years of experience being asked for. From the learning perspective, there are plenty of online resources, and MS is just like a long term bootcamp with extended syllabus. Since the time commitment is not as long as PhDs, MS has been a popular choice. When it comes to ROI, it can get tricky because there is no reverse engineering on your job application process after you land a job. If you can afford the time, money and commitment, and if you can pick the right school, I suggest to go for it (caveat: many colleges these days do DS degree just to catch the market wave). Even after MS, you have to polish your skills, and do projects to build portfolio. If you are looking for Research or Applied Scientists in top tier companies, then PhDs are well valued for roles without experience.</p><p><strong>Tip 20: </strong>As a data scientist we focus more on hard skills and sometimes we are falling back on soft skills. One of the important soft skills to build is communication and storytelling with data. As a data scientist, you will collaborate with multiple functions across the organization. Most of the time, the way the DS team is structured is either as a core data science team with 360 connections with other functions or as a lateral bridging to work along with product managers and engineers. Either way the most important skill that helps here is cross collaboration and teamwork. Other soft skills that are important to sharpen are: Can you challenge others with data in a tactical manner? How good of a listener you are? Your storytelling with data skills ( this one relates with data visualization skills but from a presentation point of view). How much patience do you have to work with data until it confesses ? Do you have enough passion to dedicate lots of effort until you find something interesting ? All these skills are not technical, but they do matter.</p><p><strong>Tip 21: </strong>While continuous learning to improve the skill set is encouraged and valued for every job, it matters even more for data science. Things are changing every other day- new programming packages are getting released, new research papers are getting published, and with the advancement in compute power, we are seeing some novel findings in the field of machine learning and deep learning. What we learn today may become obsolete tomorrow. Either you keep up or you give up. AutoML is becoming popular these days as ML is getting more and more democratized. Cloud computing has opened up many opportunities when it comes to big data. There are so many online resources to update yourself with these new trends in DS.</p><p><strong>Tip 22:</strong> One of the fundamental things in DS, and yet it is not properly followed, is a well defined DS workflow while working on a project. The best framework to follow for the DS project life cycle is CRISP-DM (CRoss Industry Standard Process for Data Mining). It provides a set of guidelines to help plan, organize, and execute your DS project. The process may have slight variance across different organizations, but the idea is to follow an end-to-end workflow: starting from problem definition to model deployment. In the attached pic, I added my modified text in red to highlight a couple important things. Business understanding &lt;-&gt; Data Understanding is an iterative process. You first define what business problem you are trying to solve, then you gather some data to perform initial exploration. If you learn that the data doesn’t support the problem you defined earlier, then you take a step back and redefine the problem before it’s too late. Data Prep &lt;-&gt; Modeling is also an iterative process. This is where feature engineering happens, deriving and trying new features. During modeling, you have to revisit the problem if it aligns with the results you have. As you have noted, Data always stays at the heart of the framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wwj3EJ-AA0Rvk1fL" /><figcaption>Figure: CRISP-DM Framework of Data Mining</figcaption></figure><p><strong>Tip 23:</strong> Let’s talk about graph analysis. This can be an additional skill to learn and showcase in your resume: knowledge of graph DB, link analysis, and how to run ML algorithms on graph data. This is a complex area in itself. It is certainly more valuable if you are looking to get into domains like social media, search engine and information retrieval, finance, transportation, e-commerce, telecom and security. Well that covers almost all big industries, which means graph analysis is important. Page ranking is one common example of how google uses link analysis. Linked-in connections, FB friends, twitter followers are all examples of network graphs. Here people are nodes and connections are edges. Defining ML problems on graph data is not as straightforward as defining it on tabular data. Node classification, link prediction, community detection, betweenness score, centrality score are few popular ones. I have attached the presentation I did during one data conference while I was in WellsFargo (Finance). <a href="https://docs.google.com/presentation/d/0BytY84wRCKuqTHpXT05UQWpzNDlJY0FSaERjZ0xvZXRjQ19v/edit?usp=sharing&amp;ouid=107542174919752731147&amp;resourcekey=0-TUCfRZhtoQ3zyzOUequBkQ&amp;rtpof=true&amp;sd=true">Link to my presentation.</a></p><p><strong>Tip 24: </strong>Networking is very important in your data science career. Here is a rule I would suggest to follow: Spend some time on networking, but don’t burn all your hours on it. Make sure you find your balance point. At the initial stage, I suggest you focus on learning and developing skills, then spend some time on networking after you have some idea which domain you will prefer to enter. Focusing on connecting with people from particular industries of interest will be more helpful. Participate in DS meet-ups, attend DS conferences (because of the pandemic, most of the conferences now are online and they are also giving many free tickets), participate in online hackathons.</p><p><strong>Tip 25: </strong>So far almost all my tips were focused on what to do. Let’s talk about some of the things on what not to do. Some of my Don’ts are:</p><blockquote>Don’t try to learn all the tools at the same time. Take it slow. Build your sequential learning pattern. Understand and take dependencies.</blockquote><blockquote>Don’t spend too much time on theory. Try to blend it with end to end projects.</blockquote><blockquote>Don’t just pick a domain solely based on the job market. I agree that it should be one important factor, but your interest and passion to work in a certain domain should overrule that. If you don’t enjoy the data, the science piece is incomplete.</blockquote><p><strong>Tip 26:</strong> In data science, mentorship is very important. I have got the opportunity to work with many mentors, and everyone of them have tremendously helped me grow and shape my career. Mentorship is not about having a coding buddy, it’s not about finding a job coach, it’s not about supervising, and it’s not about tutoring. Mentors can share advice, give you meaningful critiques, help you set up your goals, brainstorm ideas with you. They are one of the most valuable resources of learning for you. Finding a right mentor can be challenging, but you just have to keep your eyes open, reach out to folks from the DS community in your connection list and simply ask. The worst that can happen is No, but that’s not the end. If you find a mentor, be respectful about their time, and have an agenda to maximize the productivity of the mentoring session.</p><p><strong>Tip 27:</strong> Cloud Computing- yes we are talking about how important it is to learn the basic fundamentals of cloud computing, and more in-depth on some of the tools for DS. ML/AI has gained momentum because of the compute power we have now as compared to the past. Most of the algorithms are not new. ML needs big data for training, and big data means more processing. The challenge is the labeling of data, which is a different topic. Current market leaders on cloud are AWS, Azure and GCP. I suggest learning how to build, train, and deploy models in any of these or others. They have a free subscription, which you can use to learn. First I suggest learning the fundamentals of running ML on cloud over youtube videos, and then you have some understanding so you can maximize the benefits of free subscription. Most of the projects we do while learning DS have small data sizes, and we can run in a local environment. But in the real world, the data scales, and the question is: does your model scale ? It’s important to consider these factors during the design.</p><p><strong>Tip 28:</strong> I am chatting about how DevOps relates to DS. Whether your role is applied DS or more towards product analytics type DS; unless you are very much research driven, you will come across the DevOps. Notebook programming will be good for the PoC stage, and once the business decision is made to build the model, you will have to collaborate with the engineering group on how to productionalize the model. CI/CD (continuous integration and continuous delivery) is the heart of DevOps model. In the ML world, this framework is known as MLOps (= ML + DevOps). MLOps adds CT (continuous training) to make it CI/CD/CT. In general, MLOps means automation and monitoring of ML systems at all steps: integration, testing, releasing, deployment and infrastructure management. The validity and stability of model after deployment, whether there was data drift after new training data is added, what is the iterative cycle of model retraining- all these are covered under MLOps. I suggest spending some time understanding this framework. Here is a good resource to review: <a href="https://ml-ops.org/">https://ml-ops.org</a></p><p><strong>Tip 29: </strong>I am covering the design of an experiment (DoE) in an ML engineering project. What phase of the ML life cycle you should introduce or do you even have to introduce DoE is a tricky question. Typical software development model will not work incase of ML engineering project. And a typical data science workflow will also fall short when it comes to building and shipping ML projects. The key component missing in the software engineering lifecycle is DoE, and the key component missing in pure DS workflow is software design. Therefore the best middle ground will be to have a tightly coupled DoE with s/w design. The implementation must be an iterative process where model results will give feedback to redesign the experiments. DoE is essentially the testing of hypotheses for each feature and how it impacts the model performance. I know this concept can be a little confusing, therefore I have created the following flowchart to explain everything. Problem statement is what triggers data wrangling and architecture design for both sides. You can see how the coupling of DS workflow, and software development model leads to successful design of ML engineering projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bzf9IvjkpLItlqaZ" /><figcaption>Figure: Design of Experiment for Data Science Project</figcaption></figure><p><strong>Tip 30:</strong> My final tip is very simple: “Ask lots of questions, data science is all about curiosity.” Do not hesitate to reach out for help; connect with the business partners, data analysts, data engineers, or whoever in your group are involved in the project. Two main reasons why we don’t feel like asking Qs: we inherently assume that it might be a stupid Q to ask, and secondly we think that we can manage to solve everything without help. Both reasons are bad choices. There is no such thing as a stupid Q, everyone is learning. That’s why companies believe in diversity &amp; inclusion. Say you managed to complete everything w/o help. Sometimes you might end up reinventing the wheel. Resource and time optimization is the key. You could have spent that time on something productive for the entire team. This is one disadvantage of working in a siloed way.</p><p>Finally sending my best wishes to all the aspiring or early career data scientists. Keep learning, stay focused; if you are in a job hunt, don’t lose hope, it’s just a matter of time that you get to land on something. Your patience is a real test, not the interviews.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=418199d33536" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[30 Days of Data Science Career Tips- Part 1]]></title>
            <link>https://medium.com/@nirmal.budhathoki/30-days-of-data-science-career-tips-part-1-a2ab998fbada?source=rss-500fd3d011d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2ab998fbada</guid>
            <dc:creator><![CDATA[Nirmal Budhathoki]]></dc:creator>
            <pubDate>Sat, 29 Oct 2022 00:57:09 GMT</pubDate>
            <atom:updated>2022-11-01T19:50:22.259Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, I did a 30 day series to talk about interview questions and answers asked in top tech companies. For the interview QnAs, I was maintaining the blog to have all questions in one article:</p><p><a href="https://medium.com/@nirmal.budhathoki/30-days-of-dsml-interview-questions-5b43a51e38c8">30 Days of DSML Interview Questions</a></p><p>Back in 2021, I had similar series but it was focused on data science career tips. I had many folks responding me if I had collected all those tips in an article. Sadly then, I never started an article on it. But thanks to Linkedin, I was able to download all those posts. To organize it better, and not to make a bulky read, I have divided total tips into two. This is part 1 covering 1–15 tips.</p><p><a href="https://medium.com/@nirmal.budhathoki/30-days-of-data-science-career-tips-part-2-418199d33536">Click here for Part 2</a></p><p><strong>Tip 1:</strong> <em>Data scientist is not the sexiest job of the 21st century.</em> Sorry to start this with a bitter reality. People come to the data science field with very high expectations. While the jobs in the past were increasingly in high demand for Data Science, job seekers have now realized that companies were/are just slapping the title for any data jobs as data scientist. Most of the time you take a job and wonder to yourself- what happened to the sexiest job ? You will be working not only to build models; but to collect, ingest and clean the data. You will be lucky if you find a well defined data engineering team to help you for pipeline building. Therefore my first tip is lower your expectations, and learn some data engineering skills.</p><p><strong>Tip 2: </strong>During the early stage of a data science career, it is very easy to become a victim of the Dunning- Kruger effect. As Wikipedia says: The Dunning-Kruger effect is a hypothetical cognitive bias stating that people with low ability at a task overestimate their ability. I have added one diagram below with my edited comments on red. Try to find that best route as I have pointed out on the graph, because sudden rise in confidence with limited knowledge, and later falling down on it can be harsh to negatively impact your own learning experience. Identify yourself where you are, and once you pass the convex point, confidence and knowledge can both follow the positive growth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/0*FaGv8f9ojLrFLx6H" /><figcaption>Figure: Dunning- Kruger Effect</figcaption></figure><p><strong>Tip 3:</strong> Ask yourself if you are ready for a data scientist role. If the answer is maybe, then you are already doubting yourself. These days the Data Scientist job sounds like ‘jack of all trades and master of none’. Companies are adding all the data related keywords in job description(JD). In effect the job seekers are tweaking their resumes by adding the same keywords to beat the automated resume filtering system. This process of keyword stuffing is cyclic, and never ending. In the early career stage, sometimes it’s better to find jobs as Data Analyst, or Business Intelligence Analyst, where you can optimize your learning time to the skills focusing in some core areas rather than trying to learn everything driven by the job market, not by what you are interested in learning. Most of the time you will end up working only on 25% of JD. For analyst roles, I suggest spending some time in Data ingestion/validation, ETL, SQL, Descriptive Statistics, and Data Visualization. These are good skills for a Data Scientist as well. Once you get an opportunity to gather some experience, you can focus your self-learning towards DS/ML skills, and build upon some domain expertise to help your next career move towards the Data Scientist role.</p><p><strong>Tip 4: </strong>Sometimes it is very easy to get yourself pulled into multiple directions while you are trying to learn and ramp up your DS skills. It has happened to me. I consider myself as a person who buys more books than I read. To overcome this, lay out a clear study plan, break it down into multiple subsets, and make sure that each subset covers an important area of DS. Based on my learning experience, I am suggesting following areas:</p><ol><li>Programming skills</li><li>Math (including Statistics &amp; Probability)</li><li>Data Wrangling</li><li>Machine Learning</li><li>Data Visualization</li><li>Big data transformation</li><li>Product sense</li><li>Communication skills</li></ol><p>Going forward, we will be covering one tip for each of the above subset.</p><p><strong>Tip 5: </strong>I am covering the subset of ‘Programming Skills”. In my personal experience, I suggest focusing on one analytics language rather than trying to learn many. I prefer Python because it is easy to learn, has lots of great packages, open source, and the python skills can be used not only for analytics but also for other areas. Google Colab is great, since you can start coding in python right away without the hassle of setting up an environment. Once you are familiar with it, you can set up a conda environment on your own. One thing I suggest is while you learn Python, think from the Data Structures perspective, it will be helpful in future for the interview prep. One more language to ramp up is SQL. You will be writing queries for sure. Spend some time on advanced aggregations, CASE statements, Subqueries, JOINs including self-join, CTEs, and window functions.</p><p><strong>Tip 6</strong>: I am covering the subset of ‘Math (Stats &amp; Probability)’. This is one of the heaviest learning paths for DS. For Math, I will suggest starting from a refresher on linear algebra, vectors, matrix multiplication and calculus. For the Statistics &amp; Probability (S/P), it can go to any depth. Best strategy is to set an early objective of what kind of role you prefer. Most of the DS roles are aligned to one of the following:</p><blockquote>Applied Scientist: High bar on S/P and some coding</blockquote><blockquote>Data Scientist (General): Medium bar on S/P, and Data analytics</blockquote><blockquote>Research Scientist: High bar on S/P, Research Oriented (PhDs)</blockquote><blockquote>ML Engineers: Low bar on S/P, very heavy on Coding</blockquote><p><strong>Tip 7: </strong>I am going to cover “Data Wrangling” here. In short, data wrangling is the process of data handling, cleaning, validation and transforming the raw data into processed data that is ready for analysis. It actually covers the data preprocessing and exploratory data analysis (EDA). Sometimes EDA can be viewed as a separate component, but I think it can be considered under the same hood if we include variable analysis ( both univariate and multivariate), correlation analysis, and some plots for visualization of distribution. Most of the tools or packages, I will be talking here will be Python based. Following are the top 5 packages for data wrangling plus EDA: pandas, numpy, scipy, statsmodels and matplotlib. Of course there are many more, that’s the beauty of python. One more package worth of mentioning is- pandas_profiling. I suggest using it if you want to create a nice html report.</p><p><strong>Tip 8: </strong>Today I will be covering ‘Machine Learning’. It is the biggest area of the DS learning path. I will have to divide &amp; conquer this. Will be discussing ML in 3 sub-sections of Feature Engineering (FE), Modeling and Deployment.</p><p>Feature Engineering is an art, it requires both domain knowledge and creative thinking. Data by itself will never provide the predictive power to the model. Too many features will overfit the model, having redundant and/or unimportant features will degrade model’s performance; and not having enough features will compromise the model performance. I personally suggest spending a significant amount of time on learning various techniques of feature creation and selection. In a business scenario, this requires discussion with subject matter experts and stakeholders. The correlation analysis vs. causal inference (aka causation) is important to understand. This is when hypothesis testing comes into picture.</p><p><strong>Tip 9:</strong> I am covering the subsection of Modeling under the main topic of Machine Learning. I used to always get confused on algorithm vs. model. And yes there is a difference. Here is an easy way to differentiate: Model = train( algorithm + data).</p><p>Training the ML algorithm with data to find the optimized hyper-parameters and weights is called modeling. Few important things to focus during modeling are: try multiple algorithms (always start out with simple), have validation set in your data split to use it for hyper-parameter tuning, pick your accuracy metrics wisely (e.g if you have class imbalance use AUC weighted instead of accuracy), learn about overfitting and ways to handle it, bias variance tradeoff. Modeling is vast, it’s not about just calling the fit and predict functions. I suggest spending some time on understanding how some of the popular algorithms work.</p><p><strong>Tip 10:</strong> Let’s talk about model deployment. This is one of the most important parts of the ML life cycle, yet it is always overlooked from data scientists. One of the main reasons is that while learning ML projects- we do data preprocessing, EDA, feature engineering, train multiple algorithms, cross validate and pick the best model, and thats it; a great feeling of achievement thinking that data science ain’t too bad. I was one of them, until I was working in a real job as a Data Scientist, and I quickly realized that the best model in notebooks and slides will never provide any business value. Most of the online blog posts, Kaggle projects, and even the DS community learnings are geared towards model development but not deployment. I suggest putting extra time and effort into learning model deployment. If you are building a git portfolio, take your ML project an extra mile by serving your model via web service so viewers can interact with your model. Think about different ways to stand out and be competitive in the job market. MLFlow is a great open-source end-to-end ML lifecycle management tool. Look into Flask, Streamlit, Container Service, Kubeflow. At least do one end to end project.</p><p><strong>Tip 11:</strong> A picture is worth a thousand words. Yes we are talking about data visualization today. In order to pass along the right message, I spent a few minutes to plot two visualizations for total post views on my tips so far. V1 on the left shows the bar chart of post views. This chart is a little crowded, and it is hard to draw insights out of it. V2 on the right is a time-chart, looks much cleaner, and I can draw some insights out of it- like the overall trend has slowed down, one data point is really sticking out ( tip 6 about Statistics was a popular one), it seems like I missed posting on 6th March. It’s amazing how a well structured and rightly picked chart can make a difference. Data visualization is an art. It comes handy during EDA, and also when you are presenting your findings to leadership and stakeholders. I suggest being tool agnostic when it comes to data visualization. Companies may use a variety of tools, what’s important is how you tell your story with data. Tableau and PowerBI are the leaders based on the Gartner report. Some python packages like matplotlib, seaborn, bokeh, and plotly are the popular ones out there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VqWBMAYh8PqhkWj2" /><figcaption>Figure: Comparing two visuals exploring same data</figcaption></figure><p><strong>Tip 12: </strong>Today I will be covering the importance of big data processing for DS. The definition of big data started with 3 Vs (volume, velocity, variety), but now it seems like more Vs are getting added to cover the bigger meaning of big data. For data science, big data is good news. Machine learning models, and more importantly deep learning models crave big data. But the challenge is how to efficiently handle and process big data. This is where data scientists need to wear their data engineering hat, and collaborate with their engineering team to build an effective data pipeline. Spark is the most popular framework of choice for big data processing. You can use it as python api (pyspark) or use it via its native language Scala. The learning curve on python is easier, but using it on Scala is 10 times faster. I will suggest starting with pyspark and then level up with Scala. Recently, I also felt that I need to ramp up my data engineering skills. Follow the hashtag #dataengineering in linkedin, tons of good information there.</p><p><strong>Tip 13: </strong>Today I will be talking about ‘product sense’ from a DS perspective. Oftentimes I have found Data Scientists a little hesitant about whether product skill is required to be a good DS. I argue with that statement. While you don’t need to know the full skillsets of a product manager, you have to understand the business value you are providing to the product. Either directly or indirectly you are supporting some product and delivering customer service ( either internal or external). Many tech companies now have specific roles for DS that are more geared towards product oriented analysis, and their interviews are structured to cover product case study. For those roles, hypothesis testing and experimental design skills are must have. At the end of the day, your stakeholders or customers care less about how many layers you added in your neural net, if you are not bringing up any business value.</p><p><strong>Tip 14: </strong>Today I will be covering the topic of communication skill. Most of the time, we focus on hard (technical) skills, and soft skills remain underrated. For DS roles, communication is very crucial, as you will be supporting multiple functions within the organization: like getting with business folks to gather requirements while identifying and framing the problem, then sharing your analytical findings back to them. Also during the project itself, it is important to cross collaborate with data engineers, software developers, product managers, and other key business stakeholders. One of the challenges I came across was communicating the data/model findings to non-tech audiences. In one of the interviews, I was asked to explain p-value to a business leader who is from a non-statistics background, and I struggled for a minute. The overall idea is how well you can break down your analysis in terms of business OKRs (Objectives and Key Results), and how you measure it by KPIs (Key Performance Indicators). I can suggest a great book to read if you are interested. Here is the summary of it:</p><p><a href="https://strategyfieldguide.com/articles/measure-what-matters-john-doerr/">https://strategyfieldguide.com/articles/measure-what-matters-john-doerr/</a></p><p><strong>Tip 15: </strong>Today’s post is about the importance of domain knowledge. If you are already doing DS for sometime, then you should already have a domain of interest to align your career. If you are new and trying to break into DS, then you might be unsure about what domain will interest you. However I suggest picking at least 2–3 domains of interest, and build your portfolio to include 1–2 projects from each domain. For example: it does not make sense to apply for a job at an e-commerce company if your portfolio doesn’t have any projects in that domain. Once you have gained some experience, you will realize which industry will interest you for a long term career, but it’s worth having multiple options in the beginning. Some of the popular industries fostering DS are finance, healthcare, social networking, e-commerce and retail, cyber security, gaming, telecom, and more.</p><p>I hope the tips were helpful to you. I will be sharing the Part-2 soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2ab998fbada" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[30 days of Data Science and Machine Learning Interview Questions]]></title>
            <link>https://medium.com/@nirmal.budhathoki/30-days-of-dsml-interview-questions-5b43a51e38c8?source=rss-500fd3d011d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b43a51e38c8</guid>
            <dc:creator><![CDATA[Nirmal Budhathoki]]></dc:creator>
            <pubDate>Sun, 03 Jul 2022 07:26:50 GMT</pubDate>
            <atom:updated>2022-11-21T00:32:56.717Z</atom:updated>
            <content:encoded><![CDATA[<p>I have started this article to keep track of all my linked-in posts on 30 days of data science and machine learning (DSML) interview questions. My goal is to cover the real interview questions asked by top companies like #google, #amazon, #meta and #microsoft. I have also shared my posts on how these companies define their data science roles. Those posts are helpful read to understand the variation among these companies when it comes to the job titles, responsibilities and desired skills.</p><p><a href="https://www.linkedin.com/posts/activity-6948039599445213184--d9G/?utm_source=linkedin_share&amp;utm_medium=member_desktop_web">Data Science jobs in Google and Amazon</a></p><p><a href="https://www.linkedin.com/posts/activity-6948519626309861376-2A8C?utm_source=linkedin_share&amp;utm_medium=member_desktop_web">Data Science jobs in Meta and Microsoft</a></p><p>I will be continuously editing this article, adding the questions as I post them. I have taken the feedbacks from my connections and readers to consolidate all the questions and answers into one article.</p><p><strong>Question 1: What is bias-variance trade-off ? If you can explain to a mix of technical and non-technical audience.</strong></p><p><strong>Answer:</strong> This question can come up in many DS interviews. Sometimes we just explain the conceptual definition and fail to bring the examples in the discussion. My goal is not providing in-depth answers, but I will try to cover the main highlights that we should not miss.</p><ol><li>Error = Bias + Variance + irreducible error. The reducible errors are bias and variance, but there is always a tradeoff.</li><li>Bias is defined by the simple assumptions made by the model. For example- assuming the linear relationship when the decision boundary is non-linear. Example- you have an image classifier, you are using a simple model with high bias, meaning the features are simple to miss the categorization of cats vs. jaguars vs. lions etc, because most of the common features are alike and those simple features are not able to distinguish.</li><li>For a high biased model, both training and test errors are high. That means high bias leads to an under fit model. Example of high biased model is a linear regression.</li><li>Variance is when the model is complicated, it tries to learn too well from training data including the noise, and therefore fails to make good predictions on test data. Similar example- if your model is complicated with complex features like detecting the fur-color, which may help you to predict everything in your training data, but what if the model is unable to predict on test data that has a new or unseen fur color. Is fur color a good feature or was it just a noise ?</li><li>For a high variance model, training error is very low, but test error is high. High Variance leads to over fit model. Example of high variance model is a decision tree.</li></ol><figure><img alt="Visual chart showing Model’s Error compared to the Model’s Complexity" src="https://cdn-images-1.medium.com/max/1024/1*HYY3mR7swbnCkmB_0Jw9xg.png" /></figure><p><strong>Question 2 : Given a sample table with emails sent vs. received by the users, calculate the response_rate which is given as emails sent/ emails received, and list all the users that fall under the 25th percentile based on the response_rate.</strong></p><pre>Below is the schema of table ‘<strong><em>gmail_sample_data</em></strong>’:<strong><br>from_user:</strong> <em>string</em> — user id of the email from<br><strong>to_user:</strong> <em>string</em> — user id of the email sent to<br><strong>day: </strong><em>int</em> — the day when email was sent</pre><p>I have populated some sample data in my bit.io account. If you are not aware of bit.io , it is very helpful to set up PostGres DB in few seconds without the hassle of installing PostGres and its dependencies. The free tier account comes with 1 DB, 1 CPU and 1 GB Memory, which is enough to practice queries with small sample data.</p><p>When I approach SQL question, I first come up with a logical plan to approach the problem before jumping on query. It is important that you share your thought process with interviewer. Here is my logical plan:</p><blockquote>1. Get the count of emails sent group by user id</blockquote><blockquote>2. Get the count of emails received group by user id</blockquote><blockquote>3. To get response rate, we have to make sure we bring both 1 &amp; 2 on same row because SQL operates on row. We can do a join of 1 &amp; 2</blockquote><blockquote>4. Calculate the response_rate</blockquote><blockquote>5. Use window function for ntile with offset=4 (quartile bucketing)</blockquote><blockquote>6. Final result will be all values that fall under the 1st quartile</blockquote><p>Once you have the logical plan, writing query becomes easier and efficient. I always prefer CTEs, you can also run the same with sub-queries. Below is the final query:</p><pre>With sent_emails as (select from_user, count(*) as emails_sent from gmail_sample_data g1 group by 1),</pre><pre>received_emails as (select to_user, count(*) as emails_received from gmail_sample_data g2 group by 1),</pre><pre>summarized_view as (select from_user as user_id, emails_sent*1.0/ emails_received as response_rate from sent_emails<br>left join received_emails on sent_emails.from_user = received_emails.to_user),</pre><pre>final_view as (select user_id, response_rate, ntile(4) OVER (ORDER BY response_rate) as quartile from summarized_view)</pre><pre>select user_id, response_rate from final_view<br>where quartile =1</pre><p>Link to my bit.io where you can run the above query: <a href="https://bit.io/nbudhath/trial">https://bit.io/nbudhath/trial</a></p><p>Credit: I came across similar question in <a href="https://platform.stratascratch.com/coding/10351-activity-rank?code_type=1">StrataScratch</a>, but this one is different variation. I have used the same dataset to practice my query. StrataScratch is a great resource for practicing SQL. I have worked with <a href="https://www.linkedin.com/in/nathanrosidi/">Nathanael</a> on sharing few of my SQL interview questions as well.</p><p><strong>Question 3: What are the assumptions of linear regression ?</strong></p><p>This is a popular question in DS interviews. The interviewer is looking not only for explanations but also for examples. You may come across different versions online. But the ones you must include are following four:</p><p><strong>1. Linear relationship:</strong> There has to be a linear relationship of the dependent variable (also referred as output variable or target variable) with the given independent variables (also referred as input variable or features). For example: We are predicting the price of house, and let’s say the given input variables are square_footage, num_rooms, location, year_built, and age. There could be more features but for simplicity let’s stick to these for now. The linear relationship assumption says- we should be able to model the regression function as:</p><p>Price = B0 + B1*square_footage + B2*num_rooms + B3*location + B4*year_built + B5*age</p><p>(*note: there will be more coefficients for categorical variables like location, which will be handled by creating dummy variables)</p><p><strong>2. Little to No multicollinearity:</strong> This assumption is tied to the first assumption where it says the input variables are independent, meaning there is little to no correlation among the input variables. In an ideal case, only the target variable should be correlated with input features but not among the features themselves. In our example: we will see high correlation between year_built and age, because it is obvious that higher the age, lower the year_built. We should only use one variable; age will be a good choice as it is already numerical.</p><p><strong>3. Homoscedasticity:</strong> The residuals or errors (true price — predicted price) should have the constant variance. Easy way to remember: your residual plot (residuals vs. predicted values) should show evenly distributed data points along the zero-line (mean error). Heavily skewed or outliers in the input features result in heteroscedasticity. Removing outliers, transformations of variables using log, square root, inverse or box-cox can help. In our example, if some neighborhood has few houses that are very high or very low in price, that may affect to violate this assumption.</p><p><strong>4. Normality of residuals:</strong> Once we calculate the residuals for all the data points, we can check with histogram or Q-Q plot for normality of residuals. If our model is good, it should be able to capture all trends (highs and lows) in the observed data. We can also interpret this as- the errors in our model should be consistent across the full range of observed data. If we had found the good fit line, residuals should follow normal distribution.</p><p><strong>Question 4: What is regression to mean (RTM) ? How does the correlation plays its role on RTM ?</strong></p><p>Regression to mean is the statistical phenomenon where the rare or extreme events ( both lower and upper bound) are more likely to be followed by typical events, and over the time extreme events regress towards the mean or average of the distribution.</p><p>One good example is the sports industry when some rookie players perform extremely good or bad in his/her first season. However in both cases, there is a higher chance that both of them will regress towards mean in up coming seasons. Success in sports is attributed to Talent + Luck. Suppose the talent is almost comparable for those two players, so luck has made a difference for one to perform well vs. the other. In this case, their performance will slowly converge towards the mean as compared to other players.</p><p>Imperfect correlation leads to RTM. When the association of target variable to input feature is weak or moderate, there is some explanatory power between the two but it is not perfect, that means there are other factors, which we are not aware of yet.</p><p>When you walk into doctor’s office for simple check up, if your blood pressure in the first measurement is extremely high or low, they will take another measurement to make sure they avoid any systemic errors before concluding anything. This is the most important part, that when we find some extreme samples in our data, before we jump into any conclusions, we should consider few more experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qmYNn9UMSPJqvrguy338ow.png" /><figcaption>Regression to Mean Example- When Players converge to Mean Performance as the years progress</figcaption></figure><p><strong>Question 5: You friend purposed a game of dice, where you are given two fair six-sided dice and asked to roll. If the sum of the values on the dice equals seven, then you win $21. You must pay $5 every time you roll the dices. Do you play this game?</strong></p><p>This question with solution can be found in many online resources, so I don’t think it will repeat, but similar questions with slight variations can always come.</p><p>Solution: When we roll two dices, the total possible events = 6*6 =36</p><p>Now what the events of interest ? For this we need to find what are winning combinations = pairs that add to 7, which are (1,6), (2,5), (3,4), (4,3), (5,2), (6,1) = 6</p><blockquote>Probability of winning , P(Win) = 6/36 = 1/6</blockquote><blockquote>Probability of losing, P(Loss) = 1- 1/6 = 5/6</blockquote><blockquote>Earning from win (E_Win) = $21 — $5 (to play) = $16</blockquote><blockquote>Earning from loss (E_Loss) = — $5 (negative)</blockquote><p>Now we are simply calculating the expected earnings as:</p><blockquote>= P(Win) * E_Win + P(Loss) * E_Loss</blockquote><blockquote>= 1/6*$16 + 5/6 * (-$5)</blockquote><blockquote>= — $1.5 (negative)</blockquote><p>On average you will be losing money, so you should not play the game.</p><p><strong>Question 6: How do you make an unfair coin a fair one ?</strong></p><p>Let’s say we have an unfair coin with probability of head , P(H) = 0.4 , and probability of tail , P(T) = 1–0.4 = 0.6. In fact you can pick any split beside 0.5 for the unfairness to either head or tail.</p><p>Now how can we get a fair deal from this unfair coin ? Let’s say we do two tosses. Possible results are: HH, HT, TH, TT.</p><p>P(HH) = P(H) * P(H) = 0.4 * 0.4 = 0.16</p><p>P(HT) = P(H) * P(T) = 0.4 * 0.6 = 0.24</p><p>P(TH) = P(T) * P(H) = 0.6 * 0.4 = 0.24</p><p>P(TT) = P(T) * P(T) = 0.6 * 0.6 = 0.36</p><p>If you look at above results, P(HT) and P(TH) have same probability of occurrence as 0.24. So instead of doing just one toss for head or tail, we should do two tosses but only consider the events P(HT) and P(TH). In that case the coin can still act as a fair coin.</p><p><strong>Question 7: Explain geometric vs. negative binomial distribution ?</strong></p><p>Most of the common distributions we come across are- Bernoulli, Binomial, Uniform, Normal, Poisson, and Exponential. However the geometric and negative binomial distributions are not as widely discussed as others, but they are very important to understand.</p><p>Both geometric and negative-binomial are related to the binomial distribution having independent trials with two possible outcomes. However the random variable highlights different aspect of the experiment. In Bernoulli, random variable is focused on defining the number of successes, but in geometric and negative binomial, random variable is focused on number of trails we conduct to get specific number of successes (one or many).</p><p>Consider an example when we toss a fair coin 3 times. Possible results are {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT}. Let X be the random variable representing the number of heads. The probability of getting exactly 2 heads, is P(X=2) = 3/8 = 0.375</p><p>Now, let’s look at it from geometric distribution, where the focus of random variable is number of trails to get first success (lets say getting head is the success). Let x be the number of trials at which the first head occurs.</p><blockquote>Then, P (X=x) = P( 1st success on xth trail)</blockquote><blockquote>= P( 1st (x-1) failures and success on xth trail)</blockquote><blockquote>= (1 — p)^ (x-1) * p (where p is probability of success and x= 1,2,3…..)</blockquote><blockquote><strong>Example: Find the probability of getting a head on your third toss ?</strong></blockquote><blockquote>P (X= 3)= (1- 0.5)^(3–1) * 0.5 = 0.5² * 0.5 = 0.125</blockquote><p>The negative binomial generalizes the geometric distribution by considering any number of successes instead of one.</p><blockquote>P (X=x) = P( <em>kth</em> success on <em>xth</em> trail)</blockquote><blockquote>= Combination (x-1, k-1) * (1-p) ^ (x-k) * p ^ k , where x= k, k+1, k+2…</blockquote><blockquote><strong>Example: What is the chance of seeing 3rd head on 5 tosses?</strong></blockquote><blockquote>Here p= 0.5, x= 5, and k= 3. If we use the above combination formula:</blockquote><blockquote>P(X=5) = [4!/(2!2!)] [1-(1/2)]² [1/2]³ = 6/32 = 0.1875</blockquote><p><strong><em>Some interesting facts:</em></strong></p><blockquote>1. When number of success is one (k=1 in above equation), negative binomial becomes the geometric distribution.</blockquote><blockquote>2. Negative binomial is called as pascal distribution.</blockquote><blockquote>3. In geometric and negative binomial distribution, random variable is number of trails, in the binomial and Bernoulli the random variable is the number of successes. When number of trails= 1, the binomial becomes Bernoulli.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BRgiOZyNnuF_SXNtovfhw.png" /><figcaption>Image Credit: @JBStatistics</figcaption></figure><p><strong>Question 8: We have a ‘user_accounts’ table having account and user information along with the date based on account used by the user. Calculate the monthly growth rate of users for each account.</strong></p><pre>Below is the schema for table &#39;<strong>user_accounts</strong>&#39;:</pre><pre><strong><em>date:</em></strong> string in ‘MM/DD/YY’ format<br><strong><em>account_id:</em></strong> string<br><strong><em>user_id:</em></strong> string</pre><p>Output should have account_id, year, month, growth_rate ordered in increasing order of months.</p><p>Sharing my bit.io link where I added some sample data and the query:</p><p><a href="https://bit.io/nbudhath/trial">https://bit.io/nbudhath/trial</a></p><p>Always remember these 3 steps to tackle the SQL question:</p><blockquote>1. Ask Questions and Clear the doubts about data and assumptions</blockquote><blockquote>2. Build a Logical plan- communicate the plan back to interviewer</blockquote><blockquote>3. Write the query</blockquote><p>First of all, it is important to ask questions and clear any doubts or assumptions beforehand. Few clarified questions with interviewer:</p><blockquote>1. What is relationship between account and user ?</blockquote><blockquote><em>Response: Account to user is one-to-many relationship. One account can have many users, but one user is associated with only one account at a time.</em></blockquote><blockquote>2. How to handle null values while calculating previous month data ?</blockquote><blockquote><em>Response: If the previous month is missing, simply disregard that month for analysis.</em></blockquote><p><strong>Logical plan:</strong></p><blockquote>1. Get the monthly active users (mau) for each account</blockquote><blockquote>2. Use window function lag to get last month’s mau for each account: The lag function always have null values for first instance since there is no data for its previous month. The null value in our growth rate calculation throws error, even if we replace it with 0, we will run into divide by zero error. Therefore it is important to clarify edge cases like this beforehand.</blockquote><blockquote>3. Use (current — last) / last to get monthly growth rate</blockquote><p><strong>Actual Query:</strong></p><pre>with mau as<br>(select account_id,<br>date_part(‘year’, TO_DATE(date,’MM/DD/YY’)) as year,<br>date_part(‘month’,TO_DATE(date,’MM/DD/YY’)) as month,<br>count(distinct user_id) as current_month<br>from user_accounts<br>group by 1,2,3<br>order by account_id, year, month<br>),<br>monthly_change as(<br>select *,<br>LAG(current_month, 1)<br>OVER (PARTITION BY account_id order by account_id, year, month) as last_month <br>from mau)<br>select<br>account_id, year, month,<br>Round((current_month — last_month)*100.0 / last_month,2) as growth_rate<br>from monthly_change<br>where last_month is not null</pre><p><strong>Question 9 : Explain the difference between Random Forest (RF) vs. Gradient Boosting (GB)? Which one is better?</strong></p><p>Sometime we fail to understand the core concepts. Interviewers will get many clues while you are explaining it whether you just studied for interview question or if you know them well. I will include following key points on discussion.</p><blockquote>1. Both RF and GB are based on ensemble learning. RF is based on bagging — bootstrap aggregating technique where random samples with replacement are selected for training each tree, and later results are combined- Avg. for regression task and max votes for classification task. GB uses boosting technique where many weak learners are combined in series to build a strong learner.</blockquote><blockquote>2. When it comes to overfitting, RF is better at handling it because of the bootstrap random sampling technique, since each tree never sees the full set of training data. However in GB, all training data is used by each tree, however each iteration the new tree will be fitted on the residuals to learn from past mistakes. So GB is more prone to overfit than RF.</blockquote><blockquote>3. Another advantage of RF is the trees are independent since it sees different random samples to train, the parallelism in training can be achieved, but for GB since the trees are sequentially built, we cannot achieve parallelism.</blockquote><p>For the second question. I will not take side to any of them, rather both can be equally efficient depending on the problem space. It’s better to evaluate both against your own business metrics and make decision accordingly.</p><p>Fun fact: Most of the popular kaggle models are Gradient Boosting- different variations like XGBoost, Light GBM, Catboost, Adaboost. If you are competing in kaggle, go for GBM first :)</p><p><strong>Question 10: Given a string s containing just the characters ‘(‘, ‘)’, ‘{‘, ‘}’, ‘[‘ and ‘]’, determine if the input string is valid, meeting following conditions:</strong></p><blockquote>1. Open brackets must be closed by the same type of brackets.</blockquote><blockquote>2. Open brackets must be closed in the correct order.</blockquote><p>Let me clear one important thing when it comes to coding round for data science interviews. One of the most confusing part of data science interview process is the DSA (Data Structures and Algorithms). Some companies do only data analytics part covering data wrangling skills in pandas or SQL, while some companies do DSA round.</p><p>Since data science interview process is not yet as streamlined as software engineering, it is little vague when it comes to the coding round. In my experience whenever there is DSA round, it is usually easy to medium level leetcode questions. Unless you are interviewing for ML engineer roles, you should be fine with arrays, strings, stacks and queues, and some questions from linked lists and binary tree. You can skip dynamic programming, graphs and trie for data science rounds. But always make sure with recruiters.</p><p>I have attached the screenshot and the link to my Colab notebook for solution to this problem.</p><p>Link to Colab: <a href="https://colab.research.google.com/drive/1gOlmZf2lwjuNzMq896Q-BlS6l7UPFFWT#scrollTo=hQiXzcXZlsYP">https://colab.research.google.com/drive/1gOlmZf2lwjuNzMq896Q-BlS6l7UPFFWT#scrollTo=hQiXzcXZlsYP</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FpgKxt2PHWeVXXn2atGnJA.png" /></figure><p><strong>Question 11: If there is a 20 percent probability that you will see falling star in any 30-minute time interval, what is the proba­bility that you see at least one falling star in the period of an hour (60 minutes)?</strong></p><p>One trick on answering questions where it is asking us the probability of something happening at least once is to flip the same question to:</p><blockquote>1. calculate the probability of something NOT happening</blockquote><blockquote>2. probability of at least once happening = 1 — P(Not happening)</blockquote><p>The probability of seeing a falling star in 30 mins = 0.2</p><p>The probability of NOT seeing a falling star in 30 mins = 1–0.2 = 0.8</p><p>Since the events of not seeing a star in first 30 mins is independent of not seeing a star in second 30 mins, the probability of NOT seeing falling star in 60 mins = 0.8 ^2 = 0.64</p><p>Now we are almost there, the probability of seeing at least one falling start in 60 mins = 1–0.64 = 0.36</p><p><strong>Question 12: What is the loss function in logistic regression called? If you can explain it, and is it possible to use it for multi-class classification?</strong></p><p>The loss function in logistic regression is called log loss or binary cross entropy. In logistic regression before the sigmoid function is applied, there is always predicted probability value calculated by the model. That means there are target labels (1 or 0) and predicted probability values. The binary cross entropy is the cost function used to minimize the loss from the predictions made by the model.</p><p>Let’s take a simple example of two classes- Green vs. Red, one of them becomes our class of interest. Let’s pick Green. Usually, class of interest is also called positive class or label 1. Consider following three samples:</p><pre><strong>sample A (right classification)<br></strong>predicted probability = 0.36<br>predicted label = 0<br>actual label = 0</pre><pre><strong>sample B (misclassified)<br></strong>predicted probability = 0.60<br>predicted label = 1<br>actual label = 0</pre><pre><strong>sample C (right classification)<br></strong>predicted probability = 0.70<br>predicted label = 1<br>actual label = 1</pre><p>Formula for binary cross entropy is attached as screenshot for readability purpose. In the formula, yi is actual label, and p(yi) is predicted probability of belonging to class yi. Let’s calculate loss for these test cases:</p><blockquote>For sample A-</blockquote><blockquote>Binary cross entropy = — 0 . log (0.36) — 1 . log(0.64) = 0.44</blockquote><blockquote>For sample B-</blockquote><blockquote>Binary cross entropy = — 0 . log (0.60) — 1. log (0.40) = 0.91</blockquote><blockquote>For sample C-</blockquote><blockquote>Binary cross entropy = — 1 . log (0.70) — 0. log (0.30) = 0.35</blockquote><p>As we can see, the mis classification example has higher loss, also where our probability for right class is higher, loss is lower.</p><p>The probability values are always 0 to 1, the log of which is always negative, so to make the loss positive we have -Ve sign. In this example, I have individually calculated loss for these samples, but as the formula shows we calculate for all N samples and try to minimize the average loss.</p><p>To answer the second part of question, we can use multi-class cross entropy, and use SoftMax instead of sigmoid. Or we can use one vs. all technique and use the same method as we discussed above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*IHUEwHkLPifeemk0m05mQw.png" /></figure><p><strong>Question 13 : Given a trips and a users table, write a SQL query to find the cancellation rate of requests with the unbanned users for each day between “2013–10–01” and “2013–10–03”.</strong></p><pre><strong>Schema for ‘<em>trips</em>’ table:</strong></pre><pre>id: unique id for the table<br>client_id : this is client identification number<br>driver_id: this is driver identification number<br>status: this is the status of request( completed, cancelled_by_client , or cancelled_by_driver)<br>request_at: requested date</pre><pre><strong>Schema for ‘<em>users</em>’ table:</strong></pre><pre>user_id : primary key for this table (it can be either client or driver)<br>banned: boolean field if user is banned or not (Yes/No)<br>role: user role ( either client or driver)</pre><p>I have added sample data and tables in my bit.io database, so you can run and test the query. If you have leetcode account, you can also try there.</p><p>Logical Query Plan:</p><blockquote>1. cancellation rate = number of cancelled requests / total requests</blockquote><blockquote>2. filtering criteria: both clients and drivers should be unbanned, and date range filter for 10/1 to 10/3</blockquote><blockquote>3. the users table has to be joined twice since this is a common table- once to get driver’s unbanned status, and again to get client’s unbanned status</blockquote><blockquote>4. finally we just apply group by on the joined table for the cancellation rate per day</blockquote><p>The exact query is attached as a screenshot from my bit.io editor.</p><p>Link to my DB with query: <a href="https://bit.io/nirmalbudhathoki/randomDB">https://bit.io/nirmalbudhathoki/randomDB</a></p><p>Leetcode link: <a href="https://leetcode.com/problems/trips-and-users/">https://leetcode.com/problems/trips-and-users/</a></p><p>You can also find the explanation for results in leetcode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pT2YHdPQCP4QbPpj88XCkw.png" /></figure><p><strong>Question 14: If 50% of candidates who receive a first interview receives their second interview; 90% of your friends who got a second interview felt good about their first interview; 70% of your friends who did not get a second interview also felt good about the first interview. If you feel that you did good in your first interview, what is the probability you will receive a second interview?</strong></p><p>This is a typical Bayes theorem question. Sometimes the first part is not given, and interviewer wants to see if you ask that question.</p><p>Let’s denote ‘pass’ as being invited to second interview, and ‘fail’ as not hearing back. So we need to find probability of passing when you are feeling good.</p><pre><strong>P(pass ∣ good) = ?</strong><br>P(pass) = 0.5<br>P(fail) = 1- 0.5 = 0.5<br>P(good ∣ pass) = 0.90<br>P(good ∣ fail) = 0.70</pre><pre>Let’s use the Bayes theorem such as:</pre><pre>P(pass | good) =<br>P(good ∣ pass) * P(pass) / P(good)</pre><pre>P(good) =<br>P(good | pass) * P(pass) + P(good | fail) * P(fail) <br>= 0.90 * 0.50 + 0.70 * 0.50<br>= 0.8</pre><pre>Now,</pre><pre><strong>P(pass | good) </strong>= 0.90 * 0.50 / 0.8 = 0.56</pre><p>So even if 90% of your friends feel good about their first interview and got called in for second round, there is actually only 56% chance of making to second round regardless of you feeling good about it.</p><p><strong>Question 15: SQL question asked in DoorDash data scientist interview: Given the below table schema:</strong></p><pre><strong>order_datetime</strong>: date read as string<br><strong>restaurant_id</strong>: int<br><strong>order_total</strong>: double</pre><p><strong>Write a SQL query that returns top 1% revenue generating restaurants for the month of May 2020. Total revenue is given by cumulative sum of all orders for that month. Make sure to use evenly distributed buckets while getting the top 1%.</strong></p><p>This question is testing your ability to use the ntile window function. Since it is asking the top 1%, the idea is to use ntile based on total order, and use 100 buckets, and return the top 1% result.</p><p>Always share the logical query first:</p><blockquote>1. calculate the total revenue for each restaurant using sum of order_total</blockquote><blockquote>2. Apply given filters for the date to get data for only May 2020</blockquote><blockquote>3. Use ntile window function with n=100 over total revenue sorted descending to get the top 1%</blockquote><blockquote>4. group by restaurant id</blockquote><blockquote>5. select the ntile = 1 to get the top 1% from above results</blockquote><p>I have added some sample data and the query can be ran in my bit.io account. Screenshot of the query is added as attachment.</p><pre>Query:</pre><pre>WITH orders as (<br>SELECT restaurant_id, sum(order_total) as total_revenue,<br>ntile(100) OVER (ORDER BY sum(order_total) desc)<br>FROM doordash_delivery<br>WHERE date_part(&#39;month&#39;,TO_DATE(order_datetime,&#39;MM/DD/YY&#39;)) = 05 and<br>date_part(&#39;year&#39;, TO_DATE(order_datetime,&#39;MM/DD/YY&#39;)) = 2020<br>GROUP by restaurant_id)<br>SELECT restaurant_id, total_revenue FROM orders<br>WHERE ntile =1<br>ORDER BY total_revenue desc</pre><p>Link to my bit io account: <a href="https://bit.io/nirmalbudhathoki/randomDB">https://bit.io/nirmalbudhathoki/randomDB</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sDFaZlx2TeVTgxBKOQ-OJg.png" /></figure><p><strong>Question 16: This question is popularly known as ants challenge. There are three ants at three ends of a triangle. Assuming they move with same speed, and starting at same time, what is the probability that they do not collide?</strong></p><p>This question is fairly simple. It is designed to test your ability of problem solving skill more than knowing probability. Once you understand and break it down, the problem itself is not hard at all, but sometimes we may overthink on it, specially in an interview setting.</p><p>Solution: Given the ants start at same time, and same speed, there are only two choices of direction for each ant at each corner of the triangle- it can either go clockwise or anti-clockwise. So we have 3 ants with 2 possible moves- this leads to following combinations:</p><pre>1. AAA<br>2. AAC<br>3. ACA<br>4. CAA<br>5. ACC<br>6. CAC<br>7. CCA<br>8. CCC</pre><p>This is almost like flipping coin three times and getting the combinations. You don’t have to list those choices, but I am just doing for explanation. You can directly calculate using 2³ = 8</p><p>Now the question has asked the probability of not colliding, which means all three have to move in same direction, giving us only two possible options for that- which are all moving anti-clockwise (AAA) or all moving clockwise (CCC).</p><p>Probability (not Colliding) = 2/8 = 0.25</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*virbiWo-GWz1xmaD1iipNQ.png" /></figure><p><strong>Question 17: This is one of the DS interview questions asked to cover the understanding of your data wrangling skills. How do you explain the group-by clause to someone who does not know any SQL ?</strong></p><p>Questions like these are asked to see your explainability skill. As a data scientist you will be working with a mix of stakeholders- both technical and non-technical. It is your responsibility to be able to explain your findings from data exploration or your insights/ results from model. This is actually a very simple question, yet sometimes I have seen folks struggle to explain it properly. Remember you are not asked to explain it from SQL perspective.</p><p>I always suggest to create an example. This is how I would approach this question. Imagine that we are tasked to measure average height of all students in a school based on various attributes. If we are simply asked to calculate the average height of all students, then we simply measure heights of all students and take an average.</p><p>What if we are asked to measure the average height by each grade ? Now the concept of group or category is introduced. Here grade is one categorical attribute. Now we group the students by grade, and then measure the average height for each grade. What if we are further asked to measure average height categorizing by their grade and gender ? Now we have two grouping attributes: grade and gender, therefore we have to form sub groups of boys and girls within each grade. Gender is another categorical attribute used. This is the exact concept of group-by clause in SQL, where we calculate aggregates (average height in our example) categorized by various attributes ( grade and gender in our example).</p><p><strong>Question 18: During regression, is having more independent variables always helps to have better performing model ? The goodness of fit (R2) test always shows higher score as we keep adding independent variables. How do you interpret that metric, and is that a good metric to measure ?</strong></p><p>This is one of most popular questions in data science interviews. It might show up with different variations , but the idea is to understand how R2 works, and why it is not always a good metric to use for evaluating your model.</p><p>Answer: No, having more independent variables will not always help us to have better performance, because some of the variables can infuse noise to the model than signal. However the R2 can be misleading since it always remains same or shows improvement as we add more independent variables. It never penalizes for adding redundant or noise variables.</p><p>We can use adjusted R2 instead, which is a better metric for goodness of fit than R2. The adjusted R2 will penalize for adding any unnecessary variables. The formula for both R2 and Adjusted R2 is attached in image. As we can see that adjusted R2 has ‘p’ as a factor to control the number of independent variables used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Czb5BNXjMQXO5PbBSSTuQ.png" /></figure><p><strong>Question 19: There is one biased coin (having both side tails) in a jar filled with 10 coins. You randomly picked one coin. You are not allowed to check both sides but you can only toss it. You tossed it five times and you got all tails. What is the probability that you pulled the biased coin ?</strong></p><p>Initially this question does not sound like Bayes theorem, but once you break it down, this is a Bayes problem.</p><p>Let the event of tossing fair coin is F, event of tossing biased coin is B, and event of tossing all 5 tails is T, which in this case has already happened.</p><p>Then the question is asking us to find probability of unfair coin when we got 5 tails in a row, or P(B | T) = ?</p><p>Using Bayes theorem,</p><blockquote>P(B|T) = P( T| B) * P (B) / P(T)</blockquote><blockquote>P(T|B) = 1 because we know that biased coin has both sides as tails</blockquote><blockquote>P(B) = 1/10 = 0.1 because there is only one biased coin in the jar</blockquote><blockquote>P(T) = P( T| B) * P (B) + P( T| F) * P (F)</blockquote><blockquote>P(F) = 9/10 = 0.9 because there are 9 fair coins in the jar</blockquote><blockquote>If the picked coin was fair, then probability of getting tail = head = 1/2 =0.5</blockquote><blockquote>P( T| F) = Probability of getting 5 tails from a fair coin = 0.5⁵ = 0.031</blockquote><blockquote>Now filling in all the values:</blockquote><blockquote>P(T) = P( T| B) * P (B) + P( T| F) * P (F)</blockquote><blockquote>= 1*0.1 + 0.031*0.9</blockquote><blockquote>= 0.1279</blockquote><blockquote>Then,</blockquote><blockquote>P(B|T) = P( T| B) * P (B) / P(T)</blockquote><blockquote>= 1 * 0.1 / 0.1279</blockquote><blockquote>= 0.78</blockquote><p><strong>Question 20 : What are various data imputing techniques you are aware of ? And follow up depending on whether you mention or not- how does the technique MICE works ?</strong></p><p>Asking about missing values and how to impute them is one of the most common questions in DS interviews. Most of the times we talk about the techniques like- dropping nulls, imputing with mean or median, imputing with forward or backward fill, using modeling like regression etc., however the least talked technique is MICE (Multivariate Imputation By Chained Equations).</p><p>Let’s take a simple example to understand the steps in MICE. Imagine the data with features: age, income, credit score and target variable whether loan is approved or denied. In real world the features will be more, but we are simplifying to understand the concepts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wcGCK8Fx4RnPGrbS1T-YKg.png" /></figure><p>Below are the steps to perform MICE imputing: Before we begin , we can drop target column- loan in this example.</p><blockquote>1. First we impute missing values for all features using one of the other applicable techniques mentioned above (mostly mean or median)</blockquote><blockquote>2. Remove the imputed value from one feature- age. Make age as a new target variable. The rows with age becomes training data, and rows missing age becomes our test data. We run regression using other existing features (income and credit score with imputation) to predict age.</blockquote><blockquote>3. Do the same for other features. This completes the iteration.</blockquote><blockquote>4. Calculate the diff of two dataframes, the goal is to get absolute difference as close as zero for all the imputed values. We can set the acceptable threshold. If the diff is higher than that repeat 2–3.</blockquote><p><strong>Question 21: What is self supervised learning ? How does it differ from semi-supervised learning?</strong></p><p>When we come across various types of machine learning algorithms, we mostly see — supervised, unsupervised, semi supervised and re-enforcement learning. One of the less talked algo type is self- supervised learning. As the name implies, it is the learning technique where model is supervised on its own from the data itself.</p><p>Having labels for supervised learning is desired but it’s not always feabile as it is expensive, resource intensive, and not always realistic. On the other hand, grouping or clustering is not always the type of task we want the model to learn, therefore having complete unsupervised learning is also not always ideal.</p><p>This is where the concept of self-supervised learning comes in. When we have big data, we want model to learn from the data and inherently create labels. For example- we as human beings have created many generalized knowledge or common sense from the things we have seen or perceived. When babies see a lots and lots of pictures of cat, and they have not seen cats in real, when they actually see a cat , they can relate back to their learnings and label it as a cat. This is the core idea 💡 that we want ML models to supervise itself to make prediction on unseen data, based on the data it has seen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*1BaJkR2M0aAE-4v5Cvkryg@2x.jpeg" /></figure><p>To answer the follow up question, semi-supervised learning is when we actually have some ground truth or labels , based on which we are creating more labels for all of our training data, then we can use any supervised models on it. On self- supervised learning we do not have any labels at all.</p><p><strong>Question 22 : This was one of the questions asked for FB / Meta ‘s Data Science role. Calculate each user’s average session time. A session is defined as the time difference between a page_load and page_exit. For simplicity, assume a user has only 1 session per day and if there are multiple of the same events on that day, consider only the latest page_load and earliest page_exit. Output the user_id and their average session time in seconds.</strong></p><pre>Table schema:</pre><pre>user_id: int → unique id for the user<br>timestamp: datetime → timestamp of the event<br>action: varchar → event type (whether page_load, page_exit, ….. etc.)</pre><p>Logical Query Plan:</p><blockquote>1. Since we are taking one session per user per day, we can define two CTEs- session start and session end</blockquote><blockquote>2. For session start- we consider max time for latest page load time</blockquote><blockquote>3. For session end- we consider min time for earliest page exit time</blockquote><blockquote>4. Join session start and session end tables on user id and date</blockquote><blockquote>5. Calculate the AVG on joined results</blockquote><p>I have added the data in my bit io account, table is called user_activity. All the SQL questions that I have been sharing can be practiced in the sample data added. Here is the link:</p><p><a href="https://bit.io/nirmalbudhathoki/randomDB">https://bit.io/nirmalbudhathoki/randomDB</a></p><p>Here is the query:</p><pre>WITH session_start AS (<br>SELECT<br>user_id,<br>DATE(timestamp) AS date,<br>MAX(timestamp) AS last_page_load<br>FROM user_activity<br>WHERE action = &#39;page_load&#39;<br>GROUP BY user_id, DATE(timestamp)), <br>session_end AS (<br>SELECT<br>user_id,<br>DATE(timestamp) AS date,<br>MIN(timestamp) AS first_page_exit<br>FROM user_activity<br>WHERE action = &#39;page_exit&#39;<br>GROUP BY user_id, DATE(timestamp))<br>SELECT<br>ss.user_id,<br>ROUND(AVG(EXTRACT (epoch from (se.first_page_exit - ss.last_page_load))),2) AS avg_session_time_sec<br>FROM session_start ss<br>JOIN session_end se<br>ON ss.user_id = se.user_id AND ss.date = se.date<br>GROUP BY ss.user_id</pre><p><strong>Question 23: Are you aware of categorical encoding techniques like label encoding and one-hot encoding ? Ensure to explain pros and cons of each method ?</strong></p><p>This is a very popular question in data science interviews, and I have seen lots of repetition with slight variations, but the core idea is explaining Label Encoding Vs. One-hot encoding.</p><p>Label encoding: Each categorical variable is assigned with a numerical value starting at 0. The unique categories are sorted first. For example- if color is the feature, and we have red , green and blue. The label encoding assigns 0 to blue, 1 to green and 2 to red. Some of the pros for this method is -</p><blockquote>1. Simple to understand</blockquote><blockquote>2. Dimensionality is not increased one per feature rather than one per category</blockquote><blockquote>3. It does not create sparse matrix of features</blockquote><p>One of the major drawback of this method is- since numbers are assigned to ordered list, the model might think higher value as higher importance. In our example assigning 2 to red does not mean red is more valuable color.</p><p>One-hot encoding: This is a technique of creating dummy variables where each category becomes one feature, and each row only gets one bit high(1) and rest low(0). In above example- blue, green and red becomes three features. For the row where value is blue, the features will look like below:</p><p>blue green red as 1 0 0</p><p>The advantage of one-hot encoding method is that all features will have same value (either 1 or 0) so there are no confusion of higher number to higher importance. However the drawbacks are:</p><blockquote>1. increase variables and sparsity</blockquote><blockquote>2. there is something called dummy variable trap, where outcome of one variable can be predicted with help of others, which will create the problem of multicollinearity.</blockquote><p><strong>Question 24 : Can you use Bayes theorem to test two hypotheses ?</strong></p><p>The answer is yes. Bayes theorem is used to calculate conditional probability given some information, or in other words, one event has already happened, and we are trying to calculate the probability of another event based on that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*qhfWG-t1QPYQvym1of-_aw.png" /></figure><p>Given two events A and B, the Bayes theorem states:</p><blockquote>P(A | B) = P( B | A) * P(A) / P(B)</blockquote><blockquote>where:</blockquote><blockquote>P(A |B) : is probability of A given B. From hypothesis perspective, we can formulate it as probability of hypothesis A, given the evidence B. This is also called posterior since we are trying to calculate it.</blockquote><blockquote>P(B |A) : is probability of B given A, also known as likelihood of observing the evidence, given the hypothesis is true.</blockquote><blockquote>P(A): is probability of A without any condition, also known as prior, because it is the prior to B happening or without B.</blockquote><blockquote>P(B): is probability of B without any condition, also known as marginal.</blockquote><p>Using the Bayes theorem, we can calculate posterior for both hypothesis against the same evidence, so we calculate:</p><p>P( hypothesis_1 | B) and P(hypothesis_2 | B). We can choose the one with higher of the two.</p><p>In fact Naive Bayes uses similar method to categorize the classes based on higher posterior using same features (evidence).</p><p><strong>Question 25: What are various types of bias in statistics ?</strong></p><p>This is one of the common data science questions that is company agnostic. When they are testing the machine learning breadth, these type of questions are pretty common. The interviewer is looking for your understanding of concepts and how you will explain it. Adding examples add a lot of value in these types of Qs.</p><p>There are various types and subtypes, but I would like to add below four:</p><blockquote>1. Selection Bias: This occurs when the data selected for sampling is not the true representative of population. For example- I collected the data about covid patients from one hospital because it was easier to access, however the scope of my research is all patients across entire united states.</blockquote><blockquote>2. Self- selection bias: This occurs when we let the subjects of interest to make their own selection for the research. For example- if we allow patients an option to opt in/ out for the study of efficacy of covid vaccine, people who don’t believe in vaccine will never chose to survey, and the samples may be incomplete.</blockquote><blockquote>3. Observer bias: This occurs when the observers who are recording the results of the experiment are driven by their preferences of desired or known outcomes. For example- medical researchers might be inclined to prove the vaccine is working if they are involved in the process of inventing, as well as testing.</blockquote><blockquote>4. Survivorship bias: This occurs when the samples chosen are only survivors, and some of the crucial attributes from non-survivors never made to analysis. For example- if we consider only the companies that survived the recession for the analysis of financial performance in tech industry, then we are infusing the bias based on survivors, and neglecting others.</blockquote><blockquote>One popular example of suvivor bias was analyzing the war flights to add armor on the spots that were hit by bullets. Infact the spots without holes were found to be more important to add armor because the flights that were brought down could have been hit on those weak spots; the ones with holes already suggest the flight can survive the impact.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*1eEAua5zVkckgWj5lFohiQ@2x.jpeg" /></figure><p>Other types of biases are automated bias, measurement bias, recall bias, implicit bias etc.</p><p>The important thing to consider while answering general discussion questions like this is- to keep it simple, but always share examples.</p><p><strong>Question 26 : How do you split the train-validation-test samples on time series data ? Can we apply regular Cross Validation technique ?</strong></p><p>This is another common data science interview questions that I have seen across different companies. Sometimes this can be a little tricky question since they are testing your understanding of how time series data should be treated while training a model.</p><p>When we are dealing with time series data, the random split or our regular cross validation technique is a terrible idea since that will lead into a data leakage problem. Time series prediction or forecasting usually deals with data where we must be cautious that test set or validation set is always the succeeding set, and training data is always the preceding set. If we fail to do so, some data from validation or test set will be leaked to training set, and that will lead to an overfit model.</p><p>Unlike regular k-fold cross validation, for time series, the training set will be increasing on a rolling basis. For example: if we have six months of data , and each month is represented as 1, 2, 3, 4, 5, 6 as subsets of data, then the 5-fold cross validation split will look like:</p><blockquote>Fold 1 : training [1], validation [2]</blockquote><blockquote>Fold 2 : training [1 2], validation [3]</blockquote><blockquote>Fold 3 : training [1 2 3], validation [4]</blockquote><blockquote>Fold 4 : training [1 2 3 4], validation [5]</blockquote><blockquote>Fold 5 : training [1 2 3 4 5], validation [6]</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*rgdUtW7idjrflUdePrFB7Q.jpeg" /></figure><p>As we can see that each iteration will give us more training samples to train the model, and the indexing has to be done by timestamp. At the end, average is taken just like regular cross validation, or we can even consider taking weighted average based on training data size. If we do weighted average, the assumption is higher the training samples, better will be accuracy so we must assign inverse weightage.</p><p><strong>Question 27: What is Central Limit Theorem (CLT) ? How does it help in Machine Learning ?</strong></p><p>This is another common question that comes when you are getting testing on Stats and ML breadth.</p><p>Here is the book definition: For a population with mean μ and standard deviation σ , if take sufficiently large random samples (n&gt;=30), then the distribution of the sample means will be approximately normally distributed, regardless of the population distribution.</p><p>Wait a minute ✋ — The definition sounds little confusing, there are too many ‘distribution’ words in there. However the core meaning is simple: Regardless of how the population is distributed, if we randomly take enough samples, the means of those samples will be closer to normal distribution.</p><p>Explaining the definition part of CLT it is not bad. However, interviewer is also looking your understanding of its application on ML. In fact the follow up question weighs more here.</p><p>Machine Learning models tries to learn pattern from training data, and generalize it to make prediction on test data. The training data will consists of deterministic (mostly static, if this happens then that will happen) and random (uncertain) information. If everything was deterministic, we do not need ML, we can simply use rule based system.</p><blockquote>Let y be the target variable and X (x1, x2, x3, …., xn) be input features. Now, y can be expressed as:</blockquote><blockquote>y = f(X) + Σ error</blockquote><blockquote>= deterministic function (X) + Σ error</blockquote><p>According to CLT, when we have enough training samples, y should follow normal distribution, and since y is the combination of deterministic results from a function (our model) and sum of random errors, the errors should also follow normal distribution. If errors are normally distributed, we can apply linear algorithms to the dataset for better results. If not, may be the data is incomplete or the model is incorrect and we have to explore non-linear algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULKkfVGGUy-V1qiXt69V4g.jpeg" /><figcaption>Normally Distributed Pile of Timber wood logs</figcaption></figure><p>PS: I googled CLT for images, and it came up as Cross Laminated Timber. So I created this pic of normally distributed timber logs pile :)</p><p><strong>Question 28 : What is power of test in hypothesis testing ? How is it measured and why it is important ?</strong></p><p>This was one of the questions asked by Meta in their DS interview. As I mentioned before, Meta’s Data Scientist jobs are more into analytics than core ML. Data Scientists are usually attached to the product team and work on various statistical modeling, therefore questions on hypothesis testing is very common with them.</p><p>Power of test is the probability of rejecting Null Hypothesis (H0), when the null is indeed false. So this is actually the measurement of making a right decision, unlike alpha (which is measuring the False Positives or wrong decision against null). Therefore it is referred as power of test- higher the better.</p><p>Power of test is calculated as 1- β , where β is the probability that we fail to reject null when we should have rejected it, or in other words it is the probability of we making Type II errors (False Negatives). Therefore doing 1- β, also referred as probability of True Positives, will give us what we need for measuring power of test.</p><p>In the context of binary classification, the power of a test is called sensitivity, its true positive rate.</p><p>Usually it is recommended to calculate ahead of experiment, which helps us to determine if the sample size we are considering for the experiment is good enough.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*gycU7b9ieI3ct8mugBRuXA.png" /><figcaption>Image credit: From the book ‘Statistics from A to Z: Confusing Concepts Clarified’ by Andrew Jawlik</figcaption></figure><p>Sharing a link for further study with examples. In fact for many stats topics, I always refer to Penn State’s Stats class, which is free :)</p><p>Link: <a href="https://online.stat.psu.edu/stat415/lesson/25/25.1">https://online.stat.psu.edu/stat415/lesson/25/25.1</a></p><p><strong>Question 29: This is a SQL question with two parts, asked in one of the Amazon interviews. Given a user_activity table:</strong></p><ol><li><strong>What is the number daily active users ?</strong></li><li><strong>List of users who spend at least one hour in a day.</strong></li></ol><blockquote>Schema:</blockquote><blockquote>user_id — unique ID of the user</blockquote><blockquote>timestamp- the timestamp of the activity</blockquote><blockquote>action- activity performed by the user</blockquote><p>Here is the logical plan that I recommend to discuss with interviewer before you start coding.</p><p>Number one rule of coding: NEVER jump into coding right away.</p><blockquote>Logical Query Plan:</blockquote><blockquote>1. The first question is pretty straight, just count the distinct user_id and group them by day</blockquote><blockquote>2. For the second question, we follow below steps:</blockquote><blockquote>2a. Create CTE (or subquery) to get the difference of Min and Max timestamp per user per day, which gives the time spent by user</blockquote><blockquote>2b. Apply filter of 1 hour to get required list</blockquote><p>Here is the link to my bit io, where I have the sample data and query for you to try it: <a href="https://bit.io/nirmalbudhathoki/randomDB">https://bit.io/nirmalbudhathoki/randomDB</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rj8Rtl9EakXRshThYACuxg.png" /></figure><p><strong>Question 30 : What is regularization ? If you are aware of L1 vs. L2 , how do they differ, and which one you prefer to use ?</strong></p><p>This is one of very common questions that I have seen across many companies. I can say the likelihood of this question being asked in machine learning section is fairly high.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*t0RUQ4g0Cx32CUwtnbsBdQ.png" /><figcaption>L1 vs. L2 Regularization</figcaption></figure><p>If I have to give one liner definition: Regularization is nothing but penalizing a model to avoid overfitting. Try to include below points into the discussion:</p><p>⭕️ There are two types: L1 ( also called Lasso) and L2 (also called Ridge). As we mentioned, regularization if adding penalty to model, L1 and L2 differs on how we add the penalty to the cost function. As you can see on the attached image, L1 adds the “absolute value of magnitude” of the coefficient as a penalty term to the loss function, while L2 adds the “squared magnitude” of the coefficient as the penalty term.</p><p>⭕️ Both regularizations have a hyper-parameter λ. Larger the λ the stronger penalty is applied. Likewise, if λ is zero, regularization is deactivated. Using too large values may lead to under-fit model, therefore we use cross validation to pick optimal value for λ.</p><p>⭕️ We can also combine and use both, and that is actually called elastic-net regularization.</p><p>⭕️ L1 can be more robust to outliers since we are dealing with absolute values, however L2 gives us differentiable function, which makes the function more computationally efficient, leading to a balanced minimization of the weights.</p><p>⭕️ L1 can also be used in feature selection as the use of absolute value may lead to drop some of the unnecessary features.</p><p>Lastly, thank you everyone for following me on this journey. I hope these questions will help you prepare for your data science interviews. Looking forward to do more of similar series in future 😊</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b43a51e38c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science for Cyber Security- Opportunities vs. Challenges]]></title>
            <link>https://medium.com/@nirmal.budhathoki/data-science-for-cyber-security-opportunities-vs-challenges-d516a8a01200?source=rss-500fd3d011d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d516a8a01200</guid>
            <category><![CDATA[machine-learning-in-cyber]]></category>
            <category><![CDATA[big-data-cyber]]></category>
            <category><![CDATA[ai-cyber]]></category>
            <category><![CDATA[predictive-modeling-cyber]]></category>
            <category><![CDATA[data-science-and-cyber]]></category>
            <dc:creator><![CDATA[Nirmal Budhathoki]]></dc:creator>
            <pubDate>Fri, 23 Aug 2019 22:30:56 GMT</pubDate>
            <atom:updated>2019-08-23T22:30:56.633Z</atom:updated>
            <content:encoded><![CDATA[<p><em>*Disclaimer</em>: This blog is focused on a very high level understanding of some opportunities and challenges of using data science in cyber security industry. If you are looking for more in-depth technical approach, this might not be the right blog. I value your time for what you will gain on what you spend. In future, I will definitely like to write some technical blogs in this subject area.</p><p>If you are still reading, that’s great. Lets cruise…The field of cybersecurity — both offensive and defensive security has come along a long way since the first computer virus called <em>Creeper, </em>created by Bob Thomas in 1971, that successfully disguised itself to move from one host to another within the <em>intranet</em> and leaving its trail on the hosts. The program was a research project, and it was not intended to harm anything.</p><p>In today’s era, where we have a massive spider web of <em>internet, </em>the attack surface is huge, and we are not talking about simple programs but advanced persistent threats. They are advanced because the attackers/hackers have a full spectrum of intelligence-gathering techniques at their disposal. They are persistent because they like to follow a ‘low-and-slow’ approach to intrude and maintain long-term access on the target devices. They are threats because they have both capability and intent to harm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*16Syotmb_qRhOZjKO9TDUQ.jpeg" /><figcaption>Cyber Security is a complex entity in itself (Source: <a href="http://www.cs.utsa.edu/~shxu/socs/">http://www.cs.utsa.edu/~shxu/socs/</a>)</figcaption></figure><p>On the other hand, Data Science, Big Data &amp; Artificial Intelligence (AI) with its sub-domains like Machine Learning and Deep Learning are making tremendous progress in many business areas like Sales and Marketing, Product Recommendation, Image Classification, Risk Analysis, Fraud Detections, Autonomous driving, and the list goes on. I have been working in the Cyber Security industry for while, and now I have also gained some knowledge and professional skills in Data Science. I used to always pause and ponder- where is that perfect intersection or connection of Data Science and Security. If you browse the internet to search for articles, research papers, and more importantly datasets to work on network/cyber security area, the chances of finding what you want or need are lower as compared to other areas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*dzCQcSrhl6U2IF6XI_ufiA.jpeg" /><figcaption>Artificial Intelligence connecting to Security</figcaption></figure><p>Now let’s talk about what are some of the biggest opportunities and challenges for this connection.</p><p><strong>Opportunities: </strong>Both data science and Cyber security jobs are in high demand. The mix of these two skillsets will definitely give you more strengths to become marketable. Besides the big names like Symantec and McAfee, there are many companies that are trying to pioneer and invent predictive models in cyber security space. Some of those companies are Cylance (now acquired by BlackBerry), Darktrace, Crowdstrike, Carbon Black (recently acquired by VMware), Obsidian Security, and more. You can already realize that- if big companies are acquiring the start-ups in billions of dollars, undoubtedly that subject area is hot in market.</p><p>Some of the use-cases where machine learning (both supervised vs. unsupervised approaches) will shine in cyber are: detecting and predicting zero-day vulnerabilities (<em>signature or rule based systems can only catch what was detected in past</em>), predicting malicious vs. benign activities (<em>spam vs. ham emails, malware vs. benign code, good vs. bad urls etc.</em>) , preventive/prescriptive maintenance of network devices, automating the repetitive tasks of incident management and security operation centers, predicting anomalous user behavior to prevent insider threats, risk modeling or scoring based on the knowledge of existing infrastructure and past data breaches, graph analytics (also called network or link analysis) for catching bots.</p><p><strong>Challenges: </strong>I would like to re-iterate the fact that the knowledge sources, and datasets are still very limited in the public space to learn from. It is sort of depressing to see that KDD Cup 1999 dataset is still the cleaner and easily accessible data when it comes to intrusion detection. One of the biggest challenges is the confidentiality of the data itself. No companies will like to share the data about their network. Some of the sources I have found to be useful are: <a href="https://vizsec.org/data/">https://vizsec.org/data/</a> , <a href="https://github.com/jivoi/awesome-ml-for-cybersecurity">https://github.com/jivoi/awesome-ml-for-cybersecurity</a>. If you find more, please share the links on comments. Using techniques like k-NN it is also possible to generate synthetic data to research. However, synthetic data will not be able to capture true variance needed for models to learn.</p><p>Some of the other challenges are dealing with high number of false positives. For example, if we predict a good code as malware, then that is a false positive. On the flip side, if we fail to predict a malware and we call it a good code, then that is false negative. There is always a tradeoff between false positives vs. false negatives. In cyber, unlike other fields, it is always hard to make that decision to lower one vs. the other. If the false positive rate is high, we are adding more time, resources and human intervention to resolve.</p><p>Another challenge is creating a baseline network activity when it comes to predicting anomalous behavior. In order to predict something as anomalous, we first need to know- what is a normal behavior. There are so many data points (logs) feeding into the system, and it is not easy to come up with a baseline. What we think as normal behavior in one subnet, might not be true for another subnet or across the networks, depending on the situations.</p><p>Beside some of these challenges, I still see some light at the end of the tunnel. Also the new findings and success stories in deep learning (like RNNs, especially LSTMs as log data is time-series) with automated feature engineering have given new hopes for cyber industry.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d516a8a01200" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>