<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shoresh Shafei on Medium]]></title>
        <description><![CDATA[Stories by Shoresh Shafei on Medium]]></description>
        <link>https://medium.com/@shoresh?source=rss-78c58cf441f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*n-w18mhebrYh0XecDVne2g@2x.jpeg</url>
            <title>Stories by Shoresh Shafei on Medium</title>
            <link>https://medium.com/@shoresh?source=rss-78c58cf441f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:21:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shoresh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Troubleshooting Data Science Interview Process]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/troubleshooting-data-science-interview-process-what-can-go-wrong-what-you-can-do-about-it-bd964be1813a?source=rss-78c58cf441f------2"><img src="https://cdn-images-1.medium.com/max/2600/0*o3Mrx685r4rsqngt" width="5000"></a></p><p class="medium-feed-snippet">What Can Go Wrong &amp; What You Can Do About It!</p><p class="medium-feed-link"><a href="https://medium.com/data-science/troubleshooting-data-science-interview-process-what-can-go-wrong-what-you-can-do-about-it-bd964be1813a?source=rss-78c58cf441f------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/troubleshooting-data-science-interview-process-what-can-go-wrong-what-you-can-do-about-it-bd964be1813a?source=rss-78c58cf441f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd964be1813a</guid>
            <category><![CDATA[data-science-interview]]></category>
            <category><![CDATA[data-science-careers]]></category>
            <category><![CDATA[career-advice]]></category>
            <dc:creator><![CDATA[Shoresh Shafei]]></dc:creator>
            <pubDate>Sat, 22 May 2021 19:28:52 GMT</pubDate>
            <atom:updated>2021-05-22T22:05:57.839Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Start Your Own Mafia!]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://shoresh.medium.com/start-your-own-mafia-5461110fac2b?source=rss-78c58cf441f------2"><img src="https://cdn-images-1.medium.com/max/1500/1*uYLs9rBC49OO68WEI_IR2A.jpeg" width="1500"></a></p><p class="medium-feed-snippet">I was a bit hesitant about using the title above until I found out that Reid Hoffman has already used it in his book. As you will see, by&#x2026;</p><p class="medium-feed-link"><a href="https://shoresh.medium.com/start-your-own-mafia-5461110fac2b?source=rss-78c58cf441f------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://shoresh.medium.com/start-your-own-mafia-5461110fac2b?source=rss-78c58cf441f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5461110fac2b</guid>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[paypal-mafia]]></category>
            <category><![CDATA[professional-development]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Shoresh Shafei]]></dc:creator>
            <pubDate>Sat, 15 May 2021 10:16:38 GMT</pubDate>
            <atom:updated>2021-05-15T10:16:38.029Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Thinking About Switching Career to Data Science? Pick the Right Strategy!]]></title>
            <link>https://medium.com/data-science/thinking-about-switching-career-to-data-science-pick-the-right-strategy-86880eb17181?source=rss-78c58cf441f------2</link>
            <guid isPermaLink="false">https://medium.com/p/86880eb17181</guid>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Shoresh Shafei]]></dc:creator>
            <pubDate>Sat, 08 May 2021 13:18:05 GMT</pubDate>
            <atom:updated>2021-05-08T23:49:16.315Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L0vrkb4RjNC7Tm3I" /><figcaption>Photo by <a href="https://unsplash.com/@bdchu614?utm_source=medium&amp;utm_medium=referral">Brendan Church</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Are you a professional thinking about a career transition to data science? Read on!</p><p>The field of data analytics continues to attract different groups from fresh graduates to those with years of experience in other fields. These groups, on the other hand, must have different strategies for transitioning to a career in data science.</p><p>Fresh graduates may have more flexibility in allocating their time to learning or sharpening their skills and might even want to take some time off to attend bootcamps, participate in data science fellowships or self-study. However, those with work experience can make a smoother transition if, rather than quitting or taking time off, leverage their competitive advantages they gain during their time in industry. To be specific:</p><p>To Become A Data Scientist:</p><ol><li><strong>DO NOT QUIT YOUR JOB!! <br></strong>Do not even think about it! It is rarely a good option, no matter how excited and determined you are!<br>Not only for many of us our job is the main source of income, getting to work on projects and interacting with people have a significant impact on our well-being, help us build and expand our network and connect us to new opportunities. Let me be clear, under no circumstances do not quit your job to prepare yourslef for becoming a data scientist!!</li><li><strong>Read, Consistently!</strong><br>As trivial as it looks, the gigantic volume of blog posts, articles, books, videos, tutorials, talks, slides and presentations, online courses, … are in your service, most of them for FREE, to guide you in the direction you want to go. Use them and use them often! Use these resources to not only learn new skills but also to learn more about the differences between career paths in data science- from product analysts, business analysts, statisticians, …-, get a sense of the trends in data science and to figure out where you see yourself a fit!<br>Read consistently: data science is a vast field and the more you read and learn, the more valuable you become for your future employer!</li><li><strong>Leverage Your Network</strong><br>i. Use your network to connect to data scientists and speak with them about their roles, experiences, projects, and a career path in analytics.<br>ii. Use your network to connect to the opportunities you may not be aware of! Let them know you want to transition to data science and you appreciate if they can help you along the way.<br>iii. Use your network to find roles with an overlap between your roles, responsibilites and skills and data science roles (More on this in #5)</li><li><strong>In Your Current Role</strong><br>In your current role, when it is possible, identify and commit to data analysis work. Projects that require data analysis, coding, statistical analysis, hypothesis testing, visualization, and even building machine learning models like linear and logistic regressions can teach you a ton about data science work. They can also help you to get a sense if you actually like these kind of tools and techniques. Remember, despite the hype, we spend a big chunk of our time <a href="https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/?sh=2cbc8c136f63">cleaning</a>, something that many of us are not excited about!!</li><li><strong>Your Experience Is Your Competitive Advantage And What Makes You Stand Out!</strong><br>A smooth way to transition to data science roles is to find a role where you can take advanatge of your current knowledge and skills and at the same time, allows you to broaden your data skills. This way you will use your competitive advantage (your skills and domain expertise coming from your current role) and in the meantime you expose yourself to learning opportunities in real-world scenarios (the data science projects you get to work on). Remember: few people may have your experience and domain knowledge that is required for jobs in a particular area. As an example, someone with a finance background has a much bigger appeal to a financial institution than someone with a background in healthcare!</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86880eb17181" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/thinking-about-switching-career-to-data-science-pick-the-right-strategy-86880eb17181">Thinking About Switching Career to Data Science? Pick the Right Strategy!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Quality for Everyday Analysis]]></title>
            <link>https://medium.com/data-science/data-quality-for-everyday-analysis-d3aa1442c31?source=rss-78c58cf441f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3aa1442c31</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[getting-started]]></category>
            <dc:creator><![CDATA[Shoresh Shafei]]></dc:creator>
            <pubDate>Fri, 23 Oct 2020 15:23:35 GMT</pubDate>
            <atom:updated>2021-05-15T10:19:48.568Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/getting-started">Getting Started</a></h4><h4>What is Data Quality, why it matters and how to do it right!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4SE4AG92W8IhdTVf" /><figcaption>Photo by <a href="https://unsplash.com/@randyfath?utm_source=medium&amp;utm_medium=referral">Randy Fath</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>In Data We Don’t Trust!</h4><p>A while ago, a friend of mine presented a compelling analysis that convinced the managers in a mid-size company to make a series of decisions based on the recommendations of the newly-established data science team. However, not long after, a million dollar loss revealed that the insights were wrong. Further investigations showed that while the analysis was sound, the data that was used was corrupt.</p><p>The realization that the data and not the analytical approach was the problem was a relief for my friend. However, the report led to a negative perception of his capabilities and damaged his reputation; something that took him a while to reclaim.</p><p>The story of my friend can happen to anyone using data. Excited about the state-of-the-art tools and techniques, the impact we want to make, or sometimes being rushed by deadlines, we may be tempted to ignore the fundamental step <em>prior to</em> any analysis: assessing Data Quality. Otherwise, no matter how advanced the methodology <em>bad data</em> guarantees a waste of time and resources and wrong and misleading outcomes. Data analysis and predictive models are the true manifestation of <em>garbage in, garbage out!</em> In Thomas Redman’s <a href="https://hbr.org/2018/04/if-your-data-is-bad-your-machine-learning-tools-are-useless">words</a>,</p><blockquote>If Your Data Is Bad, Your Machine Learning Tools Are Useless.</blockquote><p>Bad data happens for many <a href="https://blog.medisolv.com/articles/six-root-causes-of-data-fitness">reasons</a>: bugs in the ETL processes, manual entry, data integrations, loss of expertise, changing business logic and legacy data can all lead to data quality issues.</p><p>No matter how rigorous the ETL process is, it still cannot catch all the data issues. One reason is that many of these issues like inaccuracies can be discovered only when data is used. Therefore, very few data scientists, if any, are lucky to start their analysis with a clean and ready dataset. Datasets are usually full of errors and inconsistencies that need to be fixed. Consequently, while assessing the data quality is not as sexy as training a machine learning model, it is the most crucial part of any analysis.</p><h3></h3><p>Programming: 10% writing code. 90% figuring out why it doesn&#39;t workAnalyzing data and ML: 1% writing code. 9% figuring out why code doesn&#39;t work. 90% figuring out what&#39;s wrong with the data</p><p>Therefore, any user of data needs to have a fundamental understanding of the variety of issues that can happen to data and to be equipped with tools and techniques to detect, resolve and communicate these issues. That’s why assessing that data is <strong>complete</strong>, <strong>unique</strong>, <strong>timely</strong>, <strong>consistent</strong>, <strong>valid</strong> and <strong>accurate</strong> should be a prerequisite to any analysis.</p><p>Here I will introduce the basic concepts behind Data Quality, discuss the cost of bad data, review six dimensions of Data Quality assessment and go through tools and techniques that can be used to deal with quality issues when they arise.</p><h4>What is Data Quality?</h4><p>Data quality is the <a href="https://towardsdatascience.com/an-introduction-to-data-quality-951cc6fe0274">assessment of usefulness and reliability of data</a> to serve its purpose. To make it more clear, consider the following scenarios:</p><ul><li>When dataset does not include the most recent records.</li><li>When revenue numbers read from two different sources do not match!</li><li>When a dataset has many names for the same country like “US”, “United States”, “United States of America”, … plus all the typos.</li><li>When dataset has missing values.</li></ul><p>In the first example above, data is not timely (up to date). In the second one, data is not consistent, and in the third and fourth cases, data is not valid and complete, respectively. We will get back to these concpets in the context of data quality shortly.</p><h4>The Cost of Bad Data</h4><p>The consequences of using bad data go way beyond some incomplete rows or inconsistent records. Simply put,</p><blockquote><a href="https://hbr.org/product/data-driven-profiting-from-your-most-important-business-asset/1912-HBK-ENG">It costs 10x</a> as much to complete a unit of work when the data are flawed in any way as it does when they are perfect.</blockquote><p><a href="https://www.ringlead.com/blog/true-cost-of-bad-data/">Ringlead</a> visualizes the cost of bad data based on the number of data records. According to this estimate, the cost of a poor record is approximately $100: 100k bad<em> </em>records costs 2 million annually ignoring the new records added to the data. These number for organizations like Google with tremendous amount of data is significantly higher.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*4P6QSCcwGPWs2IuhQaneAQ.png" /><figcaption>Sources 1 and 3 in the chart are from Sirius Decisions — The Impact of Bad Data on Demand Creation and Small Business Association 2013 Census, respectively.</figcaption></figure><p>Another study by <a href="https://www.ibmbigdatahub.com/infographic/four-vs-big-data">IBM</a> estimated that the annual cost of bad data in the United States is around $3.1T- T for Trillion! Thomas Redman explains why:</p><blockquote><a href="https://hbr.org/2016/09/bad-data-costs-the-u-s-3-trillion-per-year"><em>The reason</em></a><em> bad data costs so much is that decision makers, managers, knowledge workers, data scientists, and others must accommodate it in their everyday work. And doing so is both time-consuming and expensive.</em></blockquote><p>Another eye-opening number is from <a href="http://www.data.com/export/sites/data/common/assets/pdf/DS_Gartner.pdf">Gartner</a>: poor data quality is responsible for the failure of 40% of all business initiatives and it also impacts the labor productivity by as much as 20%.</p><h4>Who Is Responsible for Data Quality?</h4><p>It is the overall organization’s responsibility to provide a holistic view of the data ecosystem by defining roles, processes and technologies to support data quality. Nevertheless, numbers show that the organizations are not doing a great job in this regard! According to <a href="https://www.edq.com/resources/data-management-whitepapers/2019-global-data-management-research/">Experian</a>, 29% of organizational data are inaccurate and at the same time 95% of businesses struggle to implement a data governance program. The same report states that 75% of respondents believed that data quality checks are the business users’ responsibility, not the IT infrastructure. Therefore, if you think you can trust the available checks in place and assessing the quality of data is not your responsibility, think again!</p><p>The other aspect of the problem is that not all the issues can be detected when data is created or stored. Two important moments in the <a href="https://hbr.org/2013/12/datas-credibility-problem?referral=03759&amp;cm_vc=rr_item_page.bottom">life of data</a> are when data is created and when it is consumed. While data is expected to be fixed at the creation moment, however, it will not be assessed until it is used. That’s why analysts, as the consumers of data, play an important role in detecting and resolving the flaws.</p><p>Therefore, I argue that <a href="https://hbr.org/2016/05/data-quality-should-be-everyones-job">data quality assessment is everyone’s job</a>; data users are encouraged to use their technical expertise and domain knowledge to detect inconsistencies and make sure that the analyses are based on reliable sources that pass quality control measures.</p><h4>Six Dimensions of Quality Data</h4><p>So far we reviewed the significance of data quality assessment and quantified the cost of bad data. We emphasized the critical role that analysts, or generally speaking data users, play in making sure data meet standards before it is <em>consumed</em>. In this section, we introduce the requirements that data should meet.</p><p>To assess the quality of a dataset, it is evaluated against 6 dimensions:</p><ol><li>Quality data is <strong>complete<br>Completeness</strong> of data means that the main features/attributes required for the analysis do not have missing values. Missing values can skew the analysis and lead to misleading trends.</li><li>Quality data is <strong>unique<br></strong>Duplicates have detrimental impacts. They take storage space, extend the running time of the code, and skew the analysis. Checking for <strong>Uniqueness </strong>guarantees that there is one only one record of each observation in the data.</li><li>Quality data is <strong>timely<br></strong>Addresses, phone numbers, emails and credit cards are constantly changing, and consequently, a record in the data that was valid last month may not be useful now. <strong>Timeliness</strong> ensures that data is up to date and the most recent records reflect the most recent changes.</li><li>Quality data is <strong>consistent</strong><br><strong>Consistency</strong> refers to the lack of contradictions within a dataset or between <a href="https://www.whitepapers.em360tech.com/wp-content/files_mf/1407250286DAMAUKDQDimensionsWhitePaperR37.pdf">different datasets</a>. A 5-feet tall newborn or mismatching revenue between sales and usage tables are both examples of inconsistency in the data.</li><li>Quality data is <strong>valid</strong><br><strong>Validity</strong> means that data has the right type, format and range. Data has the right type and format if it conforms to a set of pre-defined standards and definitions: 08–12–2019 is invalid when the standard format is defined as YYYY/mm/dd.<br>In the meantime, validity assures that data is in an acceptable range. Outliers, an important concept in Statistics and Data Science, are datapoints that do not meet this requirement.</li><li>Quality data is <strong>accurate</strong><br><strong>Accuracy</strong> reflects the correctness of the data such as the correct birth date and the accurate number of units sold, etc. <br>You should note that accuracy is different from validity: while the latter focuses on type, format and range, the former aims at form and content. As discussed in <a href="https://books.google.com/books?hl=en&amp;lr=&amp;id=x8ahL57VOtcC&amp;oi=fnd&amp;pg=PP1&amp;dq=Data+Quality:+The+Accuracy+Dimension&amp;ots=pZUtmo02AZ&amp;sig=phzHR5YwAUrcYLErN8PLd_JdO7A#v=onepage&amp;q=Data%20Quality%3A%20The%20Accuracy%20Dimension&amp;f=false">The Data Quality; The Accuracy Dimension</a>, accuracy is about correctness while the validity is about being among the possible accurate values: For an employee with $50k salary, a $45K salary is inaccurate but valid (in acceptable range).</li></ol><p>Before proceeding, we shold note that meeting 100% of the data quality requirements with real-life data is almost impossible. One workaround is to define a threshold for each dimension we defined above below which the data is assessed as poor quality. For example, 90% completeness allows the data to be used only if 10% of data is missing. The threshold, to some extent, depends on the business context. The higher the threshold, the higher quality data you have for you use case.</p><h4>Assessing &amp; Improving Data Quality with Analytical Techniques</h4><p>In this section, we go through some of the tools and techniques that analysts can use in order to check and improve the quality of their data. Examples in this section are in SQL.</p><p><strong>Completeness<br></strong>Incomplete data presents itself in the form of missing values. The metrics to measure it is the ratio of missing records for the features that are necessary (not optional) for the analysis. For example, if 30 out of 100 records in your analysis is missing the industry feature that you need for your marketing campaign, the completeness of your data is 70%.</p><p>First, check the number of missing values:</p><pre>SELECT <br> 1 — COUNT(col_A) / COUNT(*)<br>FROM<br> table_name;</pre><p>Alternatively, to count the number of empty cells</p><pre>SELECT <br> COUNT(col_A)<br>FROM<br> table_name<br>WHERE <br> col_A = ‘’;</pre><p>If the ratio is beyond your acceptable threshold, you may want to use one of the options below:</p><p>1. Use other data sources, if available: <br>When I was working on an unsupervised segmentation model for outbound marketing, I needed to know whether the business was engaged in e-commerce and in particular, what payment providers they use. The problem was that 90% of these records were missing in the database. We ended up testing datasets from multiple providers and merging one with our data. The whole process of fixing incomplete records took about a month.</p><p>2. Imputation<br>Imputation, a well-known approach in machine learning, is inferring the missing values from the existing ones, e.g. by replacing the missing values with mean, median or mode of the data.</p><pre>SELECT<br> COALESCE(col_A, AVG(col_A))<br>FROM<br> table_name;</pre><p>You can use IFNULL() as well. In Python you can use dropna(), fillna(), interpolate(), etc. For an overview of more advanced imputation methods, check out <a href="https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779">Data Imputation with examples</a>.</p><p>3. Manual fix <br>Handling missing data manually is risky and non-scalable and not recommended and you can only use it when there is a handful of missing values and you know the business logic well.</p><p>4. Dropping missing values <br>Only when you have a large dataset and few missing values such that it does not impact the analysis, you may drop the null values.</p><pre>SELECT<br> *<br>FROM <br> table_A<br>WHERE<br>col_A IS NOT NULL;</pre><p>Be aware of losing information and trends due to dropping important records (like outliers) with some missing values.</p><p><strong>Uniqueness<br></strong>Statistical packages are well-equipped to handle duplicate values. To check for duplicates values in SQL:</p><pre>SELECT<br> col_A, <br> col_B, <br> col_C, <br> COUNT(*) AS cnt<br>FROM<br> table_A<br>GROUP BY <br> col_A, col_B, col_C<br>HAVING cnt &gt; 1;</pre><p>To remove duplicates:</p><pre>SELECT<br> col_A, <br> col_B, <br> col_C<br>FROM<br> table_A<br>GROUP BY <br> col_A, col_B, col_C;</pre><p>Other approaches for removing duplicates are using DISTINCT(), ROW_NUMBER() or even self-join. In Python, duplicated() tells you about the duplicate values and drop_duplicates() takes care of removing them.</p><p><strong>Timeliness<br></strong>Data analysts need to check the timelines of data to make sure it is current! Using a table that is not being updated or a record that has changed but the change has not reflected in data leads to undesired outcomes. While there are no particular technical tools, checking the last time data was updated, or the minimum and maximum dates in the table should help.</p><pre>SELECT<br> MIN(date), MAX(date)<br>FROM<br> table_A;</pre><p>If your data is partitioned, you can replace the date above with partition_date.</p><p><strong>Consistency</strong><br>Inconsistency in data can happen within a dataset or between datasets. Programming languages make it easy to check datasets against one another and detect inconsistencies. Ultimately, you will have to use your business judgment, consult a domain expert or compare your data against a trustworthy source.</p><p>As a personal experience, I studied declined transactions in the Payment Industry where I had to resolve the inconsistencies between datasets from two different organizations. In the absence of a unique source of truth, I ended up researching the organizations that were maintaining these datasets and picked the one that had well-known customers and a better reputation.</p><p><strong>Validity<br></strong>1. Outliers<br>Several important statistics such as mean and standard deviation are sensitive to extreme values, a.k.a, outliers. Therefore, the data analyst needs to pay special attention to identifying and treating them: whether they are legitimate extreme values or have been recorded as a result of flaws in measurement tools.</p><p>From using the business knowledge and visualization techniques such as scatter plot or Box Plots, or using analytical techniques such as Density-Based Clustering and Isolations Forests, there are a myriad of techniques to treat outliers. Check out <a href="https://towardsdatascience.com/a-brief-overview-of-outlier-detection-techniques-1e0b2c19e561">this article</a> for a review.</p><p>Boxplot is a simple approach to identify outliers. Boxplots visualize a distribution by plotting the five important numbers: minimum, maximum, 25th percentile, median and 75th percentile of the distribution. To find outliers in the data, look at the regions that are 1.5 IQR (Interquartile Range) smaller than 25th or greater than 75th percentiles where IQR is the distance between 75th and 25th percentiles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qrf0gwIESQiS3qNo" /><figcaption>(Photo credit: <a href="https://towardsdatascience.com/understanding-boxplots-5e2df7bcbd51">Understanding Boxplots</a>)</figcaption></figure><p>Generally speaking, removing outliers is not recommended as not all outliers are the same! To distinguish between legitimate vs. illegitimate datapoints, sometimes the former is labelled as an anomaly. However, most of the time anomalies and outliers are used interchangeably.</p><p>2. Type and Format</p><p>For each attribute/feature in your data, check the format and data type (date, integer, string) to make sure it complies with standards. To check the data typein SQL</p><pre>SELECT COLUMN_NAME, DATA_TYPE <br>FROM INFORMATION_SCHEMA.COLUMNS<br>WHERE <br> TABLE_NAME = table_A</pre><p>In Python, you can simply use</p><pre>df.dtypes</pre><p>where df is your Pandas dataframe.</p><p>Skimming through the output is one way for a quick check on the data values and formats. In SQL, you can use something like</p><pre>SELECT <br> col_A, <br> COUNT(*)<br>FROM <br> table_A<br>GROUP BY <br> col_A<br>ORDER BY <br> col_A ASC<br>LIMIT 100</pre><p>where incorporating ORDER BYand LIMIT allows you to check for errors among the values that may be erroneous. In Python, value_counts()can give you some ideas about your inputs.</p><p><strong>Accuracy<br></strong>Accuracy in the data can be achieved using the domain knowledge as well as running your numbers against another trustworthy source. Conduct exploratory analysis and use visualizations and statistical approaches to identify trends and numbers that <em>do not make sense </em>and discuss them with a domain expert.</p><h4>Final Words</h4><p>In academia, and before submitting our article to journals for peer-review process, we used to wait a week or two and then review the manuscript once again. This lag normally led to substantial improvements in the quality of the work: We fixed errors we had not noticed before, improved the visualizations, made some paragraphs more clear and revisited data and calculations. Eventually, in the long run, it helped us build a reputation for creating quality work, and in the meantime reduced the time the review process took which in turn led to the quicker publication of the paper.</p><p>We are the best judge of our own work, and better than anyone else can identify the flaws and strengths of what we have created. The assessment of Data Quality is probably the most important step is delivering a high quality and impactful work that can help improve the business overall. I hope this short article helps raise awareness about the tools and techniques in our disposal to use in your analytical work and make your work even more quality!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3aa1442c31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-quality-for-everyday-analysis-d3aa1442c31">Data Quality for Everyday Analysis</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Physics to Finance: My First Year in Industry!]]></title>
            <link>https://medium.com/data-science/from-physics-to-finance-my-first-year-in-industry-bac58044d474?source=rss-78c58cf441f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bac58044d474</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[careers]]></category>
            <dc:creator><![CDATA[Shoresh Shafei]]></dc:creator>
            <pubDate>Sat, 27 Jun 2020 15:39:18 GMT</pubDate>
            <atom:updated>2020-12-13T15:24:16.958Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Physics to Finance: My First Year in Industry</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bKTmlgtdQ2XFQLhV" /><figcaption>Photo by <a href="https://unsplash.com/@christianchen?utm_source=medium&amp;utm_medium=referral">Christian Chen</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>This is a revised edition of the article I published almost 4 years ago. I have been lucky to experience an exciting journey so the answer to a question many of you may have is No! I never regretted my decision of leaving academia.</em></p><p>Many graduate and postdocs pursue non-research careers in Industry. This is the story of one of them.</p><p>It was April 25th, 2016 at 5 pm when I left my office at Duke University to start a new, and different journey Monday next week at 9 am.</p><p>After getting two masters and a Ph.D. in theoretical physics and spending two years as a postdoctoral associate in computational biochemistry, working with highly-respected scientists in their fields, publishing 12+ peer-reviewed papers and winning international awards, seeking a permanent academic position seemed like the logical next step.</p><p>During my studies, however, I was fortunate to participate in professional development activities organized by scientific organizations like the Optical Society of America and the International Society for Optics &amp; Photonics, a. k. a. SPIE, where I got in touch with industry leaders and learned about the challenges they deal with. The more I explored, the more I found myself fascinated by the <em>real-world</em> problems and opportunities outside academia.</p><p>The problem was that after years of creating a successful career in academia, one block at a time, I had little clue about the world out there and the roles fitted for a theoretical physicist; nevertheless, I was confident that there are not many positions that could make use of my knowledge of Quantum Mechanics or Electrodynamics!</p><p>It was unfortunate that learning about the second-order optical nonlinearities, that have many applications in medical fields and telecommunication, or photo-damaged DNA repair by photolyases, that makes life possible on earth, did not get many hiring managers excited. In fact, one of the first lessons I learned along the way was</p><blockquote>to focus on the skills I had acquired rather than the projects I had accomplished.</blockquote><p>Skills such as problem solving, numerical analysis, programming, publishing article and presenting.</p><p>Months of reading, learning, attending workshops, making phone calls, and coffee meetups to explore different industry positions directed me toward Data Science. My search for a new career path was in an era where, for the first time in history, we were able to make use of the gigantic amount of data we generate and store. This, in turn, has led to a high demand for skillful people in math, statistics, and programming to turn raw data into insights. Meanwhile, any segment of industry appreciates critical thinking, the ability to acquire new skills, effective communication and quick adaptation to a new workplace, many of which I had experienced before.</p><p>The need for data analysis is becoming such an indispensable part of the companies that a data science candidate can imagine a career in any sector, from a big enterprise in health care or aviation to startups in finance or sports.</p><p>In my case, the choice of industry sector was not a big concern. During my academic career, I had realized that working with the <em>right people</em> is way more fulfilling than working on the most exciting projects. The right people<em> </em>inspire and challenge you, and you enjoy taking the journey with them more than arriving at the destination. As Paul Graham said</p><blockquote>What matters is not ideas, but the people who have them. Good people can fix bad ideas, but good ideas can’t save bad people.</blockquote><p>[Update:] Add my past couple of experience to this article, I cannot emphasize enough on the importance of the people we get to work with. To make the story short, you will end up paying for your manager and/or team’s lack of experience or vision.</p><p>I met the CTO of Spreedly at a local analytics meetup. Impressed by his passion and encouragement, I submitted an application. Later I met with the company’s CTO and CEO, submitted work samples, attended the on-site interview and a couple of weeks later, I joined Spreedly to fill their first data science position.</p><p>The clear vision of the company’s executives for my role, among many things I enjoyed during the hiring process, played an important role in my decision. I had been interviewed by other companies that had little clue why they wanted a data scientist.</p><p><a href="https://www.spreedly.com/">Spreedly</a> was a 6-year-old startup in financial technology and facilitates online payments by storing and tokenizing payment methods, easing PCI-DSS (rules you have to follow if you are exposed to financial data) compliance for merchants and most importantly, by allowing the merchants to work with multiple payment gateways through one API. Therefore, at Spreedly, I have access to a database of transactions, payment methods, and payment gateways.</p><p>As the first data science hire in a startup, the opportunities to interact with and learn from other groups are limitless. I communicate with engineers, business development, marketing, customer success and design teams on a regular basis: a unique opportunity to learn about the other fields, how different departments operate, and the financial technology as a whole.</p><p>Here are some lessons I have learned along the way:</p><ol><li>Don’t be afraid to ask: this is probably the best advice I got after joining Spreedly. The more you ask, the faster you learn!</li><li>Institutional knowledge: domain knowledge is a part of the Data Scientist <a href="https://s3.amazonaws.com/aws.drewconway.com/viz/venn_diagram/data_science.html">Venn diagram</a>. However, I would like to put emphasis on Institutional Knowledge: without understanding how the company works, what type of data is stored, what each feature exactly represents, and how the business model has evolved over time, your data analysis could be flawed.</li><li>Be patient: this becomes even more important in a startup environment where the data infrastructure may not be ready yet. One solution is to consider working on several projects and make progress on one while waiting for the right time to start the others.</li><li>Business people are your best friends! They can help you identify the important questions to target with data. Use their business intelligence to work more effectively, and also gain valuable insights on the data interpretation. Make sure you discuss your findings with them before publishing the results or presenting them to clients.</li><li>Important vs. Interesting: It is easy to focus on problems that might seem interesting but they do not align with the mission or interest of the company nor they cannot benefit the clients. Hence it is always a good idea to invest in problems that actually matter than those that solely seem interesting.</li><li>Work with data-driven people: Data science is still a new field. So no matter how great you are at your job and how exciting your findings are, if you do not work with those that appreciate the power of data and are willing to use it for decision-makings, your efforts will be wasted.</li><li>Did I mention that working with a group of people with a great sense of humor is really fun?</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/0*bVFfLCGhJx3X_sqU" /></figure><p>The path from academia to industry has been studied by various <a href="http://www.nature.com/news/life-outside-the-lab-the-ones-who-got-away-1.15802">organizations</a>. According to <a href="https://www.vitae.ac.uk/news/press-releases/new-report-explores-the-occupations-of-former-research-staff-who-move-beyond-academic-research">Vitae</a>, only 18% of the young researchers who made this transition would go back to academia. I am confident I am one of the 82%.</p><p><em>Update (December 2020):</em> About 4 years after I published this post, I still get emails and messages from many graduate students and postdocs that are cautiously evaluating their exit route from academia to industry. Here is my message to these and other groups thinking about a change, or a shift in career: as long as you are genuine about what you do, evaluate each step, learn from past and use it for future, and are not afraid of ups and down of the path, you will be fine! Do it and you will find a great joy. And make sure to keep me posted about your story…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bac58044d474" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/from-physics-to-finance-my-first-year-in-industry-bac58044d474">From Physics to Finance: My First Year in Industry!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Paving the Way to Google!]]></title>
            <link>https://medium.com/data-science/paving-the-way-to-google-36844528e3d7?source=rss-78c58cf441f------2</link>
            <guid isPermaLink="false">https://medium.com/p/36844528e3d7</guid>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Shoresh Shafei]]></dc:creator>
            <pubDate>Sun, 08 Sep 2019 15:47:16 GMT</pubDate>
            <atom:updated>2020-11-26T02:57:12.757Z</atom:updated>
            <content:encoded><![CDATA[<h4>Lessons I Learned From Many Data Science Interviews</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XaAbVc9DGHGh3AM8" /><figcaption>Photo by <a href="https://unsplash.com/@pawel_czerwinski?utm_source=medium&amp;utm_medium=referral">Paweł Czerwiński</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There is a time when you should change your job, no matter how awesome the culture and how amazing the company is, and it’s when you stop growing.</p><p>When my time arrived, I interviewed many organizations in East and West Coast. From financial organizations to technology giants and from New York and Washington D.C. to Seattle and San Francisco I was open to new adventures. In the meantime, at the end of each step and each interview, I thought about what I, as a candidate, and the interview team could have done better.</p><p>What you are about to read is not exactly a roadmap to <em>how to land the next position</em> but more like a recollection of my thoughts about the whole process.</p><p>Most of what I am about to tell have already been published as LinkedIn posts. I know a lot of people are already going through or about to start the path I finished just a couple of months ago and my hope is that my notes on finding the right company, dealing with rejection, preparing for interviews, etc. make their journey more exciting than mine.</p><p>Here is a list of topics I discuss:</p><ol><li>It is not <em>easy</em> but it is <em>worth</em> it.</li><li>Job hunting vs. online dating</li><li>My criteria for evaluating a position!</li><li>How can you help companies find you?</li><li>The ultimate interview tip: forget solutions, build systems!</li><li>Technical skills for becoming a data scientist: my take</li><li>My system for preparing for technical interviews</li><li>Non-technical interview tips</li><li>To all my friends experiencing rejection!</li><li>After-rejection prescription</li><li>Want to get hired? Work on your technical skills! Want a promotion? Learn institutional knowledge!</li><li>To interviewers: meet the candidates at their comfort zone!</li></ol><p><strong>It is not <em>easy</em> but it is <em>worth</em> it!</strong></p><p>Recruiter emails, intro calls, technical questions, weekend projects, on-site visits, rejection letters, negotiations and so on seemed like a never-ending process; a loop I was repeating over and over again.</p><p>The process is exhausting but at the end, at least for my case, was rewarding and well-worth the efforts. I learned a ton and frankly, I became a better data scientist along the way. Interviewing companies in different industry segments, size and location, and discussing a variety of exciting problems they want to hire you for was eye-opening. Every interview helped me prepare for the next one by giving me a better prospect on the needs, the challenges and the expectations.</p><p><strong>Job hunting vs. online dating!</strong></p><p>In my opinion</p><blockquote>Job hunting is a lot like online dating; there are many singles out there but too few matches. Your job is not to marry the first one that goes on a date with you but to find the one with mutual interests and long-term prospects.</blockquote><p><strong>My criteria for evaluating a position!</strong></p><p>What are your criteria for picking the future organization? The brand, the location, compensation package, the job description, the projects you will be working on or even the perks and benefits and flexible work hours; which one has the most important factor? If you had multiple options in a variety of companies, what would you pick and why? Here is my take:</p><ol><li>People<br>Very few people can have a long-term impact on the quality of your life: your life partner, your adviser, manager and the people you work with. Make sure you choose them wisely and that can alone make your life a lot more fun and fulfilling!<br>When it comes to career, I have never regretted prioritizing people over projects/companies/universities. They can make or break you and your future; they can encourage/discourage you in every step along the way; they can become a major force in your personal or professional growth or the main obstacle towards achieving your goals.<br>And my next career move was no exception. Trying to find the right team to join was the top priority.</li><li>Responsibilities<br>Through conversations with the hiring team, I wanted to figure out whether I am excited about the role and responsibilities. Once I interviewed a company in ticketing space where the whole team were passionate about sports events. For me that do not follow sports, it was obviously not a right choice.<br>Don’t solely rely on the job description as it will only give you an idea about the role. This is the case especially for companies with less experience in hiring data scientists where the requirements in the job description may not be necessary for the role they are hiring for.</li><li>Growth opportunities<br>Does the future company encourages learning/improving skills and provide tools for it? Can I find new career opportunities in the same organization if I want to make a shift? Will I be able to easily find and connect to mentors?</li><li>Career path<br>How much the position I am considering aligns with my next 3–5 year goals. I used this principle to decline a couple of offers that financially made sense but were not aligned with my longer term goals.</li><li>Compensation package<br>It matters now and it will always matter. But it is not the most important factor. Money and financial well-being is a consequence of hard work and impact and if you find an organization that encourages learning and supports growth and rewards hard work, you should not worry about the rest!</li></ol><p><strong>How can you help companies find you?</strong></p><p>Consider the following facts:</p><ul><li>7.3 million openings in May 2019,</li><li>Thanks to technology anyone can apply for any position with a click,</li><li>And yet <a href="https://zety.com/blog/hr-statistics">63% of recruiters</a> believe talent shortage is their biggest problem.</li></ul><p>That’s why you need to become your own advocate &amp; help companies separate the signal from noise. The truth is that no one knows your goals and ambitions better than you and a list of skills &amp; projects on your resume simply won’t do it justice.</p><p>If you want to increase your chances of getting hired in the right role &amp; the right company, here are a couple of things to try:</p><ul><li>Your LinkedIn profile <br>It is like your saving account. It works for you even when you’re asleep. Keep it up-to-date, use the right keywords, and make it easy for the visitors to find relevant information about you. Remember that <a href="https://zety.com/blog/hr-statistics">87% of recruiters</a> use LinkedIn to check candidates.</li><li>Your Network<br>They are your connection for a reason. A referral can increase your chances by 5 times so message them and respectfully ask for referral, guidance, advice, etc. Stop after two trials per connection. You have more important stuff to do.</li><li>Cold email: <br>Here is what I did: Find &amp; apply for the right position on LinkedIn. If the recruiter was listed in the job page, I sent them a message (a sample in comments). <br>A sample letter I used for cold emailing: <br>Hi […], <br>Reading the job description […] role at […], I found a great overlap between my current responsibilities and the job description. Please allow me to elaborate: <br>In the past couple of years, I have</li></ul><p>— worked in […],<br> — gained 3 years of experience in […],<br> — defined and measured product metrics,<br> — become experienced with […]<br> — trained machine learning models,<br> — prepared […] analysis,<br> — extensively collaborated with business teams, and<br> — built analytics dashboards.</p><p>I applied for this position [job ID here] and I hope you will consider my application. <br>Best <br>Your name, <br>Current title</p><p><strong>The ultimate interview tip: forget solutions, build systems!</strong></p><p>The tip is the realization that in most interviews, candidate and interviewer are seeking two different things: while the candidate focuses on the solution, the interviewer is seeking the system. The former relies on technical abilities while the latter is about the process you build to demonstrate them.</p><p>The system is a step by step process you use in order to help the interviewer understand your approach to evaluate and solve the problem and communicate your thinking and your ideas. Something she is way more interested in than just the right answer! If the solution is the story, your system is how you tell your story and it’s the best testimonial to your depth of knowledge and how well you communicate it.</p><p>As an example, I discuss my system for answering a technical question, like a SQL problem:</p><ol><li>Repeat the question and what you should deliver to the interviewer to make sure you both are on the same page.</li><li>Ask for a minute or two and think about the solution. Not only does this give you time to gather your thoughts but it also leave a positive impression on the interviewer: you think before you speak!</li><li>Explain your strategy and check your assumptions with the interviewer. Like “is it safe to assume there are missing values or duplicate values in the data”?</li><li>Keep the interviewer engaged: write the code as readable as possible and at each step explain what you are doing.</li><li>Review your solution and also provide suggestions on how to make it better.</li><li>Ask for feedback! On multiple occasions the feedback helped me improve the code. Sometimes it led to interesting conversations around other possible solutions to the problem.</li></ol><p><strong>Technical skills for becoming a data scientist: my take!</strong></p><p>I used to ask what skills should I acquire and how much should I know to transition to a career in data science. And most of the time the answers were not good enough!</p><p>Leaving how you want to learn them to you, here is my take:</p><ol><li>SQL <br>If you’re comfortable working with the Window Functions, you’re good to go!</li><li>Statistics <br>You can calculate confidence intervals, design an experiment and explain p-values. You’re comfortable with ANOVA, and understand the assumptions behind linear regression.</li><li>Python<br>Write simple code to handle strings, understand Numpy library and become good enough handling large datasets with Pandas and visualizing the results with Matplotlib.</li><li>Machine learning<br>You understand the basics well: from the bias-variance trade off, the difference between parametric and non-parametric algorithms and categorical vs. regression models to over- and under-sampling methods, etc. You understand how the traditional algorithms work: from Linear regression and SVM to ensemble methods. You have working proficiency in at least two algorithms such as logistic regression and random forests. You understand the different metrics for assessing your model and how to pick the right one for your problem.</li></ol><p><strong>My system for preparing for technical interviews</strong></p><ul><li>20 min<br>I read the job description, highlighted the key points and thought of how my experiences relate. This may not seem technical but this part gave the technical questions a context.</li><li>30 min<br>Through the initial conversation with the recruiter, I tried to figure out what type of technical interview I should expect.</li><li>30 min<br>I research the company online and if possible, the division they are hiring for and think about the projects and the problems they are trying to solve. If unfamiliar with the domain, I spend a couple minutes researching that field.</li><li>10–15 min<br>I checked out the interviewer (if given) on LinkedIn to learn about her expertise. It’s funny that for some reason, the less I did this, or basically the less I cared about who interviewed me, the better I did during the actual interview.</li><li>45–60 min<br>I checked out the company’s interview questions on Glassdoor and few other websites. If it was a phone interview, I only reviewed the phone interview questions and so on.</li><li>60–120 min<br>Depending on the interview, I reviewed my SQL/Python/Statistics/Machine Learning notes (I’ll share them in the near future).</li></ul><p><strong>Interview non-technical tips</strong></p><ul><li>Act like a topic expert, not a job seeker! <br>Treat the interview like a conversation and the interviewer like your colleague. Relate the questions to your past experiences, tell a story, keep the interviewer engaged and ask her questions like: how would you/ your company deal with this situation?</li><li>Phone interview at your peak time<br>I realized evenings after work hours worked better for me. Figure out your own.</li><li>There is a good chance you’re as qualified as many of your interviewers. Be humble but confident.</li><li>Data science interviews are easy if you are ready technically (assuming your skills match the job requirements)! The more you practice the better prepared you are.</li><li>Don’t worry about rejection: you almost always get a second chance (after a while) and in the meantime, there are many other great companies out there.</li><li>You have been told this a zillion times, but again, remember, you also interview the company representatives. Something you don’t ask about or don’t notice now, can become a huge problem later.</li><li>Keep the interviews with companies you care about the most to the last so you have enough time to learn from and fix past mistakes.</li></ul><p><strong>To all my friends experiencing rejection</strong></p><p>I feel your pain &amp; frustration! I went through the same process a couple of weeks ago and I have been rejected by many companies. When confronting frustration, we normally blame ourselves for not being good enough. However, at least for the majority of the people I know, it is far from the truth. Through many interviews, which for a while seemed never-ending, I came to the realization that most of my frustration was with two groups of companies:</p><ul><li>Those (normally small to mid-size) with no prior experience in hiring data scientists. Here you may see the least accurate job descriptions, and some irrelevant questions.</li><li>Those with data science team(s) but without a well-structured processes around hiring. Here you may have to answer very specialized questions based on who interviews you and the project the interviewer is working on/is interested in.</li></ul><p>Don’t blame yourself for something you do not have much control over! Instead, learn from every interview, move on and get ready for the next one. There are many companies out there that need your talent, passion and hard work. The more experience you gain, the better you manage any circumstances.</p><p><strong>After rejection prescription</strong></p><ul><li>You did great but the interviewer didn’t get you!</li><li>The problems were confusing. They were not interview-type questions.</li><li>I was not given enough explanation or time.</li><li>You didn’t deserve rejection!</li><li>And so on</li></ul><p>You are absolutely right on all accounts above. But ONLY for the first half an hour after the rejection! After the first 30 minutes, and before forgetting about the content of the interview and possible flaws in your performance, from the way you framed the problem, your approach to the SQL question or not being familiar with the company/product well-enough, think carefully about what went wrong! Write it down and make sure you take care of it. The truth is there are many things that can go south during an interview and it’s okay! What is not okay is to make the same mistake twice! You just don’t have time for that…</p><p><strong>There are many great companies out there so don’t worry if you messed up an interview!</strong></p><p>In early days, a couple of interviews that I screwed up with some of the technology giants left me in despair. I was afraid that if it continues like that, I will run out of great options and may have to end up somewhere that wasn’t a step forward in my career. Meanwhile, I already had a position in a great company working with an amazing group of people and didn’t want to change my job under any circumstances.</p><p>A couple of things helped me overcome this fear. For example, a simple search on LinkedIn shows a myriad of opportunities from great but less-known organizations with interesting and sometimes disruptive technologies that many of us have never heard about. The truth is that the world is not limited to Facebook or Google. In fact, there are more-than-you-can-even-imagine companies out there that have a healthy culture and supportive teams willing to help you grow and be the best of you.</p><p>Base your important decisions, including taking on a new role, on your hopes rather than your fears: if you are seeking and haven’t found the right organization yet, don’t give up. Keep looking!</p><p><strong>Want to get hired? Work on your technical skills! <br>Want a promotion? Learn institutional knowledge!</strong></p><p>Domain Knowledge &amp; Institutional Knowledge: When my past company’s CEO encouraged me to work on Declined Transactions as my first project after joining the team, I was disappointed! Right out of academia, I was hoping for a complex project like building the best machine learning model. Instead, I was assigned to calculating the ratio of declined to all transactions, to oversimplify it. What I did not know was that my assignment was, not if the most important, but one of the most important problems in Payment Industry.</p><p>What I was missing was not technical skills. Technically, I was capable of doing any type of analysis. 14 publications in 3 different fields of science and in a variety of demanding topics from Cosmology to Nonlinear Optics and Computational Biochemistry, winning multiple research awards and working on my data science skills for about a year, I was ready to do what it takes to showcase my technical abilities; to shine! The missing piece was not math and stats. It was the domain knowledge and business acumen!</p><p>In addition to technical capabilities, data scientists need to develop other skills that allow them to identify the projects that impact the business and empower the stakeholders and prioritize them over any other type of work. To get there, i.e. to develop business acumen, the coexistence of two pieces are necessary; domain knowledge and institutional knowledge!</p><p>Domain knowledge helps you understand the main concepts of your industry sector. Institutional knowledge, on the other hand, focuses on the company you work for: its vision, its competitive advantage, how it operates &amp; generates revenue, its priorities, the history and future plans and so on.</p><p>A domain knowledge expert in Payment Industry may identify fraud and declined transactions as very important problems. Institutional knowledge helps the expert pick one over the other based on which aligns better with the company’s vision.</p><p>Talking to experts in your field, people in your company, subscribing to newsletters, reading the company documents, all and all can help you develop your domain and institutional knowledge.</p><p>Listen to the conversations carefully, remember names and master important concepts mentioned and repeated over and over. And be patient!</p><p>And here is the good news: I have come to the realization that while we get hired for out technical skills, what makes us irreplaceable is the institutional and domain knowledge. It is not normally difficult to replace someone with someone with better technical skills, but it would be very difficult to replace someone that knows the domain well and the institution she works for very well. Therefore, while in the<strong> </strong>short run, your competitive advantage is your technical skills; in the long run, your domain and institutional knowledge become the one!</p><p><strong>Meet candidates at their comfort zone!</strong></p><p>I argue that If you want to see the candidates at their best, evaluate them in their comfort zone: a suggestion to interviewers!</p><p>During the interview process, phone or on-site, the candidates’ brains are processing a zillion different things at once. Under these circumstances, if your judgment is solely based on their capabilities in understanding problems and challenges related to the new role, you have not evaluated them to the best of their abilities.</p><p>Instead, I suggest that a part of the interview should focus on moving to candidates’ comfort zone and challenging them by asking detailed questions about their past work. This indeed should go beyond a short introduction that normally takes place at the beginning of the interviews.</p><p>This will help build a more comprehensive picture about candidates’ cognitive and technical abilities, their thought process and approaches to the problem, how they deal with challenges and how they are presenting them to you.</p><p>While this requires experienced interviewers, it makes the process fair and helps organizations attract applicants who may end up being a better fit for the role.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36844528e3d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/paving-the-way-to-google-36844528e3d7">Paving the Way to Google!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>