<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Data Science Earth - Medium]]></title>
        <description><![CDATA[We will make the world a better place with the science of data. - Medium]]></description>
        <link>https://medium.com/datasciencearth?source=rss----e0aba57873ec---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Data Science Earth - Medium</title>
            <link>https://medium.com/datasciencearth?source=rss----e0aba57873ec---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:24:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/datasciencearth" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[20 Useful Python Libraries for Data Science Projects]]></title>
            <link>https://medium.com/datasciencearth/20-useful-python-libraries-for-data-science-projects-f393faad4419?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/f393faad4419</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python-list]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[ferhatmetin]]></dc:creator>
            <pubDate>Thu, 10 Jun 2021 10:06:45 GMT</pubDate>
            <atom:updated>2021-06-10T10:06:45.039Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AS5f709K3Lqwl7UflIhiTQ.png" /></figure><p>Scikit-learn, Numpy, Pandas, Matplotlib, Plotly, Bokeh and Seaborn are some of the common Python libraries used in the field of data science. Let’s see some other libraries that can be useful for data science projects.</p><h3>1-Feature Engine</h3><p>Feature-Engine is a library for feature engineering and it allows us to select the variables we want to transform so it’s very easy to apply different engineering procedures to different feature subsets. Also, Feature-Engine transformers can be assembled within the Scikit-learn pipeline.</p><p>Feature-Engine includes transformers for:</p><ul><li>Variable transformation</li><li>Variable selection</li><li>Categorical variable encoding</li><li>Missing data imputation</li><li>Discretisation</li><li>Outlier capping or removal</li><li>Variable creation</li></ul><p>For the documentation and more:</p><p><a href="https://feature-engine.readthedocs.io/en/1.0.x/">Feature-engine: A Python library for Feature Engineering for Machine Learning - 1.0.2</a></p><h3>2-Yellowbrick</h3><p>Yellowbrick is a library for machine learning visualization. It extends the Scikit-Learn API to make model selection and hyperparameter tuning easier. It’s using Matplotlib for visualizations. Some visualizers: Precision-Recall Curves, Confusion Matrices, Residuals Plot, K-Elbow Plot, Silhouette Plot, Learning Curve, t-SNE Corpus Visualization…</p><p>For the documentation and more:</p><p><a href="https://www.scikit-yb.org/en/latest/">Yellowbrick: Machine Learning Visualization - Yellowbrick v1.5 documentation</a></p><h3>3-PDPbox</h3><p>Partial dependence plots shows the marginal effect one or two features have on the predicted outcome of a machine learning model (J. H. Friedman, 2001). PDPBox is a library for partial dependence plots.</p><p>For the documentation and more:</p><p><a href="https://github.com/SauceCat/PDPbox">SauceCat/PDPbox</a></p><h3>4-Eli5</h3><p>Eli5 is a Python package that helps to debug machine learning classifiers and explain their predictions. It helps to explain predictions of black-box estimators. It provides support for Scikit-Learn, Keras, xgboost, LightGBM, CatBoost…</p><p>For the documentation and more:</p><p><a href="https://eli5.readthedocs.io/en/latest/">Welcome to ELI5&#39;s documentation! - ELI5 0.11.0 documentation</a></p><h3>5-Researchpy</h3><p>Researchpy creates Pandas DataFrames that contains relevant statistical testing information that is required for academic research. It uses Pandas, Scipy, Numpy, Statmodels…</p><p>For the documentation and more:</p><p><a href="https://researchpy.readthedocs.io/en/latest/">Welcome to researchpy&#39;s documentation! - researchpy 0.3.2 documentation</a></p><h3>6-LIME</h3><p>LIME (Local Interpretable Model-agnostic Explanations) is an algorithm to explain predictions of black-box estimators and LIME library is one of the most popular Python libraries for model explainability.</p><p>For the documentation and more:</p><p><a href="https://github.com/marcotcr/lime">GitHub - marcotcr/lime: Lime: Explaining the predictions of any machine learning classifier</a></p><h3>7-SHAP</h3><p>SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the output of any machine learning model and Shap library allows us to visualize shapley values.</p><p>For the documentation and more:</p><p><a href="https://github.com/slundberg/shap">GitHub - slundberg/shap: A game theoretic approach to explain the output of any machine learning model.</a></p><h3>8-Missingno</h3><p>Missingno provides a small toolset of flexible and easy-to-use missing data visualizations and it helps us to get a quick visual summary of our dataset.</p><p>For the documentation and more:</p><p><a href="https://github.com/ResidentMario/missingno">GitHub - ResidentMario/missingno: Missing data visualization module for Python.</a></p><h3>9-Imblearn</h3><p>Imbalanced-learn (Imblearn) is an open source library relying on scikit-learn and provides tools when dealing with classification with imbalanced classes.</p><p>For the documentation and more:</p><p><a href="https://imbalanced-learn.org/stable/user_guide.html#user-guide">User guide: contents - Version 0.8.0</a></p><h3>10-Dataprep</h3><p>DataPrep allows us to prepare and visualize our data with a few lines of code. I think it could be an alternative to Pandas Profiling.</p><p>For the documentation and more:</p><p><a href="https://github.com/sfu-db/dataprep">GitHub - sfu-db/dataprep: Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.</a></p><h3>11-Dython</h3><p>Dython is a set of data analysis tools that can let you get more insights about your data. Dython automatically find which features are categorical and which are numerical, compute a relevant measure of association between each and every feature, and also plot it all as a heatmap.</p><p>For the documentation and more:</p><p><a href="http://shakedzy.xyz/dython/">dython</a></p><h3>12-Statsmodels</h3><p>Statsmodels is a Python module that includes classes and functions for the estimation of many different statistical models as well as for conducting statistical tests, and statistical data exploration.</p><p>For the documentation and more:</p><p><a href="https://www.statsmodels.org/stable/examples/index.html">Examples - statsmodels</a></p><h3>13-Mlxtend</h3><p>Mlxtend (machine learning extensions) is a library that contains useful tools for data science tasks.</p><p>For the documentation and more:</p><p><a href="http://rasbt.github.io/mlxtend/#welcome-to-mlxtends-documentation">Home - mlxtend</a></p><h3>14- SciPy</h3><p>SciPy is an open-source software for mathematics, science, and engineering. It contains useful functions in areas such as linear algebra, optimization, signal processing and statistics.</p><p>For the documentation and more:</p><p><a href="https://www.scipy.org/docs.html">Documentation - SciPy.org</a></p><h3>15-Pandas Profiling</h3><p>It generates reports from a pandas dataframe for a quick exploratory data analysis.</p><p>For the documentation and more:</p><p><a href="https://github.com/pandas-profiling/pandas-profiling">GitHub - pandas-profiling/pandas-profiling: Create HTML profiling reports from pandas DataFrame objects</a></p><h3>16-Sweetviz</h3><p>Sweetviz is an open-source Python library that generates high-density visualizations to Exploratory Data Analysis with just two lines of code. Output is a fully self-contained HTML application.</p><p>For the documentation and more:</p><p><a href="https://github.com/fbdesignpro/sweetviz">GitHub - fbdesignpro/sweetviz: Visualize and compare datasets, target values and associations, with one line of code.</a></p><h3>17-Dtreeviz</h3><p>Dtreeviz is a library for decision tree visualization and model interpretation. It currently supports Scikit-Learn, XGBoost, Spark MLlib and LightGBM trees.</p><p>For the documentation and more:</p><p><a href="https://github.com/parrt/dtreeviz/">parrt/dtreeviz</a></p><h3>18-category_encoders</h3><p>It is a library that includes scikit-learn-style transformers for encoding categorical variables into numeric with different techniques. Some of the encoders : Count Encoder, CatBoost Encoder, James-Stein Encoder, Target Encoder …</p><p>For the documentation and more :</p><p><a href="https://contrib.scikit-learn.org/category_encoders/">Category Encoders - Category Encoders 2.2.2 documentation</a></p><h3>19-tslearn</h3><p>tslearn is a package that provides some machine learning tools for time series analysis . This package builds on scikit-learn, numpy and scipy libraries.</p><p>For the documentation and more:</p><p><a href="https://tslearn.readthedocs.io/en/stable/quickstart.html">Quick-start guide - tslearn 0.5.1.0 documentation</a></p><h3>20-sktime</h3><p>It is a unified framework for machine learning with time series. It provides specialized time series algorithms and scikit-learn compatible tools to build time series models.</p><p>For the documentation and more:</p><p><a href="https://www.sktime.org/en/latest/">Welcome to sktime - sktime documentation</a></p><p><strong>To see other published articles : </strong><a href="https://medium.com/datasciencearth"><em>https://medium.com/datasciencearth</em></a></p><p><strong>To see published Turkish articles :</strong> <a href="https://www.datasciencearth.com/"><em>https://www.datasciencearth.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f393faad4419" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/20-useful-python-libraries-for-data-science-projects-f393faad4419">20 Useful Python Libraries for Data Science Projects</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DATA SCIENCE SURVEY 2021 — INSIGHTS]]></title>
            <link>https://medium.com/datasciencearth/data-science-survey-2021-insights-59582c707d4d?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/59582c707d4d</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[tools]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mehmet Sezer]]></dc:creator>
            <pubDate>Thu, 20 May 2021 09:47:34 GMT</pubDate>
            <atom:updated>2021-05-20T09:47:33.997Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>DATA SCIENCE SURVEY 2021 — INSIGHTS</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7aGASjXzIA22tLOaDPe_HQ.jpeg" /><figcaption><em>View interactive dashboard from </em><a href="https://public.tableau.com/profile/data.science.earth#!/vizhome/DataScienceEarthSurveyDashboard2021/2021Dashboard1"><strong><em>here</em></strong></a><strong><em>.</em></strong></figcaption></figure><h3><strong>Introduction</strong></h3><p><strong>Data Science Earth </strong>was established on the purpose of producing <strong>high-level data science solutions</strong> in 2019. The<strong> community</strong> whose members are entirely<strong> volunteers</strong> primarily aims to develop data awareness and ensure the correct use of data power in a globalizing competitive environment.</p><p><strong>Data Science Survey Team</strong>, which is part of organization, released its first survey in 2020 to obtain some exploratory insights about Data Scientists all over the world. This year, they conducted the second survey for 2021. <strong>Survey aims</strong> not only to explore some insights but also <strong>to be a guide to all Data Scientists</strong> who are just new in the Data Science ecosystem and who are already in this sector and want to improve themselves.</p><p>The <strong>guide answers many frequently asked questions about Data Science</strong> including:</p><ul><li>Where should new generations start?</li><li>What more can be added to current skills?</li><li>Which tools are necessary to learn, and which ones will most probably lead in the future?</li><li>What are the options to improve our skills?</li><li>What is the purpose of data science?</li></ul><p>When it comes to results of the survey, the dataset includes many different perspectives from different regions. The different regions imply <strong>almost 20+ countries</strong> and many cities all over the world. After our Survey Team had handled the <strong>pre-processing of dataset</strong>, they presented <strong>interactive dashboards</strong> to explore some insights for us.</p><p>Let’s continue step by step</p><h3><strong>What is the Demography of Data Scientists?</strong></h3><p>Here are some of our discoveries about demography of data scientists all over the world!</p><p>You can find some information about <strong>country, age, gender, major, education level</strong> and <strong>business level </strong>of respondents;</p><ul><li>Even if the <strong>major of respondents</strong> come from <strong>Turkey</strong>, this lasts with <strong>USA, India, Azerbaijan,</strong> and <strong>Australia</strong>. <strong>Brazil</strong> is also <strong>taken a place</strong> this survey with <strong>2021</strong>.</li><li>Although the <strong>males</strong> are clearly <strong>dominant</strong> according <strong>to females</strong>, when it is focused on <strong>generation Z</strong>, the<strong> number of male and female approaches each other </strong>and even<strong> in 2021</strong> <strong>number of female</strong> is <strong>slightly</strong> <strong>more</strong> than <strong>number of male</strong>. It clearly shows us that <strong>female labor force</strong> participation rate <strong>will increase</strong> for the Data Science area in the next future.</li><li>It can be easily seen that the total of generation <strong>Y </strong>and <strong>Z</strong> are almost entire of survey. The insight can be that the data related areas and the <strong>business title as Data Scientist</strong> is <strong>new concepts for business life</strong>. But when it is focused on just generation <strong>X,</strong> their <strong>business level</strong> mostly <strong>academician</strong>. So that, it can be interpreted that <strong>the academy goes before business life</strong>. Along with <strong>2021</strong>, it is seen the <strong>academician turns into senior expert</strong> in private sector. It can be <strong>interpreted</strong> that <strong>private </strong>sector<strong> needs </strong>expert<strong> data scientist </strong>than<strong> ever before.</strong></li><li><strong>Most of respondents</strong> generally are <strong>professional </strong>(junior), <strong>expert</strong> or <strong>student</strong>. It is <strong>clearly seen</strong> that<strong> reversed mentoring</strong> is very <strong>important</strong> for <strong>companies</strong>. Because the <strong>young talents</strong> will ensure that companies become <strong>data-driven companies</strong> in the next future.</li><li>While <strong>major of the participants</strong> are respectively <strong>Computer / Software Engineering</strong>, <strong>Statistics / Statistics</strong> and <strong>Computer Science</strong> etc. and <strong>Industrial / Management Engineering</strong>, their degree is <strong>mostly bachelor</strong>. <strong>Statistics / Statistics and Computer Science</strong> is largely preferred for the <strong>master’s degree</strong>. It shows us that <strong>Statistics / Statistics</strong> and <strong>Computer Science</strong> is <strong>basis of Data Science</strong>. It can be also seen easily that <strong>Economy / Economics / Finance / Business Admin</strong> have <strong>recently</strong> been one of the <strong>most important majors</strong> for master’s degree in consideration of <strong>pandemic</strong> which <strong>impacts negatively global economy</strong>.</li></ul><p><strong>Data Analysis / Reporting, Data Science, Artificial Intelligence, Business Intelligence</strong> / <strong>Decision Support </strong>and<strong> Business Analysis</strong> are the <strong>dominant business title</strong> of Data Science area for <strong>private sector.</strong></p><h3><strong>What is the Business Information of Data Scientists?</strong></h3><p>Here are some of our discoveries about business information of data scientists all over the world!</p><p>You can find some insights about <strong>experience</strong>, <strong>business field</strong>, <strong>company</strong> and <strong>data size</strong>, <strong>speaking languages</strong> and <strong>salary</strong> of respondents;</p><ul><li>Most of respondents know <strong>Turkish</strong> as <strong>proficient level </strong>since most of them come from Turkey. This lasts with <strong>English</strong>, <strong>Portuguese</strong>, <strong>Spanish, French</strong> and<strong> Arabic</strong>. It is a normal result because these languages are <strong>commonly spoken</strong> <strong>around the world</strong>. But in these languages while the language levels of respondents who say to know <strong>English</strong>,<strong> Portuguese and</strong> <strong>Spanish</strong> are <strong>upper intermediate</strong> and <strong>advance</strong>, this level is<strong> elementary </strong>for <strong>Arabic</strong> and <strong>French</strong> yet.</li><li>When we look at the <strong>experience of Data Scientists</strong>, the <strong>average</strong> is approx. <strong>5 years</strong> and this supports that <strong>Data Science</strong> is almost <strong>new working area</strong> for business. When it is focused on the experienced leaders, <strong>business fields</strong> as <strong>consulting</strong> and <strong>education</strong> directly attract the attention. These <strong>leaders</strong> certainly <strong>shape the future </strong>for business.</li><li>Most of respondents work <strong>in companies</strong> whose size in terms of <strong>number of workers</strong> are<strong> &lt;10</strong>. Last year, companies with <strong>less than 10 employees</strong> also stood out in this ranking. While these<strong> boutique companies</strong> especially are accepted by <strong>fresh graduated</strong> or comparatively <strong>less experienced talents</strong>, they also handled mostly relatively less sized data.</li><li>Most of respondents <strong>work on data</strong> whose<strong> size</strong> are <strong>between 100 MB and 1 TB</strong>. While the business fields are distributed smoothly including Computer (Hardware, Software, Hosting), Information Technology, Finance and Insurance, Consulting, Education, Legal Services, Automotive and Textile, it is found out that <strong>more data sized</strong> is considerably correlated with <strong>Computer</strong> (Hardware, Software, Hosting), <strong>Information Technology</strong>, <strong>Finance and Insurance</strong>.</li><li>The <strong>salary</strong> is a really important issue to should be spoken on it. <strong>Before data pre-processing</strong>, the <strong>raw data </strong>set not only include two type of <strong>salary periods</strong> including <strong>monthly</strong> and<strong> yearly</strong> but also include <strong>four different</strong> currency types including<strong> TRY</strong>, <strong>USD, EUR</strong> and <strong>GBP.</strong> This issue is a critical data handling process to <strong>convert</strong> the raw data <strong>to just one type</strong> as <strong>monthly salary period</strong> and <strong>USD currency</strong> type. After that, salary can be become to analyze. It is <strong>clearly seen</strong> that <strong>more experience</strong> <strong>more salary</strong>. Especially <strong>after 16 years’ experience</strong> in sector, it can be got <strong>more salary</strong> with achievement of business titles as manager and above on organizational chart. The insight is that the<strong> salary</strong> is sharply getting to<strong> increase after a certain experience</strong> in sector.</li></ul><h3><strong>What are the commonly used Data Science tools and for what purpose do they use?</strong></h3><p>Here are some of our exploratory insights of data scientists all over the world!</p><p>In this section, we will provide some insight on the <strong>tools</strong> that <strong>Data Scientists use the most and for what purposes</strong>. Nowadays one of the most frequently asked questions is<strong> which tools should I learn for Data Science</strong> and <strong>where and how should I start</strong>. The results of this survey show exactly the insights that answers these question marks.</p><ul><li>As a result of the incoming answers, it is understood that approx. <strong>30 different tools</strong> are used extensively by Data Scientists. Data Science always works by <strong>combining</strong> the <strong>most suitable tools</strong> to <strong>do the best job</strong>. For this reason, it would be wrong to set standards such as the following tool should be used in Data Science. The fact that we always have to keep in mind, using the <strong>right tool</strong> for every job will <strong>increase the quality</strong> and <strong>minimize the costs</strong>.</li><li>When we look at the <strong>Top 5</strong> Data Science tools; <strong>Excel, Python, SQL, Anaconda and R </strong>are standing out. With 2021 survey, in order to <strong>compare tools in itself</strong>, Data Science tools are <strong>divided into</strong> four categories including <strong>Tools of Database</strong>, <strong>ML Platforms</strong>, <strong>Tools of Programming</strong> and <strong>Software Packages</strong>.</li><li>Much of the <strong>world’s data</strong> resides <strong>in databases</strong>. SQL (or Structured Query Language) is a powerful language which is used for communicating with and extracting data from databases. As in the results given by the participants, one of the systems used by Data Scientists in daily life is <strong>SQL</strong>. A working knowledge of relational databases and queries is <strong>necessary</strong> if you want to become a data scientist. But <strong>NoSQL</strong> tools including <strong>MongoDB, Hadoop, Spark, Hive and Kafka</strong> are <strong>coming </strong>upon the<strong> heels </strong>of<strong> SQL</strong>.</li><li><strong>AWS, Google Cloud and Microsoft Azure</strong> is respectively very <strong>common ML Platforms</strong> and it can be said these three platforms are <strong>extremely dominant</strong> in the sector. The choices of the participants show that <strong>these 3 platforms</strong> are in <strong>serious competition</strong> in the sector.</li><li><strong>Anaconda</strong> is a <strong>free and open-source</strong> distribution of the <strong>Python</strong> and <strong>R</strong> programming languages for scientific computing. Thus, it takes its place in the Top 5 among the tools used by the participants. Another exploratory insight is that <strong>Python</strong> users prefer the <strong>Anaconda</strong> platform and <strong>Jupyter Notebook</strong> in it. Some <strong>useful</strong> libraries such as <strong>Numpy, Pandas, Scikit-Learn, Matplotlibt and SciPy</strong> are also used by participants.</li><li>We must admit that although we have the opportunity to use direct data from different sources today, <strong>Excel</strong> is one of the most <strong>frequently used</strong> data sources <strong>from the past to the present</strong>. <strong>Matlab, SPSS, Power BI and Tableau</strong> are the other software packages frequently used by participants.</li><li>Data Science is used in improving business processes and developing new products by using <strong>Statistics</strong> and <strong>Computer Sciences</strong> together. As can be seen in the survey results on statistics and computer science, the most used tools are <strong>Python and R</strong>. While R has been preferred more in this regard in the past, today <strong>Python</strong> is preferred because it has a <strong>strong interpreter</strong>, its <strong>use by big technology companies</strong>, it contains many <strong>different libraries</strong> and they are <strong>easy to use</strong>. You can easily see this difference, especially if you filter Junior employees and the Y-Z generations.</li><li>Although there are many tools in the field of Data Science today, each one differs from the other with a feature. The greatest flexibility of Data Scientists is due to the variety of these tools. When we asked this in the survey, Data Scientists stated that <strong>89%</strong> of them were <strong>satisfied with</strong> these tools. When it is <strong>compared with 2021</strong>, the result in terms of satisfying with tools is increased as <strong>93%</strong>. One reason for this high satisfaction which continues to rise rate might be that most of the <strong>Data Science tools</strong> are <strong>open source</strong> and have very <strong>strong communities</strong> behind them.</li></ul><p>When we look at the<strong> purpose of Data Science usage</strong>, the results do not surprise us at all. The Top 5 include <strong>Statistical Analysis</strong>, <strong>Preparing Report and Reporting Solutions</strong>, <strong>Data Management</strong>, <strong>Artificial Intelligence Applications</strong> and <strong>Processing / Accumulating Data</strong> This situation reflects the <strong>basic process of Data Science</strong> to us. The process is starting from collecting and managing data, creating a model by applying statistical analysis on it and making decisions on created reports at the end of the day.</p><h3><strong>How long does a Data Scientist work per day?</strong></h3><p>Here it is meaningful to make a distinction first. Rather than working, how much of this work do we <strong>work efficiently</strong>?. This was one of the other most frequently asked questions for us. We have reserved a section about this in our survey. We asked data scientists <strong>how long efficiently they work </strong>and how much <strong>R &amp; D</strong> they do per day. When we look at the results after performing the necessary operations on the outliers;</p><ul><li>It is understood from the survey that <strong>Data Scientists</strong> can <strong>work efficiently on average 5.3</strong> (It was 5.4 in last year)<strong> hours a day</strong>. White collars around the <strong>world</strong> work on average <strong>7–9</strong> <strong>hours a day</strong>. This is much-debated issue today. In 2019, Microsoft tested out a four-day work during a week in its Japan offices and found as a result employees were not only happier but also significantly more productive.</li><li>Data Science is a constantly changing and developing field. That’s why Data Scientists have to take time to improve themselves apart from their daily work. At that point when we examine the results of the survey, it is seen that <strong>Data Scientists allocate</strong> an <strong>average of 2.4 hours </strong>It was 2.5 in last year) a day for <strong>R&amp;D studies</strong>.</li><li>On the other hand, when we look at <strong>what Data Scientists do to improve themselves</strong> during the avg. 2.4 hours/day ; It seems that <strong>a high percentage (84%)</strong> of participants are <strong>using online channels</strong>. Especially <strong>online trainings, content and communities</strong> are the channels Data Scientists have chosen to develop themselves. It seems that this situation will increase considerably after the COVID-19. Otherwise, Data Scientists stated that they improved themselves by participating in in-company trainings. It is understood that <strong>companies </strong>that want to make a <strong>difference from the data</strong> should give <strong>importance to in-company training</strong> in order to <strong>keep and develop</strong> the <strong>human resources</strong> they have in Data Science field.</li><li>When we consider on these two averages, Data Scientists spend an average of <strong>8 hours working</strong> in a day. In this case, we recommend all data scientists to <strong>have fun</strong> with the <strong>other 8 hours</strong>, and <strong>rest the remaining 8 hours</strong>.</li></ul><h3><strong>Are Data Scientists satisfied with their current situation and what are they planning for the Future?</strong></h3><p>Although <strong>Science is a fact</strong>, we should not forget that <strong>Data Scientists are human</strong>. Human beings are <strong>seriously affected</strong> by their <strong>living and work conditions</strong>. If the most suitable living conditions and working conditions can be provided to scientists, then they will carry Science to the place it deserves in the highest quality. Therefore, developed countries are always ahead in science.</p><ul><li>When we look at the numbers, <strong>the situation</strong> is <strong>unfortunately not very satisfying</strong>. While some of the participants (<strong>30–40 %</strong>) preferred <strong>not to comment</strong> on both work and living conditions, others (<strong>40–50 %</strong>) stated that they are <strong>not satisfied</strong>. The results along with 2021 survey clearly indicate us that the <strong>conditions</strong> both living and working have become <strong>much worse</strong> day by day. Participants (<strong>45–60 %</strong>) unfortunately <strong>express not satisfied</strong>. The <strong>problems</strong> experienced all over the world and the <strong>recurrence of these problems</strong> for a long time <strong>affect people’s views</strong>. Another <strong>reality is</strong> the fact of <strong>globalization</strong>. Although it has good sides, its <strong>bad effects</strong> are faster and bigger <strong>in problematic times</strong>. All links in the supply chain are affected. For this reason, if we want <strong>success in Science of the World</strong>, the <strong>whole should be happy</strong>, not a certain segment. The <strong>other problem</strong> which people all over the world experienced is pandemic of <strong>Covid19</strong>. At the <strong>beginning of</strong> the Covid19 epidemic, <strong>people</strong> from all over the world <strong>experienced working from home</strong>, and although everyone was happy with this at first, <strong>increasing working hours</strong>, <strong>long meetings</strong> and <strong>intense pace</strong> caused the <strong>work-life balance</strong> to <strong>deteriorate</strong>. It should not be forgotten that <strong>Covid19</strong> may also have an <strong>effect</strong> on the <strong>worsening</strong> of the answers to this question, where we evaluate the living and working environment. We believe that both states and companies will take the <strong>necessary precautions</strong> in the following days.</li><li>Data Scientists (<strong>40%</strong>) stated that although they are <strong>not satisfied with their job</strong> and <strong>living conditions</strong>, they <strong>want to improve themselves</strong> in their fields of work as a <strong>future plan</strong> in the survey results. We see this as a positive situation. This rate shows us that <strong>Data Scientists have high hopes</strong> for their jobs and science. It is really <strong>pleasing</strong> that this rate has <strong>increased to 60%</strong> with the <strong>2021</strong> survey. The Covid19 outbreak has also shown us that <strong>online training</strong> is an opportunity in such situations. The results of the 2021 survey showed us that the <strong>desire to participate in online trainings</strong> increased from <strong>83% to 88%,</strong> especially for the Z generation. Apart from this, those who are not satisfied with the current situation and want to <strong>change jobs</strong> and <strong>go abroad</strong> also have a serious rate (<strong>30%</strong>). Finally, the <strong>entrepreneurs</strong> (<strong>15%</strong>), the boss of his own business, are among those who do <strong>not lose hope</strong>. However, this ratio has unfortunately <strong>declined to 11%</strong> with the 2021 results. One of the <strong>biggest reasons</strong> for this decline is probably the <strong>Covid19 outbreak</strong>. We hope all Data Scientists can do whatever they want best.</li></ul><h3><strong>Conclusion</strong></h3><p>In general, we see that the <strong>Data Science ecosystem</strong> continues to <strong>evolve with</strong> <strong>young talents</strong> and <strong>its</strong> <strong>dynamic structure</strong>. As every technology has a <strong>hype circle</strong>, this is valid in this field too. When we analyze the results in depth, we understand that we are still in the <strong>early stages</strong> of this circle.</p><p>In particular, it seems that the <strong>Y </strong>and <strong>Z generation </strong>will <strong>have a serious voice</strong> in this field <strong>in the coming years</strong> and will <strong>direct this field</strong>. It is understood that <strong>companies</strong> which want to make a <strong>difference through the data</strong> should <strong>support these young talents </strong>and <strong>invest in this field</strong>.</p><p>In this area, where <strong>online content consumption increases </strong>especially in the direction of learning, <strong>solutions </strong>are also<strong> produced quickly</strong>. The flexibility and diversity of the tools used and the strong communities behind these tools provide considerable flexibility and convenience for Data Scientists.</p><p>Last but not least, <strong>big thanks to our researchers</strong> — <a href="https://medium.com/u/abf96b2d1f53">Mehmet Sezer</a>, <a href="https://medium.com/u/6150f9aae167">Yunus Emre MIZRAKÇI</a> who contributed to this work. We can say that <strong>Data Science</strong> is a field where solutions are created in an enjoyable way as <strong>knowledge</strong> and<strong> experience</strong> are shared. We created this study to be a guide for Data Scientists. Our <strong>request</strong> from you, let us continue to <strong>contribute to Data Science all together</strong> and <strong>beautify the world with Data Science</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59582c707d4d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/data-science-survey-2021-insights-59582c707d4d">DATA SCIENCE SURVEY 2021 — INSIGHTS</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Basic Structure of Artificial Neural Networks]]></title>
            <link>https://medium.com/datasciencearth/basic-structure-of-artificial-neural-networks-9aef29df9d?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/9aef29df9d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-neural-network]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Beyza Ecem Erce]]></dc:creator>
            <pubDate>Fri, 26 Mar 2021 18:47:39 GMT</pubDate>
            <atom:updated>2021-03-26T18:47:39.155Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Note: In this article, I mention about the two basic architectures and basic elements of artificial neural networks. The concepts you need to know beforehand are simply; regression and classification types.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nf0FvEI6yjQQX0vvQvjNUw.png" /></figure><h3>Basic Structure of Artificial Neural Networks</h3><p>The Artificial Neural Networks (ANNs) are computational models that are inspired from human brain. In another words, it is the modelling of human brain work logic mathematically. The main goal is providing<em> a result(or output)</em> that in line with our purpose after passing some processes. Just as the human brain has billions of neurons, ANNs also has hundreds or thousands of <em>artificial neurons</em>.</p><p>ANNs are used for regression or classification problems and they consists of two basic architecture:</p><ol><li>Single-Layer Artificial Neural Networks</li><li>Multi-Layer Artifical Neural Networks</li></ol><h4>Single-Layer Artificial Neural Networks</h4><p>The Single-Layer Artificial Neural Networks are also called as <strong>Perceptrons</strong>. The Perceptron, is the basic component of ANNs. It is actually binary classification algorithm that is invented by Frank Rosenblatt in 1957. That is, it is an algorithm that tries to decide which output class an input belongs to.</p><p><strong>How does the Perceptron work?</strong></p><p>The perceptron consists of five components:</p><ol><li><strong>Inputs:</strong> These are the independent variables (x) that we have.</li><li><strong>Weigths: </strong>Weight parameters (w) control the strength of the connection between inputs and neurons. It can also be said to represent the effect of an independent variable on the result.</li><li><strong>Bias value(b): </strong>It is a constant value that allows to control the output value. Also, when all inputs are zero, it ensures that the process can still continue.</li><li><strong>Activation Functions: </strong>The activation function (f) defines the output of the neuron according to certain conditions.</li><li><strong>Output: </strong>The dependent variable (y) is the result we want to find. In perceptrons, the result is divided into two classes, classes 1 and 0.</li></ol><p>If we formulate the process, we can show it like this:<strong> y=f(x×w+b)</strong></p><figure><img alt="The Perceptron" src="https://cdn-images-1.medium.com/max/300/0*Kqt8CNUAmV5lMsFo.jpg" /><figcaption>The Perceptron</figcaption></figure><p><strong>The process proceeds as follows:</strong></p><ul><li>The weighted sum is calculated by first multiplication of weights and inputs, then addition of them. The bias value is included in this value.(bias=b=<em>x0×w0</em>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*XgkifN5mbeGoNpD6xkCBuw.png" /><figcaption>The weighted sum</figcaption></figure><ul><li>The activation function is applied to the weighted sum(z) and the result is found. The perceptrons use <em>step function</em> as a activation function. According to step funciton, if the weigted sum;</li></ul><p><strong>→ z&gt;0, then result is 1,</strong></p><p><strong>→ z≤0, then result is 0.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/1*ShapLLmFE-7wl-kTJDhxkg.png" /><figcaption>The output formula with activation function</figcaption></figure><p>The applied area of perceptrons is limited. Beacuse perceptrons are usually used for simple binary classification problems that are linearly separable.</p><h4>Multi-Layer Neural Networks</h4><p>As the name suggests, the multi-layer neural networks, or the multi-layer perceptrons(MLPs), consist layers more than one. Beside of the perceptrons, they can be used for non-linearly separable problems. They achieve this with the activation functions they use in their layers. The activation functions make the output of neurons nonlinear. In this way, it enables to solve more complex problems. (Without activation function, ANNs actually become a linear regression model.)</p><p>The basic layers of MLPs are:</p><ul><li><strong>1 input layer</strong>→ it includes <strong>1 neuron per input x</strong>.</li><li><strong>Hidden layers(one or more)</strong>→ The number of neurons it consists depends on the problem.</li><li><strong>1 output layer</strong> →The number of neurons it consists depends on the problem.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/0*VbpLHExmNePxdYFU.png" /><figcaption><em>The Multi-Layer Perceptron</em></figcaption></figure><p>In the first step , <strong>for every neurons of hidden layers</strong>, the same process in the perceptron is applied:</p><ol><li>The weighted sum(z) is calculated.</li><li>It is transmitted to related hidden neuron, then the activation function present in the neuron(ReLU or SELU) is applied.</li></ol><p>In the next step, the outputs of hidden layers are transmitted to <strong><em>output layer</em></strong>. As said before, the <strong>number of neurons</strong> depends on the problem in here:</p><blockquote><strong>Regression:</strong> consists of <strong>1 neuron</strong> ,</blockquote><blockquote><strong>Binary Classification: </strong>consists of <strong>1 neuron,</strong></blockquote><blockquote><strong>Multi-label Classification:</strong> consists of <strong>1 neuron per label</strong>,</blockquote><blockquote><strong>Multi-class Classification:</strong> consists of <strong>1 neuron per class</strong> in the output layer.</blockquote><p>The activation functions in neurons of output layer also depends on the task:</p><blockquote><strong>Regression:</strong> None or ReLU/Softplus(if positive outputs) or Logistic/tanh( if bounded outputs),</blockquote><blockquote><strong>Binary Classification:</strong> Logistic(sigmoid) function,</blockquote><blockquote><strong>Multi-label Classification:</strong> Logistic(sigmoid) function,</blockquote><blockquote><strong>Multi-class Classification:</strong> Softmax function.</blockquote><p>The main goal is to enable ANN to learn the most accurate weight values (so achiving most accurate result) with correct hidden layer and neuron numbers. We can do this by applying certain processes in our artificial neural network and optimizing it.</p><p>I am sure that many questions have come alive in your mind like “So how does the learning process take place in ANN? What are the details of the activation functions? How many types of artificial neural networks are there and for what purposes?”. What is described in this article is actually just the tip of the iceberg. Hope to see you in other articles we meet to enlighten these questions!</p><p><em>References</em></p><ul><li>“Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems” book by Aurelien Geron</li></ul><p>Images:</p><ul><li><a href="http://www.jessicayung.com/explaining-tensorflow-code-for-a-multilayer-perceptron/">www.jessicayung.com/explaining-tensorflow-code-for-a-multilayer-perceptron/</a></li><li><a href="https://github.com/jorgesleonel/Multilayer-Perceptron">https://github.com/jorgesleonel/Multilayer-Perceptron</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9aef29df9d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/basic-structure-of-artificial-neural-networks-9aef29df9d">Basic Structure of Artificial Neural Networks</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Write for Data Science Earth]]></title>
            <link>https://medium.com/datasciencearth/write-for-data-science-earth-6f7c5b814bd9?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/6f7c5b814bd9</guid>
            <dc:creator><![CDATA[Data Science Earth]]></dc:creator>
            <pubDate>Sat, 20 Mar 2021 19:42:17 GMT</pubDate>
            <atom:updated>2021-03-20T19:42:17.341Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*LNkhe74FyfGkrIgY5LhYaQ.jpeg" /></figure><p>Data Science Earth (DSE) team consists of experts in the field gathered to produce data science solutions.</p><p>Data Science Earth primarily aims to develop data awareness and ensure the correct use of the power of data in a globalized competitive environment.</p><p>Our team, which has been working in the field of data science for more than a year, is expanding its area of ​​influence and increasing the number of followers.</p><p>Recently, the number of our participants who follow the activities of our community, especially from abroad, has been increasing. We plan to expand our English content, which we think will be very effective in the process of expanding abroad. In this context, we created a page on Medium called DSE.</p><p>We are expecting to share and to adopt teamwork.</p><p>Join us to write and share your knowledge on data management, data analysis, business intelligence, data mining, text mining, machine learning, artificial intelligence, software, mathematics, statistics, data visualization, technology.</p><p><a href="https://www.questionpro.com/a/TakeSurvey?tt=qRgUMwXY%2Bu8%3D"><strong><em>Join Us</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f7c5b814bd9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/write-for-data-science-earth-6f7c5b814bd9">Write for Data Science Earth</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Warehouse Training — Episode 9 — What is Data Lake ? Data Lake VS Data Warehouse]]></title>
            <link>https://medium.com/datasciencearth/data-warehouse-training-episode-9-what-is-data-lake-data-lake-vs-data-warehouse-56a257f5caed?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/56a257f5caed</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Data Science Earth]]></dc:creator>
            <pubDate>Fri, 19 Mar 2021 08:51:37 GMT</pubDate>
            <atom:updated>2021-03-19T08:51:36.968Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Warehouse Training — Episode 9 — What is Data Lake ? Data Lake VS Data Warehouse</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HA9TghDjJCT_601EZ5gI_w.png" /></figure><h3>What is Data Lake?</h3><p>A Data Lake is a storage repository that can store large amount of structured, semi-structured, and unstructured data. It is a place to store every type of data in its native format with no fixed limits on account size or file. It offers high data quantity to increase analytic performance and native integration.</p><p>Data Lake is like a large container which is very similar to real lake and rivers. Just like in a lake you have multiple tributaries coming in, a data lake has structured data, unstructured data, machine to machine, logs flowing through in real-time.</p><p>The Data Lake democratizes data and is a cost-effective way to store a</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*HadktDNpUiurNybe.png" /></figure><p>ll data of an organization for later processing. Research Analyst can focus on finding meaning patterns in data and not data itself.</p><p>Unlike a hierarchal Dataware house where data is stored in Files and Folder, Data lake has a flat architecture. Every data elements in a Data Lake is given a unique identifier and tagged with a set of metadata information.</p><h3>Why Data Lake?</h3><p>The main objective of building a data lake is to offer an unrefined view of data to data scientists.</p><p>Reasons for using Data Lake are:</p><ul><li>With the onset of storage engines like Hadoop storing disparate information has become easy. There is no need to model data into an enterprise-wide schema with a Data Lake.</li><li>With the increase in data volume, data quality, and metadata, the quality of analyses also increases.</li><li>Data Lake offers business Agility</li><li>Machine Learning and Artificial Intelligence can be used to make profitable predictions.</li><li>It offers a competitive advantage to the implementing organization.</li><li>There is no data silo structure. Data Lake gives 360 degrees view of customers and makes analysis more robust.</li></ul><h3>Data Lake Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*BvueKJfvdnOWUTw4.png" /></figure><p>The figure shows the architecture of a Business Data Lake. The lower levels represent data that is mostly at rest while the upper levels show real-time transactional data. This data flow through the system with no or little latency. Following are important tiers in Data Lake Architecture:</p><ol><li><strong>Ingestion Tier</strong>: The tiers on the left side depict the data sources. The data could be loaded into the data lake in batches or in real-time</li><li><strong>Insights Tier: </strong>The tiers on the right represent the research side where insights from the system are used. SQL, NoSQL queries, or even excel could be used for data analysis.</li><li><strong>HDFS </strong>is a cost-effective solution for both structured and unstructured data. It is a landing zone for all data that is at rest in the system.</li><li><strong>Distillation tier</strong> takes data from the storage tire and converts it to structured data for easier analysis.</li><li><strong>Processing tier</strong> run analytical algorithms and users queries with varying real time, interactive, batch to generate structured data for easier analysis.</li><li><strong>Unified operations tier</strong> governs system management and monitoring. It includes auditing and proficiency management, data management, workflow management.</li></ol><h3>Key Data Lake Concepts</h3><p>Following are Key Data Lake concepts that one needs to understand to completely understand the Data Lake Architecture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*leuEd5NUzQ9SZaHQ.png" /></figure><h4>Data Ingestion</h4><p>Data Ingestion allows connectors to get data from a different data sources and load into the Data lake.</p><p>Data Ingestion supports:</p><ul><li>All types of Structured, Semi-Structured, and Unstructured data.</li><li>Multiple ingestions like Batch, Real-Time, One-time load.</li><li>Many types of data sources like Databases, Webservers, Emails, IoT, and FTP.</li></ul><h4>Data Storage</h4><p>Data storage should be scalable, offers cost-effective storage and allow fast access to data exploration. It should support various data formats.</p><h4>Data Governance</h4><p>Data governance is a process of managing availability, usability, security, and integrity of data used in an organization.</p><h4>Security</h4><p>Security needs to be implemented in every layer of the Data lake. It starts with Storage, Unearthing, and Consumption. The basic need is to stop access for unauthorized users. It should support different tools to access data with easy to navigate GUI and Dashboards.</p><p>Authentication, Accounting, Authorization and Data Protection are some important features of data lake security.</p><h4>Data Quality</h4><p>Data quality is an essential component of Data Lake architecture. Data is used to exact business value. Extracting insights from poor quality data will lead to poor quality insights.</p><h4>Data Discovery</h4><p>Data Discovery is another important stage before you can begin preparing data or analysis. In this stage, tagging technique is used to express the data understanding, by organizing and interpreting the data ingested in the Data lake.</p><h4>Data Auditing</h4><p>Two major Data auditing tasks are tracking changes to the key dataset.</p><ol><li>Tracking changes to important dataset elements</li><li>Captures how/ when/ and who changes to these elements.</li></ol><p>Data auditing helps to evaluate risk and compliance.</p><h4>Data Lineage</h4><p>This component deals with data’s origins. It mainly deals with where it movers over time and what happens to it. It eases errors corrections in a data analytics process from origin to destination.</p><h4>Data Exploration</h4><p>It is the beginning stage of data analysis. It helps to identify right dataset is vital before starting Data Exploration.</p><p>All given components need to work together to play an important part in Data lake building easily evolve and explore the environment.</p><h3>Maturity stages of Data Lake</h3><p>The Definition of Data Lake Maturity stages differs from textbook to other. Though the crux remains the same. Following maturity, stage definition is from a layman point of view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*qXBpTdiiejNjaRfx.png" /></figure><h4>Stage 1: Handle and ingest data at scale</h4><p>This first stage of Data Maturity Involves improving the ability to transform and analyze data. Here, business owners need to find the tools according to their skillset for obtaining more data and build analytical applications.</p><h4>Stage 2: Building the analytical muscle</h4><p>This is a second stage which involves improving the ability to transform and analyze data. In this stage, companies use the tool which is most appropriate to their skillset. They start acquiring more data and building applications. Here, capabilities of the enterprise data warehouse and data lake are used together.</p><h4>Stage 3: EDW and Data Lake work in unison</h4><p>This step involves getting data and analytics into the hands of as many people as possible. In this stage, the data lake and the enterprise data warehouse start to work in a union. Both playing their part in analytics</p><h4>Stage 4: Enterprise capability in the lake</h4><p>In this maturity stage of the data lake, enterprise capabilities are added to the Data Lake. Adoption of information governance, information lifecycle management capabilities, and Metadata management. However, very few organizations can reach this level of maturity, but this tally will increase in the future.</p><h3>Best practices for Data Lake Implementation</h3><ul><li>Architectural components, their interaction and identified products should support native data types</li><li>Design of Data Lake should be driven by what is available instead of what is required. The schema and data requirement is not defined until it is queried</li><li>Design should be guided by disposable components integrated with service API.</li><li>Data discovery, ingestion, storage, administration, quality, transformation, and visualization should be managed independently.</li><li>The Data Lake architecture should be tailored to a specific industry. It should ensure that capabilities necessary for that domain are an inherent part of the design</li><li>Faster on-boarding of newly discovered data sources is important</li><li>Data Lake helps customized management to extract maximum value</li><li>The Data Lake should support existing enterprise data management techniques and methods</li></ul><h4>Challenges of building a data lake:</h4><ul><li>In Data Lake, Data volume is higher, so the process must be more reliant on programmatic administration</li><li>It is difficult to deal with sparse, incomplete, volatile data</li><li>Wider scope of dataset and source needs larger data governance &amp; support</li></ul><h4>Benefits and Risks of using Data Lake:</h4><p>Here are some major benefits in using a Data Lake:</p><ul><li>Helps fully with product ionizing &amp; advanced analytics</li><li>Offers cost-effective scalability and flexibility</li><li>Offers value from unlimited data types</li><li>Reduces long-term cost of ownership</li><li>Allows economic storage of files</li><li>Quickly adaptable to changes</li><li>The main advantage of data lake is the <strong>centralization </strong>of different content sources</li><li>Users, from various departments, may be scattered around the globe can have <strong>flexible access</strong> to the data</li></ul><h4>Risk of Using Data Lake:</h4><ul><li>After some time, Data Lake may lose relevance and momentum</li><li>There is larger amount risk involved while designing Data Lake</li><li>Unstructured Data may lead to Ungoverned Chao, Unusable Data, Disparate &amp; Complex Tools, Enterprise-Wide Collaboration, Unified, Consistent, and Common</li><li>It also increases storage &amp; computes costs</li><li>There is no way to get insights from others who have worked with the data because there is no account of the lineage of findings by previous analysts</li><li>The biggest risk of data lakes is security and access control. Sometimes data can be placed into a lake without any oversight, as some of the data may have privacy and regulatory need</li></ul><h4>Summary:</h4><ul><li>A Data Lake is a storage repository that can store large amount of structured, semi-structured, and unstructured data.</li><li>The main objective of building a data lake is to offer an unrefined view of data to data scientists.</li><li>Unified operations tier, Processing tier, Distillation tier and HDFS are important layers of Data Lake Architecture</li><li>Data Ingestion, Data storage, Data quality, Data Auditing, Data exploration, Data discover are some important components of Data Lake Architecture</li><li>Design of Data Lake should be driven by what is available instead of what is required.</li><li>Data Lake reduces long-term cost of ownership and allows economic storage of files</li><li>The biggest risk of data lakes is security and access control. Sometimes data can be placed into a lake without any oversight, as some of the data may have privacy and regulatory need.</li></ul><h3>Key Difference between the Data Lake and Data Warehouse</h3><p>Here are key differences between the two data associated terms in the mentioned aspects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*zKQ_0GboDfG4TPxgUnV4xw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*F-TZb93A1FpEcA4b5-zdaw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*XnSkE5CkfDGigGjJiupL4g.jpeg" /></figure><h3>I hope my document was useful</h3><ul><li>Alperen KEZAY</li></ul><h4>Source:</h4><p><a href="https://www.guru99.com/data-warehousing.html">https://www.guru99.com/data-warehousing.html</a></p><p><a href="https://en.wikipedia.org/wiki/Data_warehouse">https://en.wikipedia.org/wiki/Data_warehouse</a></p><p><a href="https://www.javatpoint.com/types-of-data-warehouses">https://www.javatpoint.com/types-of-data-warehouses</a></p><p><a href="https://panoply.io/data-warehouse-guide/the-difference-between-a-database-and-a-data-warehouse/">https://panoply.io/data-warehouse-guide/the-difference-between-a-database-and-a-data-warehouse/</a></p><p><a href="https://aws.amazon.com/tr/data-warehouse/#:~:text=How%20does%20a%20data%20warehouse">https://aws.amazon.com/tr/data-warehouse/#:~:text=How%20does%20a%20data%20warehouse</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56a257f5caed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/data-warehouse-training-episode-9-what-is-data-lake-data-lake-vs-data-warehouse-56a257f5caed">Data Warehouse Training — Episode 9 — What is Data Lake ? Data Lake VS Data Warehouse</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Warehouse Training — Episode 8 — What is Data Mart? Data Mart VS Data Warehouse]]></title>
            <link>https://medium.com/datasciencearth/data-warehouse-training-episode-8-what-is-data-mart-data-mart-vs-data-warehouse-e86f8bca49d4?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/e86f8bca49d4</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-mart]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Data Science Earth]]></dc:creator>
            <pubDate>Fri, 19 Mar 2021 08:51:26 GMT</pubDate>
            <atom:updated>2021-03-19T08:51:26.795Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Warehouse Training — Episode 8 — What is Data Mart? Data Mart VS Data Warehouse</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1xpF-T_qDRk3wCmMN932Lw.png" /></figure><h3>What is Data Mart</h3><p>A <strong>DATA MART </strong>is focused on a single functional area of an organization and contains a subset of data stored in a Data Warehouse. A Data Mart is a condensed version of Data Warehouse and is designed for use by a specific department, unit or set of users in an organization. E.g., Marketing, Sales, HR or finance. It is often controlled by a single department in an organization.</p><p>Data Mart usually draws data from only a few sources compared to a Data warehouse. Data marts are small in size and are more flexible compared to a Datawarehouse.</p><h3>Why do we need Data Mart?</h3><ul><li>Data Mart helps to enhance user’s response time due to reduction in volume of data</li><li>It provides easy access to frequently requested data.</li><li>Data mart are simpler to implement when compared to corporate Datawarehouse. At the same time, the cost of implementing Data Mart is certainly lower compared with implementing a full data warehouse.</li><li>Compared to Data Warehouse, a datamart is agile. In case of change in model, datamart can be built quicker due to a smaller size.</li><li>A Datamart is defined by a single Subject Matter Expert. On the contrary data warehouse is defined by interdisciplinary SME from a variety of domains. Hence, Data mart is more open to change compared to Datawarehouse.</li><li>Data is partitioned and allows very granular access control privileges.</li><li>Data can be segmented and stored on different hardware/software platforms.</li></ul><h3>Type of Data Mart</h3><p>There are three main types of data marts are:</p><ol><li><strong>Dependent</strong>: Dependent data marts are created by drawing data directly from operational, external or both sources.</li><li><strong>Independent</strong>: Independent data mart is created without the use of a central data warehouse.</li><li><strong>Hybrid</strong>: This type of data marts can take data from data warehouses or operational systems.</li></ol><h3>Dependent Data Mart</h3><p>A dependent data mart allows sourcing organization’s data from a single Data Warehouse. It offers the benefit of centralization. If you need to develop one or more physical data marts, then you need to configure them as dependent data marts.</p><p>Dependent data marts can be built in two different ways. Either where a user can access both the data mart and data warehouse, depending on need, or where access is limited only to the data mart. The second approach is not optimal as it produces sometimes referred to as a data junkyard. In the data junkyard, all data begins with a common source, but they are scrapped, and mostly junked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*a4cCHSh2SKP-2sAk.png" /></figure><h3>Independent Data Mart</h3><p>An independent data mart is created without the use of central Data warehouse. This kind of Data Mart is an ideal option for smaller groups within an organization.</p><p>An independent data mart has neither a relationship with the enterprise data warehouse nor with any other data mart. In Independent data mart, the data is input separately, and its analyses are also performed autonomously.</p><p>Implementation of independent data marts is antithetical to the motivation for building a data warehouse. First of all, you need a consistent, centralized store of enterprise data which can be analyzed by multiple users with different interests who want widely varying information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/0*a-swsm64VyY1zm6D.png" /></figure><h3>Hybrid data Mart</h3><p>A hybrid data mart combines input from sources apart from Data warehouse. This could be helpful when you want ad-hoc integration, like after a new group or product is added to the organization.</p><p>It is best suited for multiple database environments and fast implementation turnaround for any organization. It also requires least data cleansing effort. Hybrid Data mart also supports large storage structures, and it is best suited for flexible for smaller data-centric applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*Y5VwthanfMV5kemZ.png" /></figure><h3>Steps in Implementing a Datamart</h3><p>Implementing a Data Mart is a rewarding but complex procedure. Here are the detailed steps to implement a Data Mart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*PzqbF36aDrgYMwm-.png" /></figure><h4>Designing</h4><p>Designing is the first phase of Data Mart implementation. It covers all the tasks between initiating the request for a data mart to gathering information about the requirements. Finally, we create the logical and physical design of the data mart.</p><p><strong>The design step involves the following tasks:</strong></p><ul><li>Gathering the business &amp; technical requirements and Identifying data sources.</li><li>Selecting the appropriate subset of data.</li><li>Designing the logical and physical structure of the data mart.</li></ul><p>Data could be partitioned based on following criteria:</p><ul><li>Date</li><li>Business or Functional Unit</li><li>Geography</li><li>Any combination of above</li></ul><p>Data could be partitioned at the application or DBMS level. Though it is recommended to partition at the Application level as it allows different data models each year with the change in business environment.</p><p><strong>What Products and Technologies Do You Need?</strong></p><p>A simple pen and paper would suffice. Though tools that help you create UML or ER diagrams would also append meta data into your logical and physical designs.</p><h4>Constructing</h4><p>This is the second phase of implementation. It involves creating the physical database and the logical structures.</p><p><strong>This step involves the following tasks:</strong></p><ul><li>Implementing the physical database designed in the earlier phase. For instance, database schema objects like table, indexes, views, etc. are created.</li></ul><p><strong>What Products and Technologies Do You Need?</strong></p><p>You need a relational database management system to construct a data mart. RDBMS have several features that are required for the success of a Data Mart.</p><ul><li><strong>Storage management:</strong> An RDBMS stores and manages the data to create, add, and delete data.</li><li><strong>Fast data access:</strong> With a SQL query you can easily access data based on certain conditions/filters.</li><li><strong>Data protection:</strong> The RDBMS system also offers a way to recover from system failures such as power failures. It also allows restoring data from these backups incase of the disk fails.</li><li><strong>Multiuser support:</strong> The data management system offers concurrent access, the ability for multiple users to access and modify data without interfering or overwriting changes made by another user.</li><li><strong>Security:</strong> The RDMS system also provides a way to regulate access by users to objects and certain types of operations.</li></ul><h4>Populating</h4><p>In the third phase, data in populated in the data mart.</p><p>The populating step involves the following tasks:</p><ul><li>Source data to target data Mapping</li><li>Extraction of source data</li><li>Cleaning and transformation operations on the data</li><li>Loading data into the data mart</li><li>Creating and storing metadata</li></ul><p><strong>What Products and Technologies Do You Need?</strong></p><p>You accomplish these population tasks using an ETL (Extract Transform Load) Tool. This tool allows you to look at the data sources, perform source-to-target mapping, extract the data, transform, cleanse it, and load it back into the data mart.</p><p>In the process, the tool also creates some metadata relating to things like where the data came from, how recent it is, what type of changes were made to the data, and what level of summarization was done.</p><h4>Accessing</h4><p>Accessing is a fourth step which involves putting the data to use: querying the data, creating reports, charts, and publishing them. End-user submit queries to the database and display the results of the queries</p><p><strong>The accessing step needs to perform the following tasks:</strong></p><ul><li>Set up a meta layer that translates database structures and objects names into business terms. This helps non-technical users to access the Data mart easily.</li><li>Set up and maintain database structures.</li><li>Set up API and interfaces if required</li></ul><p><strong>What Products and Technologies Do You Need?</strong></p><p>You can access the data mart using the command line or GUI. GUI is preferred as it can easily generate graphs and is user-friendly compared to the command line.</p><h4>Managing</h4><p>This is the last step of Data Mart Implementation process. This step covers management tasks such as-</p><ul><li>Ongoing user access management.</li><li>System optimizations and fine-tuning to achieve the enhanced performance.</li><li>Adding and managing fresh data into the data mart.</li><li>Planning recovery scenarios and ensure system availability in the case when the system fails.</li></ul><p><strong>What Products and Technologies Do You Need?</strong></p><p>You could use the GUI or command line for data mart management.</p><h3>Best practices for Implementing Data Marts</h3><p>Following are the best practices that you need to follow while in the Data Mart Implementation process:</p><ul><li>The source of a Data Mart should be departmentally structured</li><li>The implementation cycle of a Data Mart should be measured in short periods of time, i.e., in weeks instead of months or years.</li><li>It is important to involve all stakeholders in planning and designing phase as the data mart implementation could be complex.</li><li>Data Mart Hardware/Software, Networking and Implementation costs should be accurately budgeted in your plan</li><li>Even though if the Data mart is created on the same hardware they may need some different software to handle user queries. Additional processing power and disk storage requirements should be evaluated for fast user response</li><li>A data mart may be on a different location from the data warehouse. That’s why it is important to ensure that they have enough networking capacity to handle the Data volumes needed to transfer data to the data mart<strong>.</strong></li><li>Implementation cost should budget the time taken for Datamart loading process. Load time increases with increase in complexity of the transformations.</li></ul><h3>Advantages and Disadvantages of a Data Mart</h3><h4>Advantages</h4><ul><li>Data marts contain a subset of organization-wide data. This Data is valuable to a specific group of people in an organization.</li><li>It is cost-effective alternatives to a data warehouse, which can take high costs to build.</li><li>Data Mart allows faster access of Data.</li><li>Data Mart is easy to use as it is specifically designed for the needs of its users. Thus a data mart can accelerate business processes.</li><li>Data Marts needs less implementation time compare to Data Warehouse systems. It is faster to implement Data Mart as you only need to concentrate the only subset of the data.</li><li>It contains historical data which enables the analyst to determine data trends.</li></ul><h4>Disadvantages</h4><ul><li>Many a times enterprises create too many disparate and unrelated data marts without much benefit. It can become a big hurdle to maintain.</li><li>Data Mart cannot provide company-wide data analysis as their data set is limited.</li></ul><h4>Summary:</h4><ul><li>A Data Mart is defined as a subset of Data Warehouse that is focused on a single functional area of an organization.</li><li>Data Mart helps to enhance user’s response time due to a reduction in the volume of data.</li><li>Three types of data mart are 1) Dependent 2) Independent 3) Hybrid</li><li>Important implementation steps of Data Mart are 1) Designing 2) Constructing 3 Populating 4) Accessing and 5) Managing</li><li>The implementation cycle of a Data Mart should be measured in short periods of time, i.e., in weeks instead of months or years.</li><li>Data mart is cost-effective alternatives to a data warehouse, which can take high costs to build.</li><li>Data Mart cannot provide company-wide data analysis as data set is limited.</li></ul><h3>Data Mart vs Data Warehouse</h3><h4>KEY DIFFERENCE</h4><ul><li>Data Warehouse is a large repository of data collected from different sources whereas Data Mart is only subtype of a data warehouse.</li><li>Data Warehouse is focused on all departments in an organization whereas Data Mart focuses on a specific group.</li><li>Data Warehouse designing process is complicated whereas the Data Mart process is easy to design.</li><li>Data Warehouse takes a long time for data handling whereas Data Mart takes a short time for data handling.</li><li>Data Warehouse size range is 100 GB to 1 TB+ whereas Data Mart size is less than 100 GB.</li><li>Data Warehouse implementation process takes 1 month to 1 year whereas Data Mart takes a few months to complete the implementation process.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*0sxjX3sZFmwmWA9v.png" /></figure><h4>Differences between Data Warehouse and Data Mart</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*trZjrgOA4DyD5oIfZ2mVjA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*uJ8OvmnLpObZIZlVCOhlQA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*VMLE7tYy-0nmL8n0tbJRnA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*JRYTA_DXOAuajcab-aaA_g.jpeg" /></figure><p><strong><em>Alperen Kezay</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e86f8bca49d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/data-warehouse-training-episode-8-what-is-data-mart-data-mart-vs-data-warehouse-e86f8bca49d4">Data Warehouse Training — Episode 8 — What is Data Mart? Data Mart VS Data Warehouse</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Warehouse Training — Episode 7 — What is Dimensional Modeling?]]></title>
            <link>https://medium.com/datasciencearth/data-warehouse-training-episode-7-what-is-dimensional-modeling-a59998b20ad1?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/a59998b20ad1</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[dimensional-model]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Data Science Earth]]></dc:creator>
            <pubDate>Fri, 19 Mar 2021 08:51:09 GMT</pubDate>
            <atom:updated>2021-03-19T08:51:09.251Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Warehouse Training — Episode 7 — What is Dimensional Modeling?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-S5A1MZCqCM_soDMb82HCw.png" /></figure><h3>What is Dimensional Modeling in Data Warehouse?</h3><h4>What is Dimensional Modeling?</h4><p><strong>DIMENSIONAL MODELING (DM)</strong> is a data structure technique optimized for data storage in a Data warehouse. The purpose of dimensional model is to optimize the database for fast retrieval of data. The concept of Dimensional Modelling was developed by Ralph Kimball and consists of “fact” and “dimension” tables.</p><p>A Dimensional model is designed to read, summarize, analyze numeric information like values, balances, counts, weights, etc. in a data warehouse. In contrast, relation models are optimized for addition, updating and deletion of data in a real-time Online Transaction System.</p><p>These dimensional and relational models have their unique way of data storage that has specific advantages.</p><p>For instance, in the relational mode, normalization and ER models reduce redundancy in data. On the contrary, dimensional model arranges data in such a way that it is easier to retrieve information and generate reports.</p><p>Hence, Dimensional models are used in data warehouse systems and not a good fit for relational systems.</p><h3>Elements of Dimensional Data Model</h3><h4>Fact</h4><p>Facts are the measurements/metrics or facts from your business process. For a Sales business process, a measurement would be quarterly sales number</p><h4>Dimension</h4><p>Dimension provides the context surrounding a business process event. In simple terms, they give who, what, where of a fact. In the Sales business process, for the fact quarterly sales number, dimensions would be</p><ul><li>Who — Customer Names</li><li>Where — Location</li><li>What — Product Name</li></ul><p>In other words, a dimension is a window to view information in the facts.</p><h4>Attributes</h4><p>The Attributes are the various characteristics of the dimension.</p><p>In the Location dimension, the attributes can be</p><ul><li>State</li><li>Country</li><li>Zipcode etc.</li></ul><p>Attributes are used to search, filter, or classify facts. Dimension Tables contain Attributes</p><h4>Fact Table</h4><p>A fact table is a primary table in a dimensional model.</p><p>A Fact Table contains</p><ol><li>Measurements/facts</li><li>Foreign key to dimension table</li></ol><h4>Dimension table</h4><ul><li>A dimension table contains dimensions of a fact.</li><li>They are joined to fact table via a foreign key.</li><li>Dimension tables are de-normalized tables.</li><li>The Dimension Attributes are the various columns in a dimension table</li><li>Dimensions offers descriptive characteristics of the facts with the help of their attributes</li><li>No set limit set for given for number of dimensions</li><li>The dimension can also contain one or more hierarchical relationships</li></ul><h3>Steps of Dimensional Modelling</h3><p>The accuracy in creating your Dimensional modeling determines the success of your data warehouse implementation. Here are the steps to create Dimension Model</p><ol><li>Identify Business Process</li><li>Identify Grain (level of detail)</li><li>Identify Dimensions</li><li>Identify Facts</li><li>Build Star</li></ol><p>The model should describe the Why, How much, When/Where/Who and What of your business process</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*uWHbCyM-gGe4_Jut.png" /></figure><h4>Step 1) Identify the business process</h4><p>Identifying the actual business process a datarehouse should cover. This could be Marketing, Sales, HR, etc. as per the data analysis needs of the organization. The selection of the Business process also depends on the quality of data available for that process. It is the most important step of the Data Modelling process, and a failure here would have cascading and irreparable defects.</p><p>To describe the business process, you can use plain text or use basic Business Process Modelling Notation (BPMN) or Unified Modelling Language (UML).</p><h4>Step 2) Identify the grain</h4><p>The Grain describes the level of detail for the business problem/solution. It is the process of identifying the lowest level of information for any table in your data warehouse. If a table contains sales data for every day, then it should be daily granularity. If a table contains total sales data for each month, then it has monthly granularity.</p><p>During this stage, you answer questions like</p><ol><li>Do we need to store all the available products or just a few types of products? This decision is based on the business processes selected for Datawarehouse</li><li>Do we store the product sale information on a monthly, weekly, daily or hourly basis? This decision depends on the nature of reports requested by executives</li><li>How do the above two choices affect the database size?</li></ol><p><strong>Example of Grain:</strong></p><p>The CEO at an MNC wants to find the sales for specific products in different locations on a daily basis.</p><p>So, the grain is “product sale information by location by the day.”</p><h4>Step 3) Identify the dimensions</h4><p>Dimensions are nouns like date, store, inventory, etc. These dimensions are where all the data should be stored. For example, the date dimension may contain data like a year, month and weekday.</p><p><strong>Example of Dimensions:</strong></p><p>The CEO at an MNC wants to find the sales for specific products in different locations on a daily basis.</p><p>Dimensions: Product, Location and Time</p><p>Attributes: For Product: Product key (Foreign Key), Name, Type, Specifications</p><p>Hierarchies: For Location: Country, State, City, Street Address, Name</p><h4>Step 4) Identify the Fact</h4><p>This step is co-associated with the business users of the system because this is where they get access to data stored in the data warehouse. Most of the fact table rows are numerical values like price or cost per unit, etc.</p><p><strong>Example of Facts:</strong></p><p>The CEO at an MNC wants to find the sales for specific products in different locations on a daily basis.</p><p>The fact here is Sum of Sales by product by location by time.</p><h4>Step 5) Build Schema</h4><p>In this step, you implement the Dimension Model. A schema is nothing but the database structure (arrangement of tables). There are two popular schemas</p><ol><li><strong>Star Schema</strong></li></ol><p>The star schema architecture is easy to design. It is called a star schema because diagram resembles a star, with points radiating from a center. The center of the star consists of the fact table, and the points of the star is dimension tables.</p><p>The fact tables in a star schema which is third normal form whereas dimensional tables are de-normalized.</p><ol><li><strong>Snowflake Schema</strong></li></ol><p>The snowflake schema is an extension of the star schema. In a snowflake schema, each dimension are normalized and connected to more dimension tables.</p><h3>Rules for Dimensional Modelling</h3><ul><li>Load atomic data into dimensional structures.</li><li>Build dimensional models around business processes.</li><li>Need to ensure that every fact table has an associated date dimension table.</li><li>Ensure that all facts in a single fact table are at the same grain or level of detail.</li><li>It’s essential to store report labels and filter domain values in dimension tables</li><li>Need to ensure that dimension tables use a surrogate key</li><li>Continuously balance requirements and realities to deliver business solution to support their decision-making</li></ul><h3>Benefits of dimensional modeling</h3><ul><li>Standardization of dimensions allows easy reporting across areas of the business.</li><li>Dimension tables store the history of the dimensional information.</li><li>It allows to introduced entirely new dimension without major disruptions to the fact table.</li><li>Dimensional also to store data in such a fashion that it is easier to retrieve the information from the data once the data is stored in the database.</li><li>Compared to the normalized model dimensional table are easier to understand.</li><li>Information is grouped into clear and simple business categories.</li><li>The dimensional model is very understandable by the business. This model is based on business terms, so that the business knows what each fact, dimension, or attribute means.</li><li>Dimensional models are deformalized and optimized for fast data querying. Many relational database platforms recognize this model and optimize query execution plans to aid in performance.</li><li>Dimensional modeling creates a schema which is optimized for high performance. It means fewer joins and helps with minimized data redundancy.</li><li>The dimensional model also helps to boost query performance. It is more denormalized therefore it is optimized for querying.</li><li>Dimensional models can comfortably accommodate change. Dimension tables can have more columns added to them without affecting existing business intelligence applications using these tables.</li></ul><h3>Summary:</h3><ul><li>A dimensional model is a data structure technique optimized for Data warehousing tools.</li><li>Facts are the measurements/metrics or facts from your business process.</li><li>Dimension provides the context surrounding a business process event.</li><li>The Attributes are the various characteristics of the dimension.</li><li>A fact table is a primary table in a dimensional model.</li><li>A dimension table contains dimensions of a fact.</li><li>There are three types of facts 1. Additive 2. Non-additive 3. Semi- additive.</li><li>Types of Dimensions are Conformed, Outrigger, Shrunken, Role-playing, Dimension to Dimension Table, Junk, Degenerate, Swappable and Step Dimensions.</li><li>Five steps of Dimensional modeling are 1. Identify Business Process 2. Identify Grain (level of detail) 3. Identify Dimensions 4. Identify Facts 5. Build Star</li><li>In Dimensional modeling, there is need to ensure that every fact table has an associated date dimension table.</li></ul><p><strong><em>Alperen Kezay</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a59998b20ad1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/data-warehouse-training-episode-7-what-is-dimensional-modeling-a59998b20ad1">Data Warehouse Training — Episode 7 — What is Dimensional Modeling?</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Warehouse Training — Episode 6 — What is OLTP and OLTP VS OLAP]]></title>
            <link>https://medium.com/datasciencearth/data-warehouse-training-episode-6-what-is-oltp-and-oltp-vs-olap-76ce99076aee?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/76ce99076aee</guid>
            <category><![CDATA[olap]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[oltp]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Data Science Earth]]></dc:creator>
            <pubDate>Fri, 19 Mar 2021 08:50:50 GMT</pubDate>
            <atom:updated>2021-03-19T08:50:50.607Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Warehouse Training — Episode 6 — What is OLTP and OLTP VS OLAP</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZfM6rYYBmMNnEWlVh09-LA.png" /></figure><h3>What is OLTP? Definition, Architecture, Example</h3><h3>What is OLTP?</h3><p><strong>OLTP</strong> is an operational system that supports transaction-oriented applications in a 3-tier architecture. It administers the day to day transaction of an organization. OLTP is basically focused on query processing, maintaining data integrity in multi-access environments as well as effectiveness that is measured by the total number of transactions per second. The full form of OLTP is Online Transaction Processing.</p><h3>Characteristics of OLTP</h3><p>Following are important characteristics of OLTP:</p><ul><li>OLTP uses transactions that include small amounts of data.</li><li>Indexed data in the database can be accessed easily.</li><li>OLTP has a large number of users.</li><li>It has fast response times</li><li>Databases are directly accessible to end-users</li><li>OLTP uses a fully normalized schema for database consistency.</li><li>The response time of OLTP system is short.</li><li>It strictly performs only the predefined operations on a small number of records.</li><li>OLTP stores the records of the last few days or a week.</li><li>It supports complex data models and tables.</li></ul><h3>Type of queries that an OLTP system can Process:</h3><p>OLTP system is an online database changing system. Therefore, it supports database query such as insert, update, and delete information from the database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*EbE14bQK2KrlPw4o.png" /></figure><p>Consider a point of sale system of a supermarket, following are the sample queries that this system can process:</p><ul><li>Retrieving the description of a particular product.</li><li>Filtering all products related to the supplier.</li><li>Searching the record of the customer.</li><li>Listing products having a price less than the expected amount.</li></ul><h3>Architecture of OLTP</h3><p>Here is the architecture of OLTP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*ch9nKI8EnlQ5NK9e.png" /></figure><ol><li><strong>Business / Enterprise Strategy:</strong>Enterprise strategy deals with the issues that affect the organization as a whole. In OLTP, it is typically developed at a high level within the firm, by the board of directors or the top management</li><li><strong>Business Process:</strong>OLTP business process is a set of activities and tasks that, once completed, will accomplish an organizational goal.</li><li><strong>Customers, Orders, and Products:</strong>OLTP database store information about products, orders (transactions), customers (buyers), suppliers (sellers), and employees.</li><li><strong>ETL Processes:</strong>It separates the data from various RDBMS source systems, then transforms the data (like applying concatenations, calculations, etc.) and loads the processed data into the Data Warehouse system.</li><li><strong>Data Mart and Data warehouse: </strong>A data mart is a structure/access pattern specific to data warehouse environments.It is used by OLAP to store processed data.</li><li><strong>Data Mining, Analytics, and Decision Making: </strong>Data stored in the data mart and data warehouse can be used for data mining, analytics, and decision making.</li></ol><p>This data helps you to discover data patterns, analyze raw data, and make analytical decisions for your organization’s growth.</p><h3>Example of OLTP Transaction</h3><p>An example of the OLTP system is the ATM center. Assume that a couple has a joint account with a bank. One day both simultaneously reach different ATM centers at precisely the same time and want to withdraw the total amount present in their bank account.</p><p>OLTOP for ATM image</p><p>However, the person that completes the authentication process first will be able to get money. In this case, the OLTP system makes sure that the withdrawn amount will be never more than the amount present in the bank. The key to note here is that OLTP systems are optimized for transactional superiority instead of data analysis.</p><p>Other examples of OLTP system are:</p><ul><li>Online banking</li><li>Online airline ticket booking</li><li>Sending a text message</li><li>Order entry</li><li>Add a book to shopping cart</li></ul><h3>OLTP vs. OLAP</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*3jCb0atJazKzS9ve.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*Kqzom-qX63BzhhwbtxzZQA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*BjgT5Rc8FBpSEFljvavHmQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*9lcSa3VOwVh7Ps4hk6YZkA.jpeg" /></figure><h3>Advantages of OLTP:</h3><p>Following are the pros/benefits of OLTP system:</p><ul><li>OLTP offers accurate forecast for revenue and expense.</li><li>It provides a solid foundation for a stable business /organization due to timely modification of all transactions.</li><li>OLTP makes transactions much easier on behalf of the customers.</li><li>It broadens the client base for an organization by speeding up and simplifying individual processes.</li><li>OLTP provides support for bigger databases.</li><li>Partition of data for data manipulation is easy.</li><li>We need OLTP to use the tasks which are frequently performed by the system.</li><li>When we need only a small number of records.</li><li>The tasks that include insertion, updation, or deletion of data.</li><li>It is used when you need consistency and concurrency in order to perform tasks that ensure its greater availability.</li></ul><h3>Disadvantages of OLTP:</h3><p>Here are cons/drawbacks of OLTP system:</p><ul><li>If the OLTP system faces hardware failures, then online transactions get severely affected.</li><li>OLTP systems allow multiple users to access and change the same data at the same time, which many times created an unprecedented situation.</li><li>If the server hangs for seconds, it can affect to a large number of transactions.</li><li>OLTP required a lot of staff working in groups in order to maintain inventory.</li><li>Online Transaction Processing Systems do not have proper methods of transferring products to buyers by themselves.</li><li>OLTP makes the database much more susceptible to hackers and intruders.</li><li>In B2B transactions, there are chances that both buyers and suppliers miss out efficiency advantages that the system offers.</li><li>Server failure may lead to wiping out large amounts of data from the database.</li><li>You can perform a limited number of queries and updates.</li></ul><h3>Challenges of an OLTP System</h3><ul><li>It allows more than one user to access and change the same data simultaneously. Therefore, it requires concurrency control and recovery technique in order to avoid any unprecedented situations</li><li>OLTP system data are not suitable for decision making. You have to use data of OLAP systems for “what if” analysis or the decision making.</li></ul><h3>Summary</h3><ul><li>OLTP is defined as an operational system that supports transaction-oriented applications in a 3-tier architecture.</li><li>OLTP uses transactions that include small amounts of data.</li><li>OLTP system is an online database changing system.</li><li>The architecture of OLTP contains</li></ul><p>1) Business / Enterprise Strategy,</p><p>2) Business Process,</p><p>3) Customers, Orders, and Products,</p><p>4) ETL Processes,</p><p>5) Data Mart and Data warehouse, and</p><p>6) Data Mining, Analytics, and Decision Making.</p><ul><li>OLTP is an online transactional system, whereas OLAP is an online analysis and data retrieving process.</li><li>OLTP provides a solid foundation for a stable business /organization due to timely modification of all transactions.</li><li>OLTP systems allow multiple users to access and change the same data at the same time, which many times created an unprecedented.</li></ul><p><strong><em>Alperen Kezay</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76ce99076aee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/data-warehouse-training-episode-6-what-is-oltp-and-oltp-vs-olap-76ce99076aee">Data Warehouse Training — Episode 6 — What is OLTP and OLTP VS OLAP</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Warehouse Training — Episode 5 — What is OLAP and Types of OLAP]]></title>
            <link>https://medium.com/datasciencearth/data-warehouse-training-episode-5-what-is-olap-and-types-of-olap-f4fd743d11ba?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/f4fd743d11ba</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[olap]]></category>
            <dc:creator><![CDATA[Data Science Earth]]></dc:creator>
            <pubDate>Thu, 18 Mar 2021 10:08:30 GMT</pubDate>
            <atom:updated>2021-03-18T10:08:30.272Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Warehouse Training — Episode 5 — What is OLAP and Types of OLAP</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CKYTjSN-xi8as7fMBiwfSA.png" /></figure><h3>What is OLAP?</h3><p><strong>Online Analytical Processing (OLAP)</strong> is a category of software that allows users to analyze information from multiple database systems at the same time. It is a technology that enables analysts to extract and view business data from different points of view.</p><p>Analysts frequently need to group, aggregate and join data. These operations in relational databases are resource intensive. With OLAP data can be pre-calculated and pre-aggregated, making analysis faster.</p><p>OLAP databases are divided into one or more cubes. The cubes are designed in such a way that creating and viewing reports become easy. OLAP stands for Online Analytical Processing.</p><h3>OLAP Cube:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*7G2i9yT8HQ76tzNi.png" /></figure><p>At the core of the OLAP concept, is an OLAP Cube. The OLAP cube is a data structure optimized for very quick data analysis.</p><p>The OLAP Cube consists of numeric facts called measures which are categorized by dimensions. OLAP Cube is also called the <strong>hypercube</strong>.</p><p>Usually, data operations and analysis are performed using the simple spreadsheet, where data values are arranged in row and column format. This is ideal for two-dimensional data. However, OLAP contains multidimensional data, with data usually obtained from a different and unrelated source. Using a spreadsheet is not an optimal option. The cube can store and analyze multidimensional data in a logical and orderly manner.</p><p><strong>How does it work?</strong></p><p>A Data warehouse would extract information from multiple data sources and formats like text files, excel sheet, multimedia files, etc.</p><p>The extracted data is cleaned and transformed. Data is loaded into an OLAP server (or OLAP cube) where information is pre-calculated in advance for further analysis.</p><h4>Basic analytical operations of OLAP</h4><p>Four types of analytical operations in OLAP are:</p><ol><li>Roll-up</li><li>Drill-down</li><li>Slice and dice</li><li>Pivot (rotate)</li></ol><p><strong>1) Roll-up:</strong></p><p>Roll-up is also known as “consolidation” or “aggregation.” The Roll-up operation can be performed in 2 ways</p><ol><li>Reducing dimensions</li><li>Climbing up concept hierarchy. Concept hierarchy is a system of grouping things based on their order or level.</li></ol><p>Consider the following diagram</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*NseoWh388W0fWS_U.png" /></figure><ul><li>In this example, cities New jersey and Lost Angles and rolled up into country USA</li><li>The sales figure of New Jersey and Los Angeles are 440 and 1560 respectively. They become 2000 after roll-up</li><li>In this aggregation process, data is location hierarchy moves up from city to the country.</li><li>In the roll-up process at least one or more dimensions need to be removed. In this example, Quater dimension is removed.</li></ul><p><strong>2) Drill-down</strong></p><p>In drill-down data is fragmented into smaller parts. It is the opposite of the rollup process. It can be done via</p><ul><li>Moving down the concept hierarchy</li><li>Increasing a dimension</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*PGLmBcewSAuLJDKT.png" /></figure><p>Consider the diagram above</p><ul><li>Quater Q1 is drilled down to months January, February, and March. Corresponding sales are also registers.</li><li>In this example, dimension months are added.</li></ul><p><strong>3) Slice:</strong></p><p>Here, one dimension is selected, and a new sub-cube is created.</p><p>Following diagram explain how slice operation performed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*c8_31KNWgXsMC_4B.png" /></figure><ul><li>Dimension Time is Sliced with Q1 as the filter.</li><li>A new cube is created altogether.</li></ul><p><strong>Dice:</strong></p><p>This operation is similar to a slice. The difference in dice is you select 2 or more dimensions that result in the creation of a sub-cube.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/0*Blbr4VjYu6aY9Mp_.png" /></figure><p><strong>4) Pivot</strong></p><p>In Pivot, you rotate the data axes to provide a substitute presentation of data.</p><p>In the following example, the pivot is based on item types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/0*yC9vHZ5Y2rYzJFmH.png" /></figure><h3>Types of OLAP systems</h3><p>OLAP Hierarchical Structure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*RFZzzlYHA7t2UUxv.png" /></figure><h3>ROLAP</h3><p>ROLAP works with data that exist in a relational database. Facts and dimension tables are stored as relational tables. It also allows multidimensional analysis of data and is the fastest growing OLAP.</p><p><strong>Advantages of ROLAP model:</strong></p><ul><li><strong>High data efficiency.</strong> It offers high data efficiency because query performance and access language are optimized particularly for the multidimensional data analysis.</li><li><strong>Scalability.</strong> This type of OLAP system offers scalability for managing large volumes of data, and even when the data is steadily increasing.</li></ul><p><strong>Drawbacks of ROLAP model:</strong></p><ul><li><strong>Demand for higher resources: </strong>ROLAP needs high utilization of manpower, software, and hardware resources.</li><li><strong>Aggregately data limitations.</strong> ROLAP tools use SQL for all calculation of aggregate data. However, there are no set limits to the for handling computations.</li><li><strong>Slow query performance.</strong> Query performance in this model is slow when compared with MOLAP</li></ul><h3>MOLAP</h3><p>MOLAP uses array-based multidimensional storage engines to display multidimensional views of data. Basically, they use an OLAP cube.</p><p>Learn more about OLAP <a href="https://www.guru99.com/multidimensional-online-analytical-processing.html">here</a></p><h3>Hybrid OLAP</h3><p>Hybrid OLAP is a mixture of both ROLAP and MOLAP. It offers fast computation of MOLAP and higher scalability of ROLAP. HOLAP uses two databases.</p><ol><li>Aggregated or computed data is stored in a multidimensional OLAP cube</li><li>Detailed information is stored in a relational database.</li></ol><p><strong>Benefits of Hybrid OLAP:</strong></p><ul><li>This kind of OLAP helps to economize the disk space, and it also remains compact which helps to avoid issues related to access speed and convenience.</li><li>Hybrid HOLAP’s uses cube technology which allows faster performance for all types of data.</li><li>ROLAP are instantly updated and HOLAP users have access to this real-time instantly updated data. MOLAP brings cleaning and conversion of data thereby improving data relevance. This brings best of both worlds.</li></ul><p><strong>Drawbacks of Hybrid OLAP:</strong></p><ul><li>Greater complexity level<strong>: </strong>The major drawback in HOLAP systems is that it supports both ROLAP and MOLAP tools and applications. Thus, it is very complicated.</li><li>Potential overlaps<strong>: </strong>There are higher chances of overlapping especially into their functionalities.</li></ul><h3>Advantages of OLAP</h3><ul><li>OLAP is a platform for all type of business includes planning, budgeting, reporting, and analysis.</li><li>Information and calculations are consistent in an OLAP cube. This is a crucial benefit.</li><li>Quickly create and analyze “What if” scenarios</li><li>Easily search OLAP database for broad or specific terms.</li><li>OLAP provides the building blocks for business modeling tools, Data mining tools, performance reporting tools.</li><li>Allows users to do slice and dice cube data all by various dimensions, measures, and filters.</li><li>It is good for analyzing time series.</li><li>Finding some clusters and outliers is easy with OLAP.</li><li>It is a powerful visualization online analytical process system which provides faster response times</li></ul><h3>Disadvantages of OLAP</h3><ul><li>OLAP requires organizing data into a star or snowflake schema. These schemas are complicated to implement and administer</li><li>You cannot have large number of dimensions in a single OLAP cube</li><li>Transactional data cannot be accessed with OLAP system.</li><li>Any modification in an OLAP cube needs a full update of the cube. This is a time-consuming process</li></ul><h3>Summary:</h3><ul><li>OLAP is a technology that enables analysts to extract and view business data from different points of view.</li><li>At the core of the OLAP concept, is an OLAP Cube.</li><li>Various business applications and other data operations require the use of OLAP Cube.</li><li>There are primary five types of analytical operations in OLAP 1) Roll-up 2) Drill-down 3) Slice 4) Dice and 5) Pivot</li><li>Three types of widely used OLAP systems are MOLAP, ROLAP, and Hybrid OLAP.</li><li>Desktop OLAP, Web OLAP, and Mobile OLAP are some other types of OLAP systems.</li></ul><h3>What is MOLAP?</h3><p><strong>Multidimensional OLAP (MOLAP)</strong> is a classical OLAP that facilitates data analysis by using a multidimensional data cube. Data is pre-computed, re-summarized, and stored in a MOLAP (a major difference from ROLAP). Using a MOLAP, a user can use multidimensional view data with different facets.</p><p>Multidimensional data analysis is also possible if a relational database is used. By that would require querying data from multiple tables. On the contrary, MOLAP has all possible combinations of data already stored in a multidimensional array. MOLAP can access this data directly. Hence, MOLAP is faster compared to Relational Online Analytical Processing (ROLAP).</p><h3>Key Points</h3><ul><li>In MOLAP, operations are called processing.</li><li>MOLAP tools process information with the same amount of response time irrespective of the level of summarizing.</li><li>MOLAP tools remove complexities of designing a relational database to store data for analysis.</li><li>MOLAP server implements two level of storage representation to manage dense and sparse data sets.</li><li>The storage utilization can be low if the data set is sparse.</li><li>Facts are stored in multi-dimensional array and dimensions used to query them.</li></ul><h3>MOLAP Architecture</h3><p>MOLAP Architecture includes the following components</p><ul><li>Database server.</li><li>MOLAP server.</li><li>Front-end tool.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*TWo1-7TMjIABJTrx.png" /></figure><p>Consider above gien MOLAP Architectures:</p><ol><li>The user request reports through the interface</li><li>The application logic layer of the MDDB retrieves the stored data from Database</li><li>The application logic layer forwards the result to the client/user.</li></ol><p>MOLAP architecture mainly reads the precompiled data. MOLAP architecture has limited capabilities to dynamically create aggregations or to calculate results that have not been pre-calculated and stored.</p><p>For example, an accounting head can run a report showing the corporate P/L account or P/L account for a specific subsidiary. The MDDB would retrieve precompiled Profit &amp; Loss figures and display that result to the user.</p><h3>Implementation considerations is MOLAP</h3><ul><li>In MOLAP it’s essential to consider both maintenance and storage implications to creating strategy for building cubes.</li><li>Proprietary languages used to query MOLAP. However, it involves extensive click and drag support for example MDX by Microsoft.</li><li>Difficult to scale because the number and size of cubes required when dimensions increase.</li><li>API’s should provide for probing the cubes.</li><li>Data structure to support multiple subject areas of data analyses which data can be navigated and analyzed. When the navigation changes, the data structure needs to be physically reorganized.</li><li>Need different skill set and tools for Database administrator to build, maintain the database.</li></ul><h3>MOLAP Advantages</h3><ul><li>MOLAP can manage, analyze and store considerable amounts of multidimensional data.</li><li>Fast Query Performance due to optimized storage, indexing, and caching.</li><li>Smaller sizes of data as compared to the relational database.</li><li>Automated computation of higher level of aggregates data.</li><li>Help users to analyze larger, less-defined data.</li><li>MOLAP is easier to the user that’s why It is a suitable model for inexperienced users.</li><li>MOLAP cubes are built for fast data retrieval and are optimal for slicing and dicing operations.</li><li>All calculations are pre-generated when the cube is created.</li></ul><h3>MOLAP Disadvantages</h3><ul><li>One major weakness of MOLAP is that it is less scalable than ROLAP as it handles only a limited amount of data.</li><li>The MOLAP also introduces data redundancy as it is resource intensive</li><li>MOLAP Solutions may be lengthy, particularly on large data volumes.</li><li>MOLAP products may face issues while updating and querying models when dimensions are more than ten.</li><li>MOLAP is not capable of containing detailed data.</li><li>The storage utilization can be low if the data set is highly scattered.</li><li>It can handle the only limited amount of data therefore, it’s impossible to include a large amount of data in the cube itself.</li></ul><h3>MOLAP Tools</h3><ul><li><a href="https://www.oracle.com/business-analytics/essbase.html">Essbase</a> — Tools from Oracle that has a multidimensional database.</li><li><a href="http://www.oracle.com/technetwork/database/database-technologies/express-edition/downloads/index.html">Express Server</a> — Web-based environment that runs on Oracle database.</li><li><a href="https://www.yellowfinbi.com/">Yellowfin</a> — Business analytics tools for creating reports and dashboards.</li><li><a href="http://www.clearanalyticsbi.com/">Clear Analytics</a> — Clear analytics is an Excel-based business solution.</li><li><a href="https://support.sap.com/en/my-support/software-downloads.html">SAP Business Intelligence</a> — Business analytics solutions from SAP</li></ul><h3>Summary:</h3><ul><li>Multidimensional OLAP (MOLAP) is a classical OLAP that facilitates data analysis by using a multidimensional data cube.</li><li>MOLAP tools process information with the same amount of response time irrespective of the level of summarizing.</li><li>MOLAP server implements two level of storage to manage dense and sparse data sets.</li><li>MOLAP can manage, analyze, and store considerable amounts of multidimensional data.</li><li>It helps to automate computation of higher level of aggregates data</li><li>It is less scalable than ROLAP as it handles only a limited amount of data.</li></ul><p><strong><em>Alperen Kezay</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4fd743d11ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/data-warehouse-training-episode-5-what-is-olap-and-types-of-olap-f4fd743d11ba">Data Warehouse Training — Episode 5 — What is OLAP and Types of OLAP</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Warehouse Training — Episode 4 — ETL and ELT Process]]></title>
            <link>https://medium.com/datasciencearth/data-warehouse-training-episode-4-etl-and-elt-process-c8449090ee73?source=rss----e0aba57873ec---4</link>
            <guid isPermaLink="false">https://medium.com/p/c8449090ee73</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Data Science Earth]]></dc:creator>
            <pubDate>Thu, 18 Mar 2021 10:08:01 GMT</pubDate>
            <atom:updated>2021-03-20T14:14:12.955Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Warehouse Training — Episode 4 — ETL and ELT Process</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vdiqOOjt2F5n0w2IwQRt7A.png" /></figure><h3>ETL (Extract, Transform and Load) Process</h3><h4>What is ETL?</h4><p><strong>ETL</strong> is a process that extracts the data from different source systems, then transforms the data (like applying calculations, concatenations, etc.) and finally loads the data into the Data Warehouse system. Full form of ETL is Extract, Transform and Load.</p><p>It’s tempting to think a creating a Data warehouse is simply extracting data from multiple sources and loading into database of a Data warehouse. This is far from the truth and requires a complex ETL process. The ETL process requires active inputs from various stakeholders including developers, analysts, testers, top executives and is technically challenging.</p><p>In order to maintain its value as a tool for decision-makers, Data warehouse system needs to change with business changes. ETL is a recurring activity (daily, weekly, monthly) of a Data warehouse system and needs to be agile, automated, and well documented.</p><h4>Why do you need ETL?</h4><p>There are many reasons for adopting ETL in the organization:</p><ul><li>It helps companies to analyze their business data for taking critical business decisions.</li><li>Transactional databases cannot answer complex business questions that can be answered by ETL.</li><li>A Data Warehouse provides a common data repository</li><li>ETL provides a method of moving the data from various sources into a data warehouse.</li><li>As data sources change, the Data Warehouse will automatically update.</li><li>Well-designed and documented ETL system is almost essential to the success of a Data Warehouse project.</li><li>Allow verification of data transformation, aggregation and calculations rules.</li><li>ETL process allows sample data comparison between the source and the target system.</li><li>ETL process can perform complex transformations and requires the extra area to store the data.</li><li>ETL helps to Migrate data into a Data Warehouse. Convert to the various formats and types to adhere to one consistent system.</li><li>ETL is a predefined process for accessing and manipulating source data into the target database.</li><li>ETL offers deep historical context for the business.</li><li>It helps to improve productivity because it codifies and reuses without a need for technical skills.</li></ul><h4>ETL Process in Data Warehouses</h4><p>ETL is a 3-step process</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*VGk8VkYVTpQXR2jN.png" /></figure><p><strong>Step 1) Extraction</strong></p><p>In this step, data is extracted from the source system into the staging area. Transformations if any are done in staging area so that performance of source system in not degraded. Also, if corrupted data is copied directly from the source into Data warehouse database, rollback will be a challenge. Staging area gives an opportunity to validate extracted data before it moves into the Data warehouse.</p><p>Data warehouse needs to integrate systems that have different</p><p>DBMS, Hardware, Operating Systems and Communication Protocols. Sources could include legacy applications like Mainframes, customized applications, Point of contact devices like ATM, Call switches, text files, spreadsheets, ERP, data from vendors, partners amongst others.</p><p>Hence one needs a logical data map before data is extracted and loaded physically. This data map describes the relationship between sources and target data.</p><p><strong>Three Data Extraction methods:</strong></p><ol><li>Full Extraction</li><li>Partial Extraction- without update notification.</li><li>Partial Extraction- with update notification</li></ol><p>Irrespective of the method used, extraction should not affect performance and response time of the source systems. These source systems are live production databases. Any slow down or locking could effect company’s bottom line.</p><p><strong>Some validations are done during Extraction:</strong></p><ul><li>Reconcile records with the source data</li><li>Make sure that no spam/unwanted data loaded</li><li>Data type check</li><li>Remove all types of duplicate/fragmented data</li><li>Check whether all the keys are in place or not</li></ul><p><strong>Step 2) Transformation</strong></p><p>Data extracted from source server is raw and not usable in its original form. Therefore it needs to be cleansed, mapped and transformed. In fact, this is the key step where ETL process adds value and changes data such that insightful BI reports can be generated.</p><p>In this step, you apply a set of functions on extracted data. Data that does not require any transformation is called as <strong>direct move</strong> or <strong>pass through data</strong>.</p><p>In transformation step, you can perform customized operations on data. For instance, if the user wants sum-of-sales revenue which is not in the database. Or if the first name and the last name in a table is in different columns. It is possible to concatenate them before loading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*CL0SJz_Vubh1xhP6.png" /></figure><p><strong>Following are Data Integrity Problems:</strong></p><ol><li>Different spelling of the same person like Jon, John, etc.</li><li>There are multiple ways to denote company name like Google, Google Inc.</li><li>Use of different names like Cleaveland, Cleveland.</li><li>There may be a case that different account numbers are generated by various applications for the same customer.</li><li>In some data required files remains blank</li><li>Invalid product collected at POS as manual entry can lead to mistakes.</li></ol><p><strong>Validations are done during this stage</strong></p><ul><li>Filtering — Select only certain columns to load</li><li>Using rules and lookup tables for Data standardization</li><li>Character Set Conversion and encoding handling</li><li>Conversion of Units of Measurements like Date Time Conversion, currency conversions, numerical conversions, etc.</li><li>Data threshold validation check. For example, age cannot be more than two digits.</li><li>Data flow validation from the staging area to the intermediate tables.</li><li>Required fields should not be left blank.</li><li>Cleaning (for example, mapping NULL to 0 or Gender Male to “M” and Female to “F” etc.)</li><li>Split a column into multiples and merging multiple columns into a single column.</li><li>Transposing rows and columns,</li><li>Use lookups to merge data</li><li>Using any complex data validation (e.g., if the first two columns in a row are empty then it automatically reject the row from processing)</li></ul><p><strong>Step 3) Loading</strong></p><p>Loading data into the target datawarehouse database is the last step of the ETL process. In a typical Data warehouse, huge volume of data needs to be loaded in a relatively short period (nights). Hence, load process should be optimized for performance.</p><p>In case of load failure, recover mechanisms should be configured to restart from the point of failure without data integrity loss. Data Warehouse admins need to monitor, resume, cancel loads as per prevailing server performance.</p><p><strong>Types of Loading:</strong></p><ul><li><strong>Initial Load</strong> — populating all the Data Warehouse tables</li><li><strong>Incremental Load </strong>— applying ongoing changes as when needed periodically.</li><li><strong>Full Refresh</strong> — erasing the contents of one or more tables and reloading with fresh data.</li></ul><h4>Load verification</h4><ul><li>Ensure that the key field data is neither missing nor null.</li><li>Test modeling views based on the target tables.</li><li>Check that combined values and calculated measures.</li><li>Data checks in dimension table as well as history table.</li><li>Check the BI reports on the loaded fact and dimension table.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*7OOupoCsDjPsgPzk.png" /></figure><h3>ETL tools</h3><p>There are many Data Warehousing tools are available in the market. Here, are some most prominent one:</p><ol><li><strong>MarkLogic:</strong></li></ol><p>MarkLogic is a data warehousing solution which makes data integration easier and faster using an array of enterprise features. It can query different types of data like documents, relationships, and metadata.</p><p><a href="https://developer.marklogic.com/products/">https://developer.marklogic.com/products/</a></p><p><strong>2. Oracle:</strong></p><p>Oracle is the industry-leading database. It offers a wide range of choice of Data Warehouse solutions for both on-premises and in the cloud. It helps to optimize customer experiences by increasing operational efficiency.</p><p><a href="https://www.oracle.com/index.html">https://www.oracle.com/index.html</a></p><p><strong>3. Amazon RedShift:</strong></p><p>Amazon Redshift is Datawarehouse tool. It is a simple and cost-effective tool to analyze all types of data using standard SQL and existing BI tools. It also allows running complex queries against petabytes of structured data.</p><p><a href="https://aws.amazon.com/redshift/?nc2=h_m1">https://aws.amazon.com/redshift/?nc2=h_m1</a></p><p>Here is a complete list of useful <a href="https://www.guru99.com/top-20-etl-database-warehousing-tools.html">Data warehouse Tools.</a></p><h3>Best practices ETL process</h3><p><strong>Never try to cleanse all the data:</strong></p><p>Every organization would like to have all the data clean, but most of them are not ready to pay to wait or not ready to wait. To clean it all would simply take too long, so it is better not to try to cleanse all the data.</p><p><strong>Never cleanse Anything:</strong></p><p>Always plan to clean something because the biggest reason for building the Data Warehouse is to offer cleaner and more reliable data.</p><p><strong>Determine the cost of cleansing the data:</strong></p><p>Before cleansing all the dirty data, it is important for you to determine the cleansing cost for every dirty data element.</p><p><strong>To speed up query processing, have auxiliary views and indexes:</strong></p><p>To reduce storage costs, store summarized data into disk tapes. Also, the trade-off between the volume of data to be stored and its detailed usage is required. Trade-off at the level of granularity of data to decrease the storage costs.</p><h4>Summary:</h4><ul><li>ETLstands for Extract, Transform and Load.</li><li>ETL provides a method of moving the data from various sources into a data warehouse.</li><li>In the first step extraction, data is extracted from the source system into the staging area.</li><li>In the transformation step, the data extracted from source is cleansed and transformed.</li><li>Loading data into the target datawarehouse is the last step of the ETL process.</li></ul><h3>What is ELT?</h3><p>ELT is a different method of looking at the tool approach to data movement. Instead of transforming the data before it’s written, ELT lets the target system to do the transformation. The data first copied to the target and then transformed in place.</p><p>ELT usually used with no-Sql databases like Hadoop cluster, data appliance or cloud installation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*yXomcEUa4u_etkWr.png" /></figure><h4>KEY DIFFERENCE</h4><ul><li>ETL stands for Extract, Transform and Load while ELT stands for Extract, Load, Transform.</li><li>ETL loads data first into the staging server and then into the target system whereas ELT loads data directly into the target system.</li><li>ETL model is used for on-premises, relational and structured data while ELT is used for scalable cloud structured and unstructured data sources.</li><li>ETL is mainly used for a small amount of data whereas ELT is used for large amounts of data.</li><li>ETL doesn’t provide data lake supports while ELT provides data lake support.</li><li>ETL is easy to implement whereas ELT requires niche skills to implement and maintain.</li></ul><p><strong><em>Alperen Kezay</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8449090ee73" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datasciencearth/data-warehouse-training-episode-4-etl-and-elt-process-c8449090ee73">Data Warehouse Training — Episode 4 — ETL and ELT Process</a> was originally published in <a href="https://medium.com/datasciencearth">Data Science Earth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>