<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Professor Simon J. Preis, Ph.D. on Medium]]></title>
        <description><![CDATA[Stories by Professor Simon J. Preis, Ph.D. on Medium]]></description>
        <link>https://medium.com/@simon-j-preis?source=rss-8c9dbf8eb438------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vY_guJtSGbGVf8sysWlFQQ.jpeg</url>
            <title>Stories by Professor Simon J. Preis, Ph.D. on Medium</title>
            <link>https://medium.com/@simon-j-preis?source=rss-8c9dbf8eb438------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:30:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@simon-j-preis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Are Expert Systems Dead?]]></title>
            <link>https://medium.com/data-science/are-expert-systems-dead-87c8d6c26474?source=rss-8c9dbf8eb438------2</link>
            <guid isPermaLink="false">https://medium.com/p/87c8d6c26474</guid>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[knowledge-management]]></category>
            <category><![CDATA[expert-systems]]></category>
            <dc:creator><![CDATA[Professor Simon J. Preis, Ph.D.]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 17:07:48 GMT</pubDate>
            <atom:updated>2023-03-16T17:07:48.066Z</atom:updated>
            <content:encoded><![CDATA[<h4>A review of recent trends, use cases and technologies</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MyO0h5Cv-65IRdSZ" /><figcaption>Photo by <a href="https://unsplash.com/@lephunghia?utm_source=medium&amp;utm_medium=referral">Nghia Le</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Everyone is talking about machine learning. And this is not just a feeling: if you search Google for “machine learning”, you will receive almost 700 million results. But what about expert systems? Traditionally, it is the other side of the AI coin. Well, Google returns at least around 7 million results, but there is a clear gap between those two concepts. This mismatch fits also to the Google search trends: while interest in machine learning (orange line) has increased significantly in recent years, one can get the impression that expert systems (blue line) have somehow fallen into oblivion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fWw2QSQdI3lhiNbRdsiQlg.png" /><figcaption>Google Search Trends in % (Author, Data from Google)</figcaption></figure><p>So is this just a result of the AI evolution (survival of the fittest and hopefully not the “overfittest” … sorry for that AI joke ;-)? Are expert systems really outdated? Or is there still room for expert systems in the one or the other research area? I thought it would be good to collect and share some findings and experiences in order to take a stand for expert systems!</p><p>Contents:</p><ol><li>what are expert systems?</li><li>are expert systems relevant in topical research?</li><li>what are use cases for expert systems?</li><li>what are modern technologies for developing expert systems?</li></ol><h3>What are expert systems?</h3><p>An expert system (ES) is a piece of software that consists at least of a knowledge (data-)base, a domain-related set of rules and an inference engine that is able to infer new axioms. It is called ES since it acts like a human expert in a particular domain, e.g. who is able to answer tricky questions related to his/her field of expertise. From application point of view, ES have two main types of users: a) the human experts who continuously review and modify the knowledge base and b) the end users who seek for answers to domain-related questions. Compared to often called “black-box AI” approaches from deep learning, the result of an ES is always transparent: the inference engine typically provides step-wise explanations for its results and the users are able to comprehend their logical construction.</p><h3>Are expert systems relevant in topical research?</h3><p>In order to clarify the question, if ES are really dead, it is worth to look at the number of publications that are employed with ES or at least reference that term. For that purpose, I did a simple search on Google Scholar and checked the number of publications from 2005 to 2022 that contain the term ES. Let’s formulate the following working hypothesis: the publication trend for ES over the selected years is negative (to confirm that “ES are dead”).</p><p>Now let’s check what the data shows. I have created a diagram that shows the absolute numbers of publications per time frame and a trend line. The diagram clearly shows that the publication trend is still positive. Though the literature search is very high-level and noisy, my expectation is that ES would not be named in topical articles if the term would be outdated; so we can reject the hypothesis stated above. We can imply that ES are still vital and play a crucial role in research.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*sxr7u7ZOaqZuzIXAmfRoTA.png" /><figcaption>Research Trend for Expert Systems (Author)</figcaption></figure><h3>What are use cases for expert systems?</h3><p>So what are these topical research disciplines that develop and propose ES? Which domains do they serve, which requirements do they fulfill, what is the main value of ES in those projects? Of course, these are great questions to be answered via systematic literature review — but for now, I will only present three selected projects that should demonstrate the bandwidth of ES.</p><h4>Cyber Security</h4><p>ES can be used to for automated security assessment of IoT ecosystems. Rak et al. (2022) proposed an ES that produces a threat model and a list of attack plans for each identified threat for a IoT system. The results provided by the ES can be used by penetration testers to perform a systematic security test of the target IoT infrastructure.</p><h4>Project Management</h4><p>Bhattacharya et al. (2022) proposed an ES for decision making in complex regulatory and technology implementation projects. For instance, the ES validates if the project start conditions are met and allows tailoring the project plan based on project type and complexity.</p><h4>Clinical Decision Support</h4><p>Chrimes (2023) proposed an ES to support clinical decisions for COVID-19. The ES interacts with users via a chatbot to determine the potential severity of a COVID-19 infection or the possible biological system responses and comorbidities that can contribute to the development of severe cases of COVID-19.</p><h3>Modern Technologies for ES</h3><p>Especially in pure research projects, we often see Protégé as ES development tool, also in topical projects. The tool combines storage of knowledge (entities and object properties), rules (SWRL), facts (individuals) and provides an inference engine as pre-installed plugin (HermiT). I still believe that Protégé is important in education and research, especially since it is free of charge and easy to start. But I would not recommend to use it in industry projects due to its limitations regarding system integration and user interface design. In my opinion, it is a tool for universities, not for companies.</p><p>However, there are modern tools that share some of the features of Protégé and that are industry-proven. Most important to name are graph databases which allow semantic storage and semantic analysis of information. So up to a certain degree, we can represent ontologies as knowledge graphs. For that purpose, neo4j provides a plugin called “neosemantics” for RDF, which is the data model standard for ontologies also in Protégé. So we can, for instance, create ontologies in Protégé and export them to neo4j for further software integration. The plugin neosemantics also provides an inference engine, however, we can do reasoning also “by foot”via Cypher, which is the standard query language for neo4j.</p><h4>An example</h4><p>But what is inferencing or reasoning? Let’s have a look at following database that consists of persons and genders. We see that there is a number of relationships that indicate the gender of a person (in a simplified binary world) with “IS_A” and also family relationship between person nodes with “IS_Parent”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*CT0lhtnI_G5QqrMOSsrtaA.png" /><figcaption>Initial Database (Author)</figcaption></figure><p>Now we can distinguish between explicitly stored knowledge and transitive knowledge. The explicit knowledge is what we see in the graph and what I have described in the paragraph above. However, from our everyday experience we know that there are more relationships between those persons based on this available information: siblings, grandparents, uncles etc. And instead of searching and/or entering all these relationships for all facts in the database manually, we can use the reasoning technique to create this knowledge automatically.</p><h4>Grandparents</h4><p>Let’s start with an easy one: grandparents are the parents of the parents of a person. So looking on our graph database, we are looking for multilevel “IS_Parent” relationships like PersonA →PersonB →PersonC. In this case, we would state that PersonC is a grandchild to PersonA and PersonA is a grandparent to PersonC. We can infer that information via following generic statement:</p><pre>MATCH (a:Person)-[:IS_Parent]-&gt;(b:Person)-[:IS_Parent]-&gt;(c:Person) <br>return a.name as grandparent, c.name as grandchild<br><br></pre><p>The result of this statement is shown in following screenshot. We see that Max and Anna are both grandchildren of Betty. Why? Because Betty is a parent to Franz who is a parent to Max and Anna.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*KTVQx2VMBdoG99adNq5Rpw.png" /><figcaption>Grandparent Result (Author)</figcaption></figure><h4>Siblings</h4><p>Another type of transitive knowledge is the sibling relationship. We are searching for persons who have the same parents, this means we are looking for common “IS_Parent” relationships for PersonA ←PersonB and PersonC ←PersonB. We can query that information via following statement:</p><pre>MATCH (a:Person)&lt;-[:IS_Parent]-(b:Person),<br> (c:Person)&lt;-[:IS_Parent]-(b:Person)<br>return distinct a.name as sibling_a, c.name as sibling_b</pre><p>The result of this statement is shown in the following screenshot. We see that Max and Anna are siblings. Since the matching query fits for both perspectives, we get two result records. This is also technically valid, because neo4j does not support bidirectional or non-directed relationships. If Anna is a sibling to Max and Max is a sibling to Anna, then we have two separate relationships.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*qOoF0vr2CV1NbqlL6nMahg.png" /><figcaption>Sibling Result (Author)</figcaption></figure><h4>Persisting Transitive Knowledge</h4><p>There are use cases for such on-demand queries, but there are other use cases where we wish to store the new transitive knowledge to the database as well. This is especially useful, if multiple users work with the database — here it does not make sense to “hide” any transitive knowledge that is potentially business-relevant. In that case, we can use the merge clause to create new (unique) relationships between nodes as shown in the following statement:</p><pre>Match (a:Person)&lt;-[:IS_Parent]-(b:Person),<br> (c:Person)&lt;-[:IS_Parent]-(b:Person) <br>Merge  (a)-[r:IS_SIBLING]-&gt;(c)</pre><p><em>Note: besides new relationships, we could also infer and persist new nodes, labels and properties.</em></p><p>If we are then querying the full database, we see the additional “IS_SIBLING” relationships between Anna and Max:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*krXTIx-0FlwzQ65DC0yj9Q.png" /><figcaption>Modified database (Author)</figcaption></figure><p>It should be highlighted that this type of manual reasoning does not consider forward/backward chaining which are usually required features of inference engines, e.g. to support complex theorem proofs. So if you are interested in these features, you can start reading <a href="https://www.geeksforgeeks.org/difference-between-backward-and-forward-chaining/">here</a> (general introduction) and <a href="https://neo4j.com/labs/neosemantics/4.0/inference/">here</a> (inferencing with neo4j).</p><h4>Self-Learning Expert Systems</h4><p>Now the question is if we as humans always need to know these rules upfront or have to prepare manually the rules we want to query in order to extract transitive knowledge. The short answer is: no! And this leads us to an interesting link between the two often separated worlds of data-based machine learning and rule-based expert systems: we can apply decision trees to build rules from empirical data that are still “white-box” since they are transparent to the user. Decision trees are not limited to graph databases and the scikit-learn library in Python requires a data structure transformation from graph to a flat structure, e.g. pandas dataframe — so let’s ignore the graph model for now and jump directly to the point where we have a flat dataframe with selected numeric features.</p><p>Let’s assume we are working for a car seller and we want to build an expert system that tells the salesperson if a certain visitor is likely to purchase a car or not. Instead of developing rules from our own experience, we can use the historical sales data to derive the rules from previous purchase decisions. And this is something we should note when we are talking about decision trees: we are dealing with probabilities. Even though the tree would predict that a person is likely to buy a car, it does not mean that this person will really buy the car. We see a possible result of a decision tree classification for our car seller in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cpISPLGf7IVsDP16b1WwMA.png" /><figcaption>Decision Tree Classification for Car Seller (Author)</figcaption></figure><p>Besides the graphical representation, we can also extract the new decision rules in a logical “if-then” format as described in <a href="https://mljar.com/blog/extract-rules-decision-tree/">this article</a>.</p><pre>if (Age &lt;= 44.5) and (Annual Salary &lt;= 90750.0) and (Annual Salary &lt;= 69750.0) then class: No Purchase (proba: 100.0%) | based on 258 samples<br>if (Age &lt;= 44.5) and (Annual Salary &lt;= 90750.0) and (Annual Salary &gt; 69750.0) then class: No Purchase (proba: 89.02%) | based on 173 samples<br>if (Age &gt; 44.5) and (Age &gt; 47.5) and (Annual Salary &gt; 41750.0) then class: Purchase (proba: 82.78%) | based <br>on 151 samples<br>if (Age &gt; 44.5) and (Age &gt; 47.5) and (Annual Salary &lt;= 41750.0) then class: Purchase (proba: 98.46%) | based on 65 samples<br>if (Age &lt;= 44.5) and (Annual Salary &gt; 90750.0) and (Annual Salary &lt;= 119750.0) then class: Purchase (proba: <br>67.35%) | based on 49 samples<br>if (Age &lt;= 44.5) and (Annual Salary &gt; 90750.0) and (Annual Salary &gt; 119750.0) then class: Purchase (proba: 97.67%) | based on 43 samples<br>if (Age &gt; 44.5) and (Age &lt;= 47.5) and (Annual Salary &gt; 53250.0) then class: No Purchase (proba: 50.0%) | based on 34 samples<br>if (Age &gt; 44.5) and (Age &lt;= 47.5) and (Annual Salary &lt;= 53250.0) then class: Purchase (proba: 85.19%) | based on 27 samples</pre><p>So let’s assume we have validated this decision tree model and integrated it with a user interface, then the car salesperson could enter some personal visitor information to get a response which tells if this visitor is likely to buy a car or not. Let’s view following examples:</p><p>a) if a potential customer enters the shop with following attributes: female, age=28 and annual salary=78.000 — is this person likely to buy a car? → our decision tree says: no (classified with label “0”).</p><p>b) if a potential customer enters the shop with following attributes: male, age=28 and annual salary=118.000 — is this person likely to buy a car? → our decision tree says: yes(classified with label “1”).</p><p>We see that such a system would provide similar functionality like a traditional expert system for this car seller domain. The benefit is that this decision-tree-based expert system is able to learn from new data in order to improve the rules continuously. However, we need to keep in mind that this type of expert system works with probabilities, hence, the responses are typically not 100% accurate. With traditional expert systems, due to the application of logical inference, the responses are logically accurate (at least if the basic axioms are correct). But we need to consider the high manual efforts to build and maintain the knowledge base and rules, and we need also to consider that even experts can be wrong, which requires extra validation efforts. So in the end it’s a trade-off between maintenance efficiency and prediction quality.</p><h3>Conclusions</h3><p>After defining what ES are at all, we have seen that ES are still used, developed or at least referenced in research publications and the publication trend is even positive. From that perspective, we can clearly answer the initial question: are ES dead? No! We have discussed a few topical projects and their domains to see the bandwidth of ES. Finally, we have discussed some features of graph databases, represented by neo4j, to perform tasks that are typically associated with ES. These essential tasks are on the one hand the semantic storage of explicit knowledge and on the other hand the rule-based reasoning of new transitive knowledge. Of course, neo4j provides much more features in terms of data storage and analysis, but I think that the easy examples from above are comprehensible to understand the capabilities of neo4j as a knowledge base as part of an ES. In addition, we have briefly discussed how we can build rules from data via decision trees, which allows the development of self-learning expert systems.</p><p>So, when should we think about applying ES?</p><ol><li>in cases where we simply<strong> do not have data</strong> (yet) to employ machine learning models. Example: to develop and justify qualification plans for new product releases without reliable reference products.</li><li>in cases where we are <strong>rule-driven </strong>and we know the rules already and the rules are static and not directed by data. Example: to infer business-defined categorical master data based on other detailed information, e.g. the particular structure and setup of a production machine.</li><li>in cases where we <strong>have data and we want to use and understand decision rules </strong>and we are aware that <strong>rules may change </strong>over time. Instead of developing and maintaining rules manually, we can employ decision trees. Example: to understand and infer customer behaviors based on empirical data.</li></ol><p>And to close this article with a fun fact: I asked ChatGPT and it does <em>not </em>consider itself as an expert system, because it is not trained for a certain domain in order to provide expert-level advise.</p><h3>Sources</h3><p>Bhattacharya, K., Gangopadhyay, S., DeBrule, C. (2021). Design of an Expert System for Decision Making in Complex Regulatory and Technology Implementation Projects. In: Chakrabarti, A., Poovaiah, R., Bokil, P., Kant, V. (eds) Design for Tomorrow — Volume 3. Smart Innovation, Systems and Technologies, vol 223. Springer, Singapore. <a href="https://doi.org/10.1007/978-981-16-0084-5_50">https://doi.org/10.1007/978-981-16-0084-5_50</a></p><p>Chrimes, D. (2023). Using Decision Trees as an Expert System for Clinical Decision Support for COVID-19. Interact J Med Res 2023; 12:e42540<br>URL: <a href="https://www.i-jmr.org/2023/1/e42540">https://www.i-jmr.org/2023/1/e42540</a>. DOI: 10.2196/42540</p><p>Rak, Massimiliano &amp; Salzillo, Giovanni &amp; Granata,Daniele. (2022). ESSecA: An automated expert system for threat modelling and penetration testing for IoT ecosystems, Computers and Electrical Engineering, Volume 99,2022, 107721, ISSN 0045–7906, <a href="https://doi.org/10.1016/j.compeleceng.2022.107721">https://doi.org/10.1016/j.compeleceng.2022.107721 </a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87c8d6c26474" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/are-expert-systems-dead-87c8d6c26474">Are Expert Systems Dead?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Develop a Data Quality Management Project with ChatGPT]]></title>
            <link>https://simon-j-preis.medium.com/how-to-develop-a-data-quality-management-project-with-chatgpt-4c537d1d8994?source=rss-8c9dbf8eb438------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c537d1d8994</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <category><![CDATA[semiconductors]]></category>
            <category><![CDATA[data-governance]]></category>
            <dc:creator><![CDATA[Professor Simon J. Preis, Ph.D.]]></dc:creator>
            <pubDate>Sun, 29 Jan 2023 14:51:14 GMT</pubDate>
            <atom:updated>2023-01-29T22:12:35.699Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>One of the critical issues in Data Quality Management (DQM) that I experienced is the initial knowledge gap between business experts and DQ experts. Especially in the early phase of DQM in a company, when even the DQM expert is new in his/her role, it can be time-consuming and frustrating to identify a suitable pilot to demonstrate the power of DQM. How can we speed up that process to save time, efforts and frustrations?</p><p>Maybe that’s a job for A.I.! We have seen a lot of funny results when dealing with ChatGPT and we have seen a lot of discussion around its limitations (or both, e.g. when people convince it that 5+2=8, because “my wife said that”). In this story, I will use the chatbot to develop a DQM project charter step-by-step. For the sake of easier reading, I won’t include the full answers here, just the essence of ChatGPT’s feedback. Due to my professional background, I selected the semiconductor manufacturing as test scenario.</p><p>I will go through the following process in order to develop the project charter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X2g5CZyVO3_nACU7SXnQqw.png" /><figcaption>Source: Author</figcaption></figure><h3>Problem Statement: What are the most critical data quality issues in semiconductor manufacturing?</h3><ol><li>Data Integrity</li><li>Data Security</li><li>Data Governance</li><li>Data Quality Monitoring</li><li>Traceability</li><li>Data Interoperability</li></ol><h3>Business Case</h3><p>According to recent research, the monetary cost of data integrity issues in a semiconductor production line can be significant. A study published in the Journal of Manufacturing Technology Management in 2017 found that poor data integrity in the lithography process can lead to increased costs for rework and scrap, reduced yield, and decreased product quality. The study estimates that the total cost of poor data integrity in the lithography process can be as high as $1 million per year.</p><p>Another study published in the Journal of Micro/Nanolithography, MEMS, and MOEMS in 2020 found that poor data integrity in the lithography process can lead to increased reticle defects, decreased yield, and increased costs for rework and scrap. The study estimates that the total cost of poor data integrity in the lithography process can be as high as $500,000 per year.</p><p>In addition to these costs, data integrity issues can also lead to decreased production efficiency, increased downtime, and lost customers due to decreased product quality.</p><h3>Pilot area</h3><p>According to research, the lithography process is one of the unit processes that is most at risk to be influenced by poor data integrity in the wafer fabrication process. The root causes of poor data integrity in this process can include issues such as equipment malfunctions, human error, or data entry errors.</p><h3>Root Causes of Data Entry Errors at Lithography Processes</h3><p>Data entry errors at lithography processes can occur in several ways and can have a significant impact on the production process and final product quality. Some common types of data entry errors include:</p><ol><li>Incorrect data entry: This type of error occurs when operators enter incorrect data into the system, such as incorrect process parameters or incorrect equipment settings.</li><li>Data entry omissions: This type of error occurs when operators fail to enter all of the necessary data into the system, such as leaving out critical process parameters or equipment settings.</li><li>Data entry duplications: This type of error occurs when operators enter the same data multiple times into the system, leading to inconsistencies in the data.</li></ol><p>The root causes of data entry errors at lithography processes can include a lack of training or knowledge, poor communication, and high-stress work environments.</p><h3>Solutions</h3><ol><li>Providing training and education to operators on how to properly enter data into the system.</li><li>Having clear communication channels in place between operators and management to ensure that all necessary data is entered into the system.</li><li>Implementing a data validation system that can automatically check for errors in the data before it is entered into the system.</li><li>Having a standard operating procedure (SOP) in place that lays out the specific data that needs to be entered into the system, and how it should be entered.</li><li>Encouraging double-checking of the data before it is entered into the system to minimize errors.</li><li>Implementing a data management system that allows for easy access and viewing of the data and can alert if there is any errors or inconsistencies in the data.</li></ol><h3>Project Charter</h3><p><strong>Goals:</strong></p><ol><li>To reduce data entry errors by providing training and education to operators on how to properly enter data into the system.</li><li>To improve communication channels between operators and management to ensure that all necessary data is entered into the system.</li><li>To implement a data validation system that can automatically check for errors in the data before it is entered into the system.</li><li>To develop a standard operating procedure (SOP) that lays out the specific data that needs to be entered into the system, and how it should be entered.</li><li>To encourage double-checking of the data before it is entered into the system to minimize errors.</li><li>To implement a data management system that allows for easy access and viewing of the data and can alert if there is any errors or inconsistencies in the data.</li></ol><p><strong>Scope</strong>:</p><ul><li>The project will focus on the lithography process, but it is important to keep in mind that data integrity issues can occur in any process of the semiconductor production line.</li><li>The project will be applied to the entire production line.</li></ul><p><strong>Timeline</strong>:</p><ul><li>The project should be implemented in phases, starting with the training and education of the operators, and then moving on to the implementation of the data validation system and the development of the SOP.</li><li>The project should be completed within 6–12 months.</li></ul><p><strong>Team members:</strong></p><ul><li>Project manager</li><li>Process engineers</li><li>Data analysts</li><li>IT specialists</li><li>Trainers</li><li>Quality assurance team</li></ul><p>It is important to note that the success of this project will depend on the collaboration and cooperation between all the team members, and that the training and education of the operators is crucial to the success of the project.</p><h3>Milestone Plan</h3><p>Milestone 1: Training and Education of Operators</p><ul><li>Responsible Person: Process Engineer</li><li>Required Team Members: Trainers, Process Engineers</li><li>Planned FTE per Team Member: 1 FTE per team member</li><li>Planned Month of Completion: Month 1</li><li>Acceptance Criteria: All operators have completed the training and education program, and pass a test demonstrating their understanding of proper data entry procedures.</li></ul><p>Milestone 2: Implementation of Data Validation System</p><ul><li>Responsible Person: IT Specialist</li><li>Required Team Members: IT Specialists, Data Analysts</li><li>Planned FTE per Team Member: 1 FTE per team member</li><li>Planned Month of Completion: Month 2</li><li>Acceptance Criteria: The data validation system is in place and is able to check for errors in data before it is entered into the system.</li></ul><p>Milestone 3: Development of Standard Operating Procedure (SOP)</p><ul><li>Responsible Person: Process Engineer</li><li>Required Team Members: Process Engineers, Quality Assurance Team</li><li>Planned FTE per Team Member: 1 FTE per team member</li><li>Planned Month of Completion: Month 3</li><li>Acceptance Criteria: The SOP is in place, outlining the specific data that needs to be entered into the system, and how it should be entered.</li></ul><p>Milestone 4: Double-Checking of Data</p><ul><li>Responsible Person: Quality Assurance Team</li><li>Required Team Members: Quality Assurance Team, Process Engineers</li><li>Planned FTE per Team Member: 1 FTE per team member</li><li>Planned Month of Completion: Month 4</li><li>Acceptance Criteria: A system is in place for double-checking of data before it is entered into the system.</li></ul><p>Milestone 5: Data Management System</p><ul><li>Responsible Person: Data Analysts</li><li>Required Team Members: Data Analysts, IT Specialists</li><li>Planned FTE per Team Member: 1 FTE per team member</li><li>Planned Month of Completion: Month 5</li><li>Acceptance Criteria: The data management system is in place and allows for easy access and viewing of the data, and can alert if there are any errors or inconsistencies in the data.</li></ul><h3>Conclusions (from the author, not from ChatGPT)</h3><p>Honestly, it was never ever easier to compile a DQM project charter than with the capabilities provided by ChatGPT. From my own experiences, I know that it is challenging to dig as an “IT guy” through the jungle of unit-process-specific phrases, limitations and opportunities. And I can imagine that this issue exists also in other industries and business areas.</p><p>With ChatGPT, you do neither replace the role of the DQM expert nor the role of the project manager, but you give them an effective instrument to identify DQ issues, root causes and solutions fast and reliably, at least to a certain degree. Especially for project managers, it is pretty easy to get examples for business case calculations. If your company is still evaluating whether an investment in DQM software is worthwhile or not, such numbers help to convince those responsible for the budget. What I also like about the result is that ChatGPT considers also non-technical tasks in the project, such as developing SOPs, conducting trainings and optimizing the employee communication.</p><p>However, keep in mind that the project charter and milestone plan created above are just a first sketch and not meant to be copied &amp; pasted 1:1. For instance, I do not agree that the scope of the pilot project is the full production line; this would violate the intention of a pilot. Also, I am skeptical that the first milestone can be achieved within 1 month without knowing what exactly should be trained — so I am missing an DQ deep dive to find the top x critical issues at the lithography process to tackle in the pilot as a first milestone. Furthermore, I doubt that the implementation of a data management system can be achieved within 1 month. You should also consider that there could be single roles that are not yet filled or even developed in your company.</p><p>In a nutshell: there is still a lot to plan, decide and develop for human experts, so don’t worry about A.I. replacing your jobs. But I am confident that we can use the capabilities of language models to reduce some overhead efforts in DQM (and probably also other) projects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c537d1d8994" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Skills That Make a Successful Data Manager]]></title>
            <link>https://medium.com/data-science/5-skills-that-make-a-successful-data-manager-fe09bcc29c2?source=rss-8c9dbf8eb438------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe09bcc29c2</guid>
            <category><![CDATA[skills]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Professor Simon J. Preis, Ph.D.]]></dc:creator>
            <pubDate>Mon, 09 Jan 2023 17:22:50 GMT</pubDate>
            <atom:updated>2023-01-09T17:22:50.676Z</atom:updated>
            <content:encoded><![CDATA[<h4>Survive and Succeed in Data Management</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vGRCadMubqPTNeOA" /><figcaption>Photo by <a href="https://unsplash.com/@roadtripwithraj?utm_source=medium&amp;utm_medium=referral">Road Trip with Raj</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>During my industry career, I have been working in various data-centric roles such as Data Management, Data Engineering and Data Analysis. I was thinking about what went well and what was rather challenging and how people and their skills (also mine!) influenced the success of data projects. Based on these experiences, I have compiled the following list of important skills that make a good data manager! Please note that it is not based on an empirical study and I am not writing as a researcher here — this list simply aggregates some personal observations — so if you see further skills, just add them to the comments :)</p><h3>#1 Modelling</h3><p>“A picture is worth a thousand words”. This is also true in data management. Every data project hits the point where someone in the project needs to draw something that represents either the status quo or that catches the different visions of stakeholders. So, please consider a model as a key technique for efficient communication within a data project! There are really sophisticated modelling tools around that support established modelling languages. And this second part is even more important: please apply established modelling languages! One of the biggest mistakes you can make is to try to put every aspect of the data project into one chaotic PowerPoint diagram. You will end up with a monster that is neither a good data model nor a good process model. Instead, be aware that your data project has dynamic aspects ( → candidates for a process model) and static aspects (candidates for a data model or deployment model) that must surly fit together, but not necessarily within one diagram. There are established languages (e.g. BPMN, UML, Chen) and tools (e.g. visual paradigm,modelio, Enterprise Architect) that support syntax-conform language application or round-trip engineering. A model is something like a construction plan for physical buildings, so make sure that your plan is mature to avoid sloping floors and open walls in your data architecture caused by people in the project who interpret the model in different ways. And in the age of IoT, <a href="https://towardsdatascience.com/fuzzy-data-structures-curse-or-blessing-5ee34ff39cd7">fuzzy data models have implications on the physical world,</a> too. They limit the possibility of comparative data analysis or make it difficult to scale use cases.</p><h3>#2 Connecting People</h3><p>The role of a data manager in an organization is tricky. This person is often neither an IT guy who implements databases on his/her own, nor a business guy who is actually responsible for data or processes (that’s rather a Data Steward’s area of responsibility). So what’s the real value-add of a data manager (or even a data management department)? In my opinion, you need someone who is building bridges between the different data stakeholders on a methodical level. It’s rather easy to find people who consider themselves as experts for a particular business area, data analysis method or IT tool, but it is rather complicated to find one person who is willing to connect all these people and to organize their competencies as it is often required in data projects. So what I am referring to are skills like networking, project management, stakeholder management and change management HIwhich are required to build a data community step-by-step as backbone for Data Governance. Without people, a data manager will fail! So in my opinion, a recruiter who seeks for data managers should not only challenge technical skills but also these people skills. Likewise, if you are interested in a data manager position, judge yourself if you feel comfortable with a job of various origanizational responsibilities beyond technical topics.</p><h3>#3 Domain Expertise</h3><p>I have seen various approaches to build data management organizations. There were closed teams as part of bigger business departments with global accountability for data standards, or federated organizations with a small core team and “dotted-line” data managers in other departments, or data management teams that were part of the IT department. Independent of the organization’s home — my experience shows that a good data manager should have fundamental domain knowledge. If a person acts as data manager for a certain business area, she/he should understand what’s going on in the value chain, what are the critical business issues, which data exists for which purpose and what are the business processes that create and consume data and how they are connected. With this knowledge, a data manager can always think one step further if she/he receives a data change request. Maybe there are other requests in the pipeline which could be merged, maybe you have own ideas regarding the process implementation, maybe you suggest other stakeholders to involve ( → see #2), maybe there are policies (e.g. DSGVO) that must be implemented, maybe you can influence the sustainability of data models beyond this single request, … the list is long! Assuming that a data manager originally has an IT background, the creation of satisfactory domain knowledge could be a challenging task, simply because it’s industry-, company- or even department-specific and you cannot simply listen to an audio book to close that gap. However, keep calm and accept that there will be a learning curve over time, but it is essential that you are willing to learn and adopt that non-IT knowledge.</p><h3>#4 Technology Awareness</h3><p>Refining a data change request from what a business stakeholder expects to what should be really implemented is a process. In my “home” industry, the semiconductors, also business experts often have an advanced IT knowledge — this is curse and blessing at the same time :) On the one hand, you can dive deeper into technical topics with all the stakeholders, on the other hand, they tend to come up with IT solution proposals instead of problem descriptions, which is not so welcome among the IT experts, because it somehow skips their responsibility. Besides the injured sense of honor, there are good reasons why building an implementation concept is separated from the requirements analysis. One of them is that there is not “the one” solution but various options with pros and cons. As important as the domain expertise is the awareness of up-to-date data management technologies and their capabilities. For instance, not every piece of master data must be entered manually, even if the data steward expects that. Maybe there are attributes from a record that can be deduced by a business rule system or filled via data integration from other systems. Or, a problem could be solved by changing the task order in a business process, which could be supported and forced by workflows. Also data catalogs have some powerful features that help data managers in their overarching role — <a href="https://medium.com/towards-data-science/datahub-hands-on-part-i-f0709e7efec9">please view my series on a particular tool to get more information</a>. Especially if data managers have a non-IT background, this skill can be challenging. Fortunately, the technical knowledge is often not company-specific, hence, the gap can be closed with official material (videos, MOOCs, books, articles, …). From a didactic perspective, I would recommend to define a learning path for junior data managers that provides overview and insight into selected technologies.</p><h3>#5 Lean Thinking</h3><p>Last but not least, I made good experiences with applying the lean philosophy. “Lean” is typically associated to the manufacturing business, but can also be applied for data management. For instance, you can use “Poka Yoke” principles to design data processes or GUIs to avoid certain data quality issues by design. Or, you could apply “5S” to clean-up your data architecture, where data valuation helps to judge which data objects in your organization are “waste” and which are business-critical — <a href="https://dl.acm.org/doi/abs/10.1145/3537693.3537725">we have published a research paper that proposes such an approach</a>. Especially in larger data projects, e.g. when you want to integrate a whole domain into a master data management system, it’s worth to organize “Kaizen” events, where all the stakeholders work together in a workshop mode to shape the future (conceptual) data landscape. A great method is also “PDCA” which helps to improve data quality iteratively — and there are still more lean methods that I have already applied in data projects and that are useful! In my opinion, the lean approach is the key to gain sustainable data architectures. This sustainability effect can be observed as well in traditional applications of lean management as discussed in <a href="https://link.springer.com/chapter/10.1007/978-3-658-21686-3_3#Sec4">this (German) article</a>. The generic types of waste in a value chain can also be found in data architectures, such as motion (e.g. unnecessary manual efforts for data maintenance) and defects (e.g. wrong or missing data that limits the effectiveness of a data product). All in all, a probably underrated skill, but a great topic for further research!</p><h3>Conclusion</h3><p>With this story, I wanted to share my view on the optimum skill set of a data manager. A good data manager is way more than a technical expert for data management software. Typically, data management initiatives succeed (or fail) due to the (missing) support of people in an organization. Hence, there are a number of non-technical skills such as stakeholder management or project management that should be learned and applied especially in more complex data projects. Nonetheless, IT knowledge is also important especially to resolve the waste in data architectures that could be found via the lean methods. Domain knowledge is something that you won’t learn at university but over time by actually working in a certain business, but you should be open-minded for that knowledge as well and you can actively influence the slope of your learning curve e.g. by interviewing experts! And don’t forget to apply established modelling techniques in your data projects! The more people in your organization get used to good models, the more efficient will be the future communication in data projects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe09bcc29c2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/5-skills-that-make-a-successful-data-manager-fe09bcc29c2">5 Skills That Make a Successful Data Manager</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataHub Hands-On Part II]]></title>
            <link>https://medium.com/data-science/datahub-hands-on-part-ii-169c36ee082d?source=rss-8c9dbf8eb438------2</link>
            <guid isPermaLink="false">https://medium.com/p/169c36ee082d</guid>
            <category><![CDATA[data-governance-software]]></category>
            <category><![CDATA[data-ingestion]]></category>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[data-discovery]]></category>
            <category><![CDATA[datahub]]></category>
            <dc:creator><![CDATA[Professor Simon J. Preis, Ph.D.]]></dc:creator>
            <pubDate>Tue, 03 Jan 2023 16:45:30 GMT</pubDate>
            <atom:updated>2023-01-03T16:45:30.997Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Ingestion, Data Discovery and Entity Linkage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W8iKMGUXXgRm4Ysn" /><figcaption>Photo by <a href="https://unsplash.com/@miinyuii?utm_source=medium&amp;utm_medium=referral">Duy Pham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Welcome to part II of my DataHub Hands-On story! In the <a href="https://medium.com/towards-data-science/datahub-hands-on-part-i-f0709e7efec9">previous part,</a> we have discussed how to setup a data catalog from scratch, how to do semantic modelling with business terms and what’s the motivation for Data Governance (DG) at all. In this part, we will have a closer look on the rather technical parts of DataHub in order to ingest and link metadata.</p><h3>Data Ingestion</h3><p>The data ingestion (DI) section of DataHub can be accessed via the top right menu bar:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xuuUuNsv1sF94yqxaEfKew.png" /><figcaption>Source: Author</figcaption></figure><p>Once we have entered the DI area, we see a list of already configured data sources including an overview that shows execution statistics. From there we can now run a configured DI process, we can edit the configuration, we can delete or copy a configuration, or we create a new source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_4dVlNPp0EzEDnfHdQ0fQ.png" /><figcaption>Source: Author</figcaption></figure><p>There is also another button “Secrets” that can be used to configure passwords for accessing data sources. This feature helps to store the passwords secured in the DataHub database. A secret can be associated to a data source during the source configuration.</p><h4>Important Terms</h4><p>To understand how the DI configuration works, let’s quickly define a few important terms. As briefly discussed above, there are <em>sources </em>for metadata. Sources refer to the databases — typically from a productive environment — that we want to manage in our data catalog. Besides sources, DataHub also provides the concept of <em>sinks</em>. Sinks are destinations for metadata. Typically and by default, the DI engine sends the collected metadata to the “datahub-rest” sink to make them available in the DataHub UI. However, there are also options for Kafka and File export, if required. Then we have <em>recipes</em>, which are configuration files that tell the ingestion scripts where to pull the data from and where to put it. Recipes can be configured through the UI, if an adapter is available, or via a YAML script[1].</p><h4>Data Sources and Adapters</h4><p>Now, Let’s have a a look on the pre-configured database adapters (a.k.a connectors) that DataHub provides out of the box:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ZIHUokbA1CDCxEkmRNm4g.png" /><figcaption>Source: Author</figcaption></figure><p>We see a number of established SQL and NoSQL technology icons. To understand that this is not just for marketing purposes, let’s compare two selected adapters. On the left side of the following image we see a ingestion recipe for BigQuery, on the right side we see a recipe for SQLServer. The ingestion UI obviously expects different information to establish a database connection depending on the technology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XIbWL65sHxXU_fJr7ND3mg.png" /><figcaption>Source: Author</figcaption></figure><p>Most of the data sources follow a pull-based integration using a Python based metadata-ingestion system. In a first step, the metadata is pulled from the source systems, and in a second step, the information is pushed to the DataHub service tier either via HTTP or Kafka². Each source adapter has different capabilities, hence, reading the online documentation is recommended to avoid confusions. For instance, the <a href="https://datahubproject.io/docs/generated/ingestion/sources/mysql">MySQL </a>and SAP HANA adapters enable data profiling and detection of deleted entities, which is not supported by the MongoDB or Oracle adapters. In contrast, the <a href="https://datahubproject.io/docs/generated/ingestion/sources/bigquery">BigQuery </a>adapter supports these features as well as table-level lineage and a few more. To understand why there are differences between the adapter capabilities, we just need to visit github to view the code for each adapter. Reviewing the entire adapter logic is beyond the scope of this story, but we can see, for instance, that sqlalchemy is used to connect to <a href="https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/mysql.py">MySQL </a>sources whereas the <a href="https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery.py">BigQuery </a>adapter is applying the standard connection package provided by Google. So let’s conclude that each adapter is implemented independently and potentially uses different connection packages. This and the fact that also database technologies differ in metadata that they can provide at all (e.g. there is no foreign key concept in MongoDB) have implications on the ingestion capabilities of an adapter.</p><h4>Finalizing the Recipe</h4><p>There is also a filter section that helps to narrow down the schema or tables to ingest. If no filter is set, the DI engine executes the metadata scanning over all schemes, tables and views where the configured database user has access — this should be considered to avoid performance impacts on both the scanned database and the ingestion process. From security perspective, it is rather recommended to restrict the user privileges directly at the database (e.g. by creating a particular read-only DataHub user). You can toggle advanced settings in an additional section, e.g. if views should also be considered for scanning or only tables or vice versa, or both. Also important: you can select if tables or columns should be profiled (e.g. how many records in a table, how many missing values in a column). This is useful to gain transparency over the company data landscape.</p><p>Once the recipe is configured, you can specify the execution schedule. Here, you define the frequency (e.g. daily, monthly, hourly) of the DI execution. In the final step of the creation wizard, you have to assign a unique name to the new source and click “save &amp; run” which directly triggers the first metadata scanning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N48ux2W5B19jhfsjSQwa6g.png" /><figcaption>Source: Author</figcaption></figure><p>Once the process has been executed, you can view the details in order to get a first impression of the ingested assets, or to see details on the failures. In my test case, we have a MySQL database server with 78 views and 38 tables from 4 databases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nanUzWshWl-cf4E4VTrj8g.png" /><figcaption>Source: Author</figcaption></figure><h3>Data Discovery</h3><h4>Searching and filtering</h4><p>Once our metadata was pushed to the DataHub service tier, we can start our data discovery journey through the DataHub UI. As a fan of the 90s, I use the “northwind” database from Microsoft in my testcase. If I want to search for the schema, I can scroll through the tables and views or I can select some filters (e.g. to exclude views).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*WUq_EA7nuKLaBF7cNd3Ddg.png" /><figcaption>Source: Author</figcaption></figure><h4>Table Profiling Results</h4><p>Let’s assume I want to have a closer look on the product data, then I simply click on the the product dataset from that list. The first information I see is a table schema overview that shows the fields, their data type and even some data model details such as primary and foreign keys. At the top (under the dataset name), we also see at one glance that there are currently 79 records in our table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*TN9-1S8-7AbsYQY5LfBOFg.png" /><figcaption>Source: Author</figcaption></figure><p>The second sheet refers to documentation which is empty by default. Here, data custodians could add technical knowledge on that table in DataHub and/or add links to already existing material. It is an important tool to foster the data architecture maturity in a company — the data catalog is only as helpful as the knowledge shared and entered by experts. The dataset has a few more interesting sheets such as lineage and validation, which we will cover in part III — stay tuned :)</p><h4>Column Profiling Results</h4><p>Now, I want to show what the DI engine provides by simply selecting the “column profiling” option in our recipe earlier. For that purpose, let’s enter the sheet called “stats”. Here we see statistical information for each column in the selected table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ytazONZKY5MtQDgvVTMHNQ.png" /><figcaption>Source: Author</figcaption></figure><p>For instance, we see in the first row that the field “ProductID” has 100% distinct values. This finding is not surprising, because this field is the primary key of the table — it is technically defined that the values have to differ per record. We also see a NULL count and NULL percentage information, e.g. we see that two product records do not have a supplier reference. This can be considered as a critical data quality issue, because without an associated supplier, you cannot physically provide those products to customers.</p><p>What else do we have? We see descriptive measures such as min, max, mean, which are more interesting in transactional tables with higher amount of records. At least, we see at one glance some statistical measures regarding unit price of our product portfolio (e.g. half of our products cost less than $ 19.45 → Median). In addition, we see sample values for each column in our dataset.</p><h3>Bringing together what belongs together</h3><h4>Entity linkage</h4><p>Sure, the data discovery was already insightful and fosters transparency, especially if you never experienced such a feature in practice. However, the full power of a data catalog is only unleashed when we bring the IT world and business world together. Sounds great — but what does that mean? Well, the “IT world” in this sense refers to all the technical information around the datasets. The “business world” is what we have discussed in the previous part: all the business information we maintain in the glossary. Looking at real companies, it must be emphasized that (especially in bigger companies) there are different groups of people responsible for datasets and glossary. We have data custodians on the one hand who are experts on technical topics, e.g. to develop and improve databases. And we have data stewards on the other hand, who are experts on business topics, e.g. how do we indicate which product can be manufactured at which site due to which conditions — and which data is required to decide. So when we talk about linking entities, we implicitly mean connecting people!</p><p>Big Words! So how does that work with DataHub? Let’s continue with our dataset “products” which reflects a physical database table. This is our starting point in DataHub. In the screenshot below we see various marked information, which refer to multiple types of links to the “business world” that can be used jointly.</p><ol><li>Option: We can add a glossary term to a table column. This link indicates that raw data of a certain term can be found in this column, e.g. the actual product names.</li><li>Option: We can add a glossary term to the whole table. This link indicates that a certain term is rather complex and described by multiple attributes that can be found in that table.</li><li>Option: We can add a domain where the dataset belongs to. This link is similar to the linkage of glossary terms to a domain (see part I for more information).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f0bHxLdqtBIzvn8uckcGBA.png" /><figcaption>Source: Author</figcaption></figure><h4>People</h4><p>However, we still don’t have people in our model. So let’s add an owner for the dataset “products”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*14AkL0mrUbYK36k5p8lh3A.png" /><figcaption>Source: Author</figcaption></figure><p>DataHub suggests that this person should be a “technical owner” which is similar to the role of a “data custodian” in other DG frameworks. Now we need to add people to the associated glossary terms. We can simply click on the “Product” term and DataHub navigates us to the glossary record. There, we also can add owners. For instance, we can add Brian as “data steward” and Lilly as a “business owner”. If we hover over the names, we see the role assigned to the name. DataHub allows to add persons multiple times — this can be useful if a person has multiple roles (e.g. business owner + data steward). Unfortunately, the tool also allows to add the same combination of person and roles multiple times, which is not useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dL5dGH-H6wHZboBErCN6uw.png" /><figcaption>Source: Author</figcaption></figure><p>Now let’s assume that Brian is new at the company and wants to change something on the product master data table, so he needs to find the right IT expert to clarify the implementation details. In this case, he simply clicks on the “products” dataset which is visible in the “Related Entities” section (see screenshot above). Then he is forwarded to the dataset (which we have seen further above) and can see: my IT counterpart is John! And of course, also other people in the organization who need support or have change requests for product data can use this linked information — they can simply search for “product” and need only 2 clicks to collect the right data stakeholders (Lilly, Brian and John).</p><h3>Part II Conclusions</h3><p>We have seen that setting up a data ingestion process is done within a few minutes as DataHub provides a configuration wizard for a number of established database technologies. We have also seen that data discovery is a powerful feature to gain transparency over a company’s data architecture. The profiling of tables and columns is performed automatically and the statistical results can be viewed in the UI. Linking datasets and glossary terms or adding owners is also very simple, however, we have seen that consistency checks are not always performed by DataHub. We can add the same owners with the same role multiple times, but the tool prevents us at least from adding the same glossary term multiple times to a dataset. It is also possible to assign the same glossary term to multiple columns in the same table — maybe there are scenarios in practice where this flexibility is useful. In general, my practice advice is that — at least within one table — there should be a 1:1 relationship between columns and terms. If not, you should review the level of granularity in your glossary — or you have a problem in your physical data model.</p><p>Again — well done if you made it this far! In the next and final part, we will dive deeper into data quality management with data lineage and data validation. Maybe, I will write a further story to discuss the DG role concept of DataHub in comparison to other DG frameworks.</p><p>[1] <a href="https://datahubproject.io/docs/metadata-ingestion">https://datahubproject.io/docs/metadata-ingestion</a></p><p>[2] <a href="https://datahubproject.io/docs/architecture/metadata-ingestion">https://datahubproject.io/docs/architecture/metadata-ingestion</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=169c36ee082d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/datahub-hands-on-part-ii-169c36ee082d">DataHub Hands-On Part II</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataHub Hands-On Part I]]></title>
            <link>https://medium.com/data-science/datahub-hands-on-part-i-f0709e7efec9?source=rss-8c9dbf8eb438------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0709e7efec9</guid>
            <dc:creator><![CDATA[Professor Simon J. Preis, Ph.D.]]></dc:creator>
            <pubDate>Thu, 15 Dec 2022 16:53:42 GMT</pubDate>
            <atom:updated>2022-12-15T16:53:42.675Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kzsBXHiDbaqAf9BS" /><figcaption>Photo by <a href="https://unsplash.com/@swimstaralex?utm_source=medium&amp;utm_medium=referral">Alexander Sinn</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Introduction</strong></h3><p>In preparation for a new lecture, I was searching for an up-to-date open-source data cataloging tool to enable my students practicing the Data Governance (DG) theory. Among others, I came across LinkedIn’s “DataHub”. Besides the rich set of features, such as <a href="https://github.com/opendatadiscovery/awesome-data-catalogs">data discovery, data lineage and data quality</a>, it convinced with a very simple setup routine using <a href="https://atlan.com/linkedin-datahub-tutorial/">docker</a>. Since I wanted to document my test cases, configurations and findings anyway, I decided to share them as well. I am planning to release 3 story parts which will cover:</p><ol><li>Motivation for Data Governance, Background on DataHub, DataHub setup and basic glossary design</li><li>Data ingestion, linking data entities and data discovery</li><li>Users and roles, data validation, data lineage and summary</li></ol><p>So, let’s get started with part 1!</p><h3>Data Governance</h3><p>The chicken and egg problem I’ve seen in the industry is that the problems that DG tackles can’t be easily solved with IT tools, but DG isn’t at all effective without powerful IT tools either. So where to start? A company needs to understand (or better: feel) the pain of bad data across the organization (e.g. bad data slows down business processes), otherwise there is no motivation and support for DG. Then you need to convert that pain into quantitative measures, more specifically Data Quality (DQ) metrics. And once you know your enemy, you can set a goal and devise a plan to achieve that goal. For example, the completeness of master data for a specific column in a table is easy to measure and can be a starting point. Let’s say you have 60% completeness now and want to reach 90% within a year, then you need to identify and fix the root cause(s) of missing data (e.g. GUI mandatory). And of course you need to track the progress e.g. to check whether the completeness is noticeably increased by this fix and really reaches 90% after one year — because if not, someone has to raise their hand and initiate a “plan b”.</p><h4>The reality shock…</h4><p>Sounds pretty simple… but… the main problem is that you need someone who is able to see and quantify the bad data in the first place, someone who is responsible for driving this improvement, you need experts to fill in missing data maintain (or someone who has a good algorithm), you need someone who is technically able to fix something in an IT system, you need someone from the management team who supports necessary changes in business processes and someone who can implement the DQ keeps an eye on it regularly. And of course all these people need the time and need to prioritize other tasks down. And that’s the hard part of DG — identifying, motivating and organizing the right people to work in a structured way on improving the data landscape, because in reality it’s not just a small isolated problem, but — depending on company size and business complexity — a colorful bouquet of mutually influencing problems.</p><p>Unfortunately, a DG tool is not a magic wand that makes all data problems and organizational gaps disappear after installation. But at least it helps to measure and visualize DQ, speed up root cause (and impact) analysis, and establish the connection between business conditions and IT entities. And these capabilities impact the throughput rate of DG activities in an organization that still require human effort — but maybe only a small percentage compared to DG without any tool support.</p><p>And that’s what I want to explore in this article: how does DataHub fit into those expectations?</p><h3>DataHub</h3><p>The overall architecture of DataHub is depicted in following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*boolBID1veJOFFKJ.png" /><figcaption>Source: Datahubproject.io</figcaption></figure><p>We can see that on the left there are a number of connectors to different database technologies, covering both SQL and NoSQL. With these connectors it is easy to set up a metadata ingestion process as the required configuration is guided by a wizard. If the required connector is not available, it is still possible to configure the recording manually. The metadata is forwarded to the so-called serving tier, which consists of an internal data store (e.g. MySQL), a search engine (Elastic), Graph Index (neo4j) and a View More that can be queried via REST and GraphQL.</p><p>The end user works with the DataHub frontend, e.g. to search for entities, navigate the catalog, and maintain glossary terms. API and stream integrations are other features that I won’t cover in my test cases.</p><h3>DataHub Setup</h3><p>In general, you have the choice between a Managed DataHub (SaaS) or a self-hosted DataHub. I chose the second option, which in short means deploying a DataHub instance in a Docker environment. I won’t go into all the details of each step as they are well documented on the Datahub product page. The basic requirements are Docker Desktop, jq for JSON processing on the command line, Python 3.7 (minimum) and sufficient hardware (2 CPUs, 8 GB RAM, 2 GB swap space and 10 GB disk space). Once everything is ready (and all paths are set correctly :) ), you can easily install and deploy DataHub using CMD commands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GfPH1HsO8UULiiBq8wSgRg.png" /><figcaption>Source: Author</figcaption></figure><p>The user front end can then be accessed via the browser. A default user named “DataHub” is configured to be used for initial access. When it works, you’ll see a Google-like start-up screen with a central search box — however, there’s nothing to search for yet as the catalog of data is empty. Since I’ve already done some data ingestion and glossary maintenance, my DataHub instance is already showing additional information about available domains, datasets, and platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jAAis9-UYuJOid-xoyxKkQ.png" /><figcaption>Source: Author</figcaption></figure><h3>Glossary Design</h3><h4>Metadata Concepts</h4><p>On the top right corner, we see a menu bar with a few options. In order to add business knowledge to the data catalog, we are interested in the “Govern” option. Here we see following sub-options:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*zBok1tpMzXsZEwLUaFCDVA.png" /><figcaption>Source: Author</figcaption></figure><p>An appropriate catalog design requires the right understanding of the available metadata concepts:</p><ul><li><strong>Domain</strong>: a broader business area where we want to organize related data assets (e.g. all data assets which belong to Sales organization)</li><li><strong>Term group</strong>: Similar to a folder of an operating system that may contain terms or even other term groups.</li><li><strong>Terms</strong>: words or phrases with a specific business definition assigned to them.</li><li><strong>Tags</strong>: informal, loosely controlled labels that help in search &amp; discovery.</li></ul><p>There are other DG frameworks and data catalogs on the market with slightly different or additional metadata concepts, e.g. data objects, business processes and policies. Depending on the flexibility and scope a company expects from a DG tool, this can be considered as a weakness of DataHub. On the other hand, the various information needs to be gathered from somewhere and needs to be maintained, which means considerable effort — so the metadata concept set provided by DataHub is a good starting point and still allows building comprehensive data models.</p><h4>Metadata Maintenance</h4><p>The question now is how to use these concepts appropriately. There should be guidelines developed by the central DG office so that each business area (aka domain) uses the concepts in the same way. For example, you could say that “term groups” should be treated like business processes. Let’s practice this example: We want to have a business process (term group) “Equipment Procurement” that contains all terms related to the procurement of an equipment (e.g. equipment name, equipment manufacturer, equipment itself). These terms belong to the “Manufacturing” domain. This can be done in DataHub as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*zmHkS2rh5bprJOzIIPrvbg.png" /><figcaption>Source: Author</figcaption></figure><p>Then we create our term group, one of its terms and we associate the term to the according domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1OmJB2Q334vlKRZ8Qh0IEA.png" /><figcaption>Source: Author</figcaption></figure><p>Once our term is created, we can set an official definition and add also further information as links:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*nv3jBPtIt-kIxJklainVzQ.png" /><figcaption>Source: Author</figcaption></figure><h4>Semantic modelling</h4><p>That’s cool — but we can do even more semantic data modelling in our glossary! Let’s say we want to specify terms that are detailed information of an equipment, like a name or a maker. Then we can build such a hierarchy as well in DataHub using the “contains” relationship, which is similar to an object/attribute relationship:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1pdn34qJccjZyGUnjJyudA.png" /><figcaption>Source: Author</figcaption></figure><p>In addition, we can use inheritance in our glossary. For instance, we want to differentiate between equipment types like “measurement equipment” and “logistic equipment” that are not different from “equipment” but only more specific types of “equipment”. In this case, we can connect these terms via the “inherits” feature:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YyzCjpTpLxztfwW3HFblow.png" /><figcaption>Source: Author</figcaption></figure><h4>Conclusion Part 1</h4><p>Although the semantic modeling offers some cool features, my impression is that there is not always a technical logic behind it in the DataHub. For example, the inheriting terms don’t actually inherit metadata from the “parent term” (e.g., descriptions, ownership, or links) and the related “subterms” (via “contains”) don’t really know about this relationship because it doesn’t appear in their dataset. The structural information is unidirectional only. What I’m really missing is an inheritance of “contains” terms (like attributes in object-oriented programming), since terms that are part of a generic term should also be part of the associated “child” term, otherwise you have to maintain this information redundantly or you end up with inconsistent metadata. Another design option could be to avoid “contains” relationships in inherited terms, since the “parent term” can be linked and clicked — so that one can easily find the relevant “contains” relationships in the “parent” dataset. Also missing, I think, is a two-way relationship view, perhaps in a visual form as well, so that a data steward can see the big picture at a glance. Again, what is good about the glossary is the possibility to share entries, e.g. by mail or simply direct link, which supports the communication process between data stakeholders or can be integrated into other IT tools.</p><p>Well done if you made it this far :) In the next part, we will dive into the data ingestion and discuss how to connect the glossary (business world) to datasets (IT world).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0709e7efec9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/datahub-hands-on-part-i-f0709e7efec9">DataHub Hands-On Part I</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Decrease the Carbon Footprint of Digital Communication]]></title>
            <link>https://medium.com/data-science/how-to-decrease-the-carbon-footprint-of-digital-communication-f3186673818c?source=rss-8c9dbf8eb438------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3186673818c</guid>
            <category><![CDATA[email]]></category>
            <category><![CDATA[system-dynamics]]></category>
            <category><![CDATA[digital-communication]]></category>
            <category><![CDATA[climate-change]]></category>
            <category><![CDATA[carbon-footprint]]></category>
            <dc:creator><![CDATA[Professor Simon J. Preis, Ph.D.]]></dc:creator>
            <pubDate>Mon, 06 Dec 2021 20:52:48 GMT</pubDate>
            <atom:updated>2021-12-06T20:52:48.677Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Frr3V2cRZeFoMA6W" /><figcaption>Photo by <a href="https://unsplash.com/@mikaela_wiedenhoff?utm_source=medium&amp;utm_medium=referral">Mikaela Wiedenhoff</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>An assessment of influences of email behaviours on greenhouse emissions using System Dynamics</h4><h3>Introduction</h3><p>Nowadays, it is quite normal to communicate via digital tools. We have social media platforms to chat with friends, videotelephony services to conduct job interviews and, of course, the good old email. One would think that sending information via email is the “green alternative” to using a physical letter. Sure, we think about CO2 emission during the transport from A to B, and we also know the negative environmental impacts of cutting trees required for paper production. So, is an email the holy grail in terms of climate-neutral information sharing? This article will spot on this question by reviewing the carbon footprint of emails, by discussing the importance of email data structures, and by performing experiments on how email behaviours affect the CO2 emission. The novelty of this article is that we will quantify the impacts of possible levers of email behaviours on the carbon footprint. For that purpose, we will apply the simulation technique System Dynamics.</p><h3>Carbon Footprint of Emails</h3><h4>Background on Emails</h4><p>First of all, let’s briefly recover what an email actually is and where it came from. Basically, an email is an internet service to transfer text and other digital data from a sender to a recipient. An email application consists of a mailing server, which is responsible for directing the delivery and archiving of messages, and a client-oriented mailing program for creation, sending and reading messages. In addition, an email application requires network access and uses transfer protocols such as SMTP, IMAP or POP3<a href="#_ftn1">[1]</a>. The very first email was sent in 1971 — almost 20 years before the world wide web was born. The inventor of this new messaging technology was Ray Tomlinson who worked on a predecessor of the internet called “Arpanet” at that time. Prior to that, it was only possible to read digital messages on the same device where the message was originally sent. So the real achievement was the device-overarching messaging via network by using the FTP protocol. Furthermore, he introduced the “@” sign for addressing the message recipients<a href="#_ftn2">[2]</a>.</p><h4>Greenhouse Emissions caused by Emails</h4><p>There is already a number of articles employed with the CO2 emission of emails, which often refer to the same source: ‘<em>‘How Bad are Bananas?: The Carbon Footprint of Everything”, </em>published in 2010 by Mike Berners-Lee, researcher on carbon footprinting and brother of Tim Berners-Lee, inventor of the world wide web. The study built categories for emails and stated the average emission per message as follows:</p><p>1) Spam email: 0.3g CO2</p><p>2) Regular email: 4g CO2</p><p>3) Email with attachment: 50g CO2</p><p>These CO2 costs are mainly caused by the energy that is required to run a device that works with emails and that powers the network infrastructures, which must be accessed in order to transfer emails, as well as the data centers, which store the contents <a href="#_ftn3">[3]</a>. Based on the figures from Berners-Lee, we can also imply that the size of an email influences the CO2 emission: the bigger the email, the more CO2 is emitted. In 2021, we have a number of ~320 billion emails sent and received daily all over the world, which is forecasted to further increase<a href="#_ftn4">[4]</a>. Though spam mails appear to be less climate-critical, we need to consider the pure amount of spam mails compared to normal mails. Let’s focus on a few countries to reveal the proportions of <strong>daily</strong> sent emails<a href="#_ftn5">[5]</a><a href="#_ftn6">[6]</a>:</p><p>1) United States: 9.6 billion emails (total), 8.6 billion emails (spam).</p><p>2) Germany: 9 billion emails (total), 7.9 billion emails (spam)</p><p>3) China: 8.7 billion emails (total), 8.5 billion emails (spam)</p><p>4) United Kingdom: 8.6 billion emails (total), 7.7 billion emails (spam)</p><p>5) Russian Federation: 8.4 billion emails (total), 8.1 billion emails (spam)</p><p>As these countries are, among a few others, the leaders in the worldwide email statics, we can calculate the weighted average to state the spam percentage of the global email traffic: 92%. A further study came to the result that around 24% of business emails have an attachment<a href="#_ftn7">[7]</a>.</p><p>Having these figures, let’s extrapolate the yearly CO2 emission of emails worldwide (we accept rounding issues):</p><ul><li>Yearly amount of emails sent = 320 billion x 365 days = 117.8 trillion</li><li>Yearly amount of spam emails sent = 117.8 trillion * 0.92 = 108.4 trillion</li><li>Yearly amount of regular emails sent = 117.8 trillion * 0.08 * 0.76 = 7.2 trillion</li><li>Yearly amount of emails with attachment sent = 117.8 trillion * 0.08 * 0.24 = 2.3 trillion</li></ul><p>Using these results, we can now calculate the yearly CO2 emissions per email type, which is shown in the following table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*-6tbWA7v6Vow07f0zBQdiw.png" /><figcaption>Number of emails and CO2 emission per email type (Image by Author)</figcaption></figure><p>This leads to an overall CO2 emission of around 176.3 million tonnes. Quite a huge number — to make it more tangible, we can translate it into 21.2 million homes energy consumption for one year or even 21 trillion smartphones charged; or to make it even more concrete: it is equivalent to 442.3 billion miles driven by cars<a href="#_ftn8">[8]</a>, which is even more than the yearly mileage of all cars admitted in Germany (376.7 billion miles<a href="#_ftn9">[9]</a>). However, if we compare it with the global CO2 emission of fossil fuels, which was nearly 35 billion tonnes in 2020<a href="#_ftn10">[10]</a>, our emails make only 0.5% compared to that amount of greenhouse emission. Nonetheless, we can clearly state that emails are way not that climate-neutral as many might have expected.</p><h3>Drivers of Big Email Sizes</h3><p>Before we assess the levers to reduce the CO2 emissions, we want to understand the data structure of emails and which types of information it typically contains. Basically, an email consists of three parts: an envelope, a header and a body. The envelope encapsulates the message and contains all required information for sending an email. Typical information are the destination address, the email priority and its security level. The header consists of fields that are related to the message transport such as “To”, “Cc” or “From”. Each header field refers to a single line of ASCII text that specifies field name, colon and value. The body part of the message contains the actual content which is interesting for the sender. It typically consists of text, signature and attachments<a href="#_ftn1">[11]</a>. Considering these elements, we can imply that the body is the main driver for the size of an email. So let’s have a closer look on the body.</p><h4>Data Formats of Emails</h4><p>It is not only the pure amount of core information which affects the body size. In a professional mailing client application, you can select the format of the body, which impacts the size as well, which is shown in the figure below (a screenshot from MS Outlook).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*exII_Omv1ahN1fTJWn-dMQ.png" /><figcaption>Format options in MS Outlook (Image by Author)</figcaption></figure><p>To assess the impacts of data formats on the email size, I have created a sample mail with salutation, greeting, some “lorem ipsum” content and my current company signature. Then, I changed some word formats, e.g. bold and coloured. Finally, I saved the mail as a file to my computer, but not only once but multiple times with different formats. Firstly, I used the “HTML” format, which requires 39 KB. Then, I used the “Rich Text” format, which requires 66.5 KB. And finally, I used the “Plain Text” format, which requires 19.5 KB. So, we can conclude that we have three different message sizes, though the body content is exactly the same (except the format, which is lost with plain text). In addition, I can reduce the email size further if I remove my signature. Sure, we are talking about kilobytes in the age of Big Data. But when we put these results together, we can see that the percentage differences between the formats (and signature uses) are quite significant (see table below). Therefore, even if we would not reduce the number of emails, we could influence the global CO2 emissions by reducing the network traffic and storage consumption but keep sharing the same information. Of course, this argumentation builds on the assumption that the maturity of emails is HTML-formatted, which I observed especially in a business context but also in the private area when receiving newsletters or advertisements. Unfortunately, I could not find a reliable study that quantifies the percentage of used email formats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*owPSXSJmJh5_1e7B4g8pDg.png" /><figcaption>Influences of email format on email size (Image by Author)</figcaption></figure><h4>Email Attachments</h4><p>Another driver of big email sizes is the use of attachments. In earlier times, it was quite useful to share a document with the recipient by simply sending a copy of the file via email. On the one hand, it was an easy way to give access to the file to people from outside the own company. On the other hand, it was also more convenient to simply drag and drop a copy than sending a link to the internal file share and verifying that the person has access (which often required the help of an IT expert who administered the file share). Nowadays, it is much easier to share centrally stored files even with external parties. Professional office software suites that are mainly cloud-based, for instance Microsoft 365<a href="#_ftn2">[12]</a> and Google Workspace<a href="#_ftn3">[13]</a>, provide features to manage file accesses easily on your own, even granting access to external partners. With these capabilities, it is actually no longer required to send a copy of the file as an email attachment. In addition, there are also some drawbacks, if one sends a copy instead of the link to the original file. For instance, if you find a typo or bigger issue in your document right after sending the email, you have still the chance to update the original document and the recipient will access the updated original file via the link. However, if you sent the copy of the file, you would have to send the corrected file again and spot a light on your mistake. Another benefit of sharing the link instead of the copy is the ability of working collaboratively on the same file. In addition, the document management system in the background typically cares about versioning. Thus, if any of the contributors made a mistake, you can simply recover the previous version. This way of collaboration eliminates also the need of sending and storing multiple versions of the same file like File_V1, File_V1.9, File_final_REALLY_2 and so on …</p><h3>Impacts of Email Behaviours on Carbon Footprint</h3><p>We have reviewed the CO2 emissions of emails and discussed drivers that cause big email sizes. In this section, we want to assess what would happen to the CO2 emissions if we would change something on these drivers. For that purpose, we will apply System Dynamics (SD) as simulation technique using AnyLogic as modelling software. Main rational for that selection is that SD is quite powerful when working with continuous effect flows on a bigger scale where single-agent activities are not relevant. The mathematical foundation of SD is the calculus.</p><h4>Description of Experiments</h4><p>What we have not clarified so far is the CO2 emission of an email per KB, because the study from Berners-Lee only differentiated the emissions by types of emails qualitatively. An article on CO2 emissions when surfing the internet estimates that an email with the size of 1 MB emits 20g of CO2 during its whole life cycle<a href="#_ftn1">[14]</a>. The unnamed author pointed out that 20 emails of that size over a year per user create the same CO2 emission as a car that travels 1000km. If I calibrate that value to my previous example HTML email, it would emit 0.6g CO2. A “regular” email based on Berners-Lee that emits 4g CO2 would have a size of 200 KB. Furthermore, we can imply that an average email with attachment has a size of 2.5 MB in order to emit 50g CO2. These numbers do not influence the format-related size adoptions, but the signature-related one. Hence, if we accept 200 KB to be the standard email size, then the removal of the signature would only have an impact of -6.25%. on the size. So, we will work with this data in the simulation.</p><p>Let’s start with the basic rates based on the daily sent mails, which are summarized in the following table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*Gu1Kuhjqj0ie2BOdlmyKFg.png" /><figcaption>Email rates daily (Image by Author)</figcaption></figure><p>The following figure presents the SD model which I have developed to perform the experiments. It differentiates the email types (spam, regular, attachment) and additionally the formats of regular emails. The stock variables count the number of emails over time, whereas the yellow variables calculate the CO2 emission per email (sub-)type. The red variable sums up the single emissions to return the overall CO2 emission of emails over a year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*Q_p0xwBczfRICHuI_Jzelw.png" /><figcaption>System Dynamics Model (Image by Author)</figcaption></figure><p>Next, we want to perform experiments that are employed with following questions:</p><p>1) What if we more often remove our signatures from HTML emails?</p><p>2) What if we send less HTML emails with signature but more plain text emails?</p><p>3) What if we send more links to files instead of attachments in emails?</p><p>4) What if we decrease the number of recipients of emails we send?</p><p>Unfortunately, I could not find a study that states the percentage of emails with multiple recipients. Hence, I need to work with some assumptions. Let’s assume that especially in the business context we often send mails to multiple recipients. At least I experienced that people often set their managers or other interested people in CC, without expecting any action from them. So, such an email is a kind of “zero information” to them. But also the “To” recipients are often not selected meaningfully. Instead, existing mailing lists are reused to spread information that affects only parts of the addressed persons. So what I want to construct is the following situation:</p><ul><li>There is a percentage of the global email traffic that is caused by multiplying the same email to more than one recipient.</li><li>There is a potential to decrease that percentage by limiting the email recipients.</li></ul><p>For my experiment, I assume that 30% of the email traffic is caused by this effect. This percentage is applied to regular emails as well as to emails with attachment. Then, I will investigate the reduction of email traffic by scenario#1 which is to save 10% of emails and scenario#2 which is to save 30% of emails only by reducing the non-relevant email recipients.</p><h4>Results</h4><p>The following figure shows the results of the single experiments and sets them into relation. The blue line refers to the original email production and is, of course, on the top of the chart. We clearly see that the reduction of attachments (orange line) has the most significant impact on the CO2 emissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*lyAKAf3irhx62Wzngj71qg.png" /><figcaption>Influences of Email Behaviours on global CO2 Emissions in Million Tonnes (Image by Author)</figcaption></figure><p>Now, let’s focus on the particular impacts of our potential levers on the global CO2 Emission. We need to consider that the previously calculated yearly CO2 Emission (176 million tonnes) differs from the SD simulation (169 million tonnes). This is because I estimated that “regular” emails are 80% HTML and 20% plain text. Since plain text emails emit less CO2, the total annual emissions of the simulation are lower than with the previous simple calculation. The table shows that halving the amount of emails with attachments and replacing them by HTML emails with links would decrease the CO2 Emission by 30%. The reduction of email recipients by 30% would save 6.9% CO2 costs, whereas the reduction by 10% would save 2%. Using plain text instead of HTML would save 5% greenhouse emissions. The removal of signatures has the lowest impact with at least 0.4% CO2 emission decrease.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*tfyeiNbGFBoCa_AdPsopuQ.png" /><figcaption>Comparison of Results of tested Levers (Image by Author)</figcaption></figure><p>Fortunately, we can also calculate the absolute reductions of email-caused CO2 emissions, which are also shown in the table. Again, to get a better feeling for these numbers, let’s search for equivalents we all know from our everyday life:</p><ul><li>The reduction of emails with attachment would lead to a CO2 saving equivalent to <strong>10 million cars</strong> <strong>driving </strong>in one year.</li><li>When we reduce the number of recipients only by 10%, we would save greenhouse emissions equivalent to <strong>412 billion smartphones charged</strong>.</li><li>Even the last placed lever, where we reduce the signatures at 50% of our emails, would lead to positive environmental effects equivalent to <strong>148 wind turbines </strong>running for an entire year.</li><li>And if we would apply all of those measures, we would save CO2 emissions similar to almost <strong>1 million tanker trucks’ worth of gasoline</strong>.</li></ul><p>As we pointed out earlier that the global email traffic is forecasted to increase further, the percentage reductions would save even more absolute CO2 emissions in future.</p><h3>Conclusions</h3><p>We have seen that emails are not as climate-neutral as expected. But on the contrary, there’s a tremendous CO2 emissions caused by digital waste that is produced by unnecessary overhead in email data structures. There are a number of levers which could impact the emissions significantly, especially the usage of links instead of attachments. But also other, even more trivial actions, e.g. avoiding huge email signatures, would have a noticeable impact on the greenhouse emissions. And there are surely more possible levers, which we did not cover in this article, such as the deletion of older emails that are no longer important either for business or for private purposes. The key message is that it is quite easy to save CO2 emissions without spending any money — it can be simply achieved by adjusting our human email behaviours. Further research could be performed in order to assess the technical development or organization of systematic measures that apply the levers which we have analysed in this article.</p><p><a href="#_ftnref1">[1]</a> <a href="https://wirtschaftslexikon.gabler.de/definition/e-mail-33576">https://wirtschaftslexikon.gabler.de/definition/e-mail-33576</a></p><p><a href="#_ftnref2">[2]</a> <a href="https://www.faz.net/aktuell/technik-motor/digital/ray-tomlinson-der-erfinder-der-e-mail-ist-tot-14109850.html">https://www.faz.net/aktuell/technik-motor/digital/ray-tomlinson-der-erfinder-der-email-ist-tot-14109850.html</a></p><p><a href="#_ftnref3">[3]</a> <a href="https://www.bbc.com/future/article/20200305-why-your-internet-habits-are-not-as-clean-as-you-think">https://www.bbc.com/future/article/20200305-why-your-internet-habits-are-not-as-clean-as-you-think</a></p><p><a href="#_ftnref4">[4]</a> <a href="https://de.statista.com/statistik/daten/studie/252278/umfrage/prognose-zur-zahl-der-taeglich-versendeter-e-mails-weltweit/">https://de.statista.com/statistik/daten/studie/252278/umfrage/prognose-zur-zahl-der-taeglich-versendeter-e-mails-weltweit/</a></p><p><a href="#_ftnref5">[5]</a> <a href="https://www.statista.com/statistics/1270488/spam-emails-sent-daily-by-country/">https://www.statista.com/statistics/1270488/spam-emails-sent-daily-by-country/</a></p><p><a href="#_ftnref6">[6]</a> <a href="https://www.statista.com/statistics/1270459/daily-emails-sent-by-country/">https://www.statista.com/statistics/1270459/daily-emails-sent-by-country/</a></p><p><a href="#_ftnref7">[7]</a> <a href="https://medium.datadriveninvestor.com/email-attachments-generate-nearly-6-000-unnecessary-and-unsecure-files-per-employee-per-year-a38385bec7a4">https://medium.datadriveninvestor.com/email-attachments-generate-nearly-6-000-unnecessary-and-unsecure-files-per-employee-per-year-a38385bec7a4</a></p><p><a href="#_ftnref8">[8]</a> <a href="https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator">https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator</a></p><p><a href="#_ftnref9">[9]</a> <a href="https://www.kba.de/DE/Statistik/Kraftverkehr/VerkehrKilometer/vk_inlaenderfahrleistung/vk_inlaenderfahrleistung_node.html;jsessionid=3680DFA2AFAF0D0754D8E522FE14CC1E.live21321">https://www.kba.de/DE/Statistik/Kraftverkehr/VerkehrKilometer/vk_inlaenderfahrleistung/vk_inlaenderfahrleistung_node.html;jsessionid=3680DFA2AFAF0D0754D8E522FE14CC1E.live21321</a></p><p><a href="#_ftnref10">[10]</a> <a href="https://ourworldindata.org/co2-emissions">https://ourworldindata.org/co2-emissions</a></p><p><a href="#_ftnref1">[11]</a> <a href="https://www.geeksforgeeks.org/e-mail-format/">https://www.geeksforgeeks.org/e-mail-format/</a></p><p><a href="#_ftnref2">[12]</a> <a href="https://365tips.be/en/share-files-in-office-365-best-practise/">https://365tips.be/en/share-files-in-office-365-best-practise/</a></p><p><a href="#_ftnref3">[13]</a> <a href="https://support.google.com/a/users/answer/9296687?hl=en">https://support.google.com/a/users/answer/9296687?hl=en</a></p><p><a href="#_ftnref1">[14]</a> <a href="https://www.energuide.be/en/questions-answers/do-i-emit-co2-when-i-surf-the-internet/69/">https://www.energuide.be/en/questions-answers/do-i-emit-co2-when-i-surf-the-internet/69/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3186673818c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-decrease-the-carbon-footprint-of-digital-communication-f3186673818c">How to Decrease the Carbon Footprint of Digital Communication</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fuzzy data structures — curse or blessing?]]></title>
            <link>https://medium.com/data-science/fuzzy-data-structures-curse-or-blessing-5ee34ff39cd7?source=rss-8c9dbf8eb438------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ee34ff39cd7</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[semiconductors]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Professor Simon J. Preis, Ph.D.]]></dc:creator>
            <pubDate>Thu, 10 Jun 2021 15:09:14 GMT</pubDate>
            <atom:updated>2021-06-11T13:00:38.534Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fuzzy data structures — curse or blessing?</h3><h4>Challenges and Solutions from an industrial application point of view</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cXC4mrRaBETomVpU" /><figcaption>Photo by <a href="https://unsplash.com/@sortino?utm_source=medium&amp;utm_medium=referral">Joshua Sortino</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Some time ago, I found an interesting article<a href="#_ftn1">[1]</a>, which compared SQL and NoSQL data processing via Python and pointed out some drawbacks of SQL. One of the prominently discussed weaknesses of SQL was the lack of storing complex object structures (e.g. Python dictionaries) in a simple attribute, which violates the first normal form (1NF)<a href="#_ftn2">[2]</a>. In this context, the author briefly discussed the case of fuzzy data formats, which are technically possible with MongoDB. Though this issue was not further spotted in that article, I was wondering if this feature of MongoDB is really a benefit or rather a weakness, especially in the context of industrial applications where I am professionally located. As I could not find an article that pervasively addressed this question, I wanted to analyze and discuss this issue and share my findings!</p><p>After this short introduction we will briefly discuss characteristics of different database technologies. Then, we will assess challenges when working with fuzzy data structures based on two simple collections in MongoDB. Next, we will transfer the findings from the simplified examples to a real-world industrial application in the semiconductor industry. Finally, we will have a closer look on tools and methods to overcome these challenges.</p><h3>Technologies</h3><p>Firstly, the principles of data storage with MongoDB must be understood. MongoDB is typically classified as a NoSQL database concept<a href="#_ftn3">[3]</a>. However, NoSQL acts as an umbrella term for multiple technologies which do not only differ from RDBMS but do also differ significantly between each other. Well known types of NoSQL database technologies are key value stores (e.g., Amazon Dynamo), wide column stores (e.g., Cassandra), graph databases (e.g. Nio4j) and document stores, where MongoDB belongs to. A big advantage of document-oriented databases against SQL lies in the fact that data structures can be persisted and queried in the same format as in the software code that works with this data. The data format follows a JSON-like format which is semi-structured and hierarchical. According to AWS, it fits to the requirements of catalogues, user profiles and content management systems where each document can have a unique format. Document-oriented databases provide a flexible indexation, high-performance ad-hoc queries and analyses over collections<a href="#_ftn4">[4]</a>. A performance comparison between MongoDB and MySQL came to the result that the write performance of MongoDB is significantly higher, especially for larger datasets, whereas SQL performs faster queries<a href="#_ftn5">[5]</a>. Hence, it can be implied that the application of document stores is useful for scenarios that require frequent data inserts or updates, and/or deal with larger data. Such use cases exist also in industrial environments: for instance, collecting telemetry data from production equipment requires highly frequent inserts and, dependent on the machine complexity, may consist of multiple variables that refer to particular sensors.</p><h3>Challenges</h3><p>In order to assess the challenges, a sample MongoDB database is created. The <strong>first scenario</strong> uses a collection that consists of three documents as shown in Figure 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*1zOWDoQiozaifaR6qH20pg.png" /><figcaption>Figure 1: Sample Collection with Documents</figcaption></figure><p>The documents refer to persons and store some of their attributes which are name, age, city and hobbies; the “_id” attribute is the default identifier of a document. At the first glance, it looks like a straightforward data structure. But let’s focus on the attribute “hobbies”. As marked in the figure, we see that this attribute exists for each document. However, the data type of this attribute is obviously different between the documents: (1) an array, (2) an object and (3) a string. This type of fuzziness is really a difference to relational databases, where the data type of a table attribute is strictly defined for all records.</p><p>So, we see that we can create flexible — or more negatively spoken — indeterminable data structures. Let’s focus directly on an associated major challenge: MongoDB does not provide a search feature that queries as flexible as the data structures can be constructed. What does this mean in practice? Let’s have a closer look on the “hobbies” attribute values of the three documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*xs139XLktRgQ8lIvNH3F1g.png" /><figcaption>Figure 2: More detailed view on the hobbies</figcaption></figure><p>If we wish to find all persons who have “soccer” as a hobby, we as human beings can simply see that there are two relevant persons in our collection. However, imaging we have thousands or millions of such documents like on social media platforms, we would rather apply a database query that searches for something like “all persons who have soccer as a hobby”. However, this is technically not possible with this type of fuzzy data structure as MongoDB requires a unique path where it applies the filter. Though wildcard search is possible and applies also to nested fields in an object, this search tool cannot deal with different data structures<a href="#_ftn6">[6]</a>. This means practically that we could perform a search on fields that contain the phrase “hobbies” at any level of a document’s hierarchy — but the search does not consider our nested array or object since the inner field name differs from “hobbies”.</p><p>In addition, we could argue that it is also semantically not useful to search like that, because it goes into the direction of searching the famous “needle in a haystack”. So, we can state that MongoDB provides extensive flexibility when we want to store data that is similar in content but different in structure, but as soon as we want to query or analyze this data, we are limited with this fuzzy data structure. Of course, we could put the query logic to an external code, for instance, in Python, and build a function that successively searches for hobbies as string, hobbies in an array and hobbies in an object, and then returns the merged result. But this is a static solution and as long as we do not “close the door” for other structural variants, we cannot ensure that the query would not miss relevant results.</p><p>The <strong>second scenario</strong> which refers to fuzzy data structures is related to a collection that contains documents with different attributes. In our example is a set of persons that are described by different information. We now struggle with the situation that Adam is described by his hair color, whereas Brian and Eve are described by their eye color. Even worse, the attribute “eye color” uses a different spelling in the two documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*5ijy50gd-oUdImr_Y5x4Rg.png" /><figcaption>Figure 3: Collection with documents that have different attributes</figcaption></figure><p>While we are now able to filter consistently by hobbies, we cannot search straightforward for persons who have a particular hair or eye color. Again, our queries would potentially ignore documents that would semantically fit to the filter criteria. In addition, our analysis capabilities are limited as we cannot ensure that we have the same information available for all documents. Sure, this issue can also exist in SQL databases if fields in a table are not configured to be mandatory, but at least we can rely on a fixed set of attributes. However, with MongoDB, the variety of used attributes between documents in the same collection is potentially endless if continuously new documents are inserted. Hence, it is even not determinable for a data analyst which attributes are available to work with.</p><h3>Findings transfer to industrial application</h3><p>We discussed several challenges in the previous section based on very simple examples. But are these theoretical constructs or can these be real issues in industrial applications? To answer this question, let’s have a closer look on the previously mentioned use case where telemetry data is collected from machines. Figure 4 shows three machines E1 to E3 that belong to two different types T1 and T2. In addition, we see two types of interface definitions ID1 and ID2. Via those interfaces, the equipment telemetriy data is transferred to a central CIM database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*fXHOJEh_exUALuoDX5gdZQ.png" /><figcaption>Figure 4: Schematic of an equipment data integration</figcaption></figure><p>In the semiconductor industry, we typically work with an interface standard called “SECS/GEM”. An event message based on this standard has an object-like character similar to JSON. However, a benefit of SECS/GEM against JSON or XML is the reduced data overhead which results in less network traffic and faster data processing<a href="#_ftn7">[7]</a>. Anyway, a raw event message is not compliant to a relational table as it contains non-atomic information that violates 1NF. So, storing an equipment event to a SQL database always requires an ETL process. However, an event message could be directly stored to MongoDB as a document. As the structure of a SECS/GEM message body is defined by the standard, the risk discussed for scenario 1 is low in this application — as long as all equipment interfaces follow this standard (e.g. E3 and E1). But also semiconductor facilities may suffer from legacy equipment that does originally not support any data integration. Upgrades to SECS/GEM may require the maker’s support and tend to higher costs. However, there are cost-efficient alternatives to enable data integration for legacy equipment such as smart sensors, retrofit kits and edge gateways<a href="#_ftn8">[8]</a>. Having different interface definitions, we have the potential risk of different data structures as far as the SECS/GEM message format is not followed or the data is not transformed before storage. So, from industrial application perspective, the challenges mentioned in scenario 1 are theoretically possible in our described use case but can be prevented if experienced engineers foresee them. The ability to store same information in different formats is not seen as a benefit in this practical use case.</p><p>Let’s move on to scenario 2: In the semiconductor industry, we typically work with different machine types provided by different equipment makers (see E1 and E2 vs. E3). Most of these machines have internal software and built-in sensors. Most likely, the available sensors but also the internal technical names of similar sensors differ between machines and makers, even though they refer to the same type of information, e.g. pressure or temperature. Without data harmonization between the equipment interfaces, this situation would lead to the problem statement sketched in scenario 2: a) we cannot consistently filter by or analyze sensor values within the machine park if single machines do not provide that information and b) we have to deal with sensor names that refer to the same information but with different spelling. So, we could conclude that these risks are valid for our industrial use case. However, we can also imply that the capabilities of MongoDB fit to the requirement of storing different message contents and flexibly adding data of newly implemented sensors.</p><h3>How to address these challenges</h3><p>Fortunately, there are multiple techniques to overcome the previously discussed challenges. There are probably more approaches and I do not claim that the following list is complete, but I want to provide an overview on the bandwidth of techniques from reactive consistency validation until real preventive measures.</p><p><strong>1)</strong> <strong>Check Field Existence</strong></p><p>As a data analyst, you may want to check on particular fields if they exist in a collection as you expect. This prevents you from invalid filters or aggregations. MongoDB provides the “$exists” operator which can be used in a query to return only documents where the specific field is contained or where it is missing<a href="#_ftn9">[9]</a>. Getting results for both filters indicates that the field must be enjoyed with care.</p><p><strong>2)</strong> <strong>Retrieve all Fields from a Collection</strong></p><p>If you want to apply a systematic or even regular comparison to identify missing fields, checking the field existence as previously discussed is probably not the right approach. You rather want to extract a list of fields that exist in your collection at all. This can be achieved by using the “map reduce” approach. The following Python code shows an sample implementation using the pymongo API.</p><pre>import pymongo<br>from bson.code import Code<br><br>conn = &quot;mongodb://localhost:27017&quot;<br>client = pymongo.MongoClient(conn)<br><br>db = client.mytestDB<br><br>map = Code(&quot;function () {&quot;<br>            &quot;  for (var k in this) { emit(k, null); }&quot;<br>            &quot;}&quot;)<br><br>reduce = Code(&quot;function(k, s) { return null; }&quot;)<br><br>result = db.diffAttributes.map_reduce(map, reduce, &quot;r&quot;)<br>for d in result.find():<br>    for k, v in d.items():<br>        if k == &quot;_id&quot;:<br>            print(v)</pre><p>By executing this code, we receive the following printed list of unique fields from the assessed collection, which is called “diffAttributes” in this example. Of course, we could instead transfer and persist the list into a document or database or directly implement further checks within the Python code.</p><pre>name<br>_id<br>eye colour<br>eye color<br>hobbies<br>hair color</pre><p><strong>3)</strong> <strong>Check the Schema</strong></p><p>If you use MongoDB Compass for database management, you can simply analyze your database schema considering all documents in a collection. With the result as shown in Figure 5 you can easily assess the shaping of the fields in a collection. For instance, we see that there are three data types used for the field “hobbies”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*aANYZdo5s56QV-z4t4ABbg.png" /><figcaption>Figure 5: Result of a Schema analysis with MongoDB Compass</figcaption></figure><p><strong>4)</strong> <strong>Force a schema standard</strong></p><p>From our discussion in this article, one could have the impression that MongoDB builds up on anarchy. This is surly not the case. Though there are multiple ways to store complex data structures in flexible ways, we can reduce this flexibility to make our collection contents determinable. Just like with SQL databases you can define validation rules that state, for instance, which fields are mandatory, which data type must be adhered to or which values are allowed<a href="#_ftn10">[10]</a>. For instance, as shown in Figure 6, we can state that our “hobbies” attribute only allows string values. By executing this validation rule, MongoDB Compass shows which document is valid or not. In addition, we can configure that a newly inserted document that violates this rule should be declined. Hence, we can keep our collection consistent from scratch. The only limitation is that this feature is not available if MongoDB Compass is connected to a data lake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*rTbKx5q6y5sWPsYcIJvzRA.png" /><figcaption>Figure 6: Validation Rules</figcaption></figure><p><strong>5)</strong> <strong>Provide a User Interface (UI)</strong></p><p>If the data you want to store in MongoDB comes directly from user entries, you could reduce the data structure fuzziness by building a mature UI. This is similar to the approach of having a standardized equipment interface where the data is generated automatically. You may configure dropdown boxes to force allowed values, make textboxes mandatory or even provide workflows to collect data in a certain order. Hence, we ensure that only data is submitted to the database that adhere to allowed data formats.</p><p>6) <strong>Apply Data Governance (DG)</strong></p><p>It sounds trivial: but knowing if your data is consistent or not requires first of all that rules are defined and aligned between the stakeholders with regards to a specific data object. If there has never been a decision if the “hobbies” should be an array or which spelling must be followed for “eye color”, you cannot technically resolve that issue in your collection. Here comes DG into place. By having clear responsibilities and decision processes for data, you gain multiple benefits such as increased speed of IT project implementation, improved data quality and increased value of the data<a href="#_ftn11">[11]</a>. If all IT implementation activities follow the decisions from DG, we ensure data consistency by design.</p><h3>Conclusions</h3><p>In this article, we assessed the challenges and risks which may occur when allowing fuzzy data structures in a document store. We analyzed and evaluated them based on two simplified scenarios implemented in MongoDB and transferred the findings to a selected industrial use case. We implied that there are situations in industrial applications where the flexibility of MongoDB could be useful but which drawbacks might exist. Finally, we discussed techniques to overcome these challenges where we saw rather reactive approaches (e.g. field existence check) as well as preventive measures (e.g. data governance).</p><p><strong>References</strong></p><p>[1] M. Sosna, A Hands-On Demo of SQL vs. NoSQL Databases in Python -Impress your friends with SQLAlchemy and PyMongo (2021), <a href="https://towardsdatascience.com/a-hands-on-demo-of-sql-vs-nosql-databases-in-python-eeb955bba4aa">https://towardsdatascience.com/a-hands-on-demo-of-sql-vs-nosql-databases-in-python-eeb955bba4aa</a></p><p>[2] Wikipedia, First normal form (n.d.), <a href="https://en.wikipedia.org/wiki/First_normal_form">https://en.wikipedia.org/wiki/First_normal_form</a></p><p>[3] L. Schaefer, What is NoSQL? (n.d.), <a href="https://www.mongodb.com/nosql-explained">https://www.mongodb.com/nosql-explained</a></p><p>[4] Amazon, Was ist eine Dokumentdatenbank? (n.d.), <a href="https://aws.amazon.com/de/nosql/document/">https://aws.amazon.com/de/nosql/document/</a></p><p>[5] M. Shah, MongoDB vs MySQL: A Comparative Study on Databases (2017), <a href="https://www.simform.com/mongodb-vs-mysql-databases/#:~:text=MongoDB%20vs%20MySQL%3A%20Performance%20%26%20Speed,is%20more%20sensitive%20to%20workload">https://www.simform.com/mongodb-vs-mysql-databases/#:~:text=MongoDB%20vs%20MySQL%3A%20Performance%20%26%20Speed,is%20more%20sensitive%20to%20workload</a></p><p>[6] MongoDB, Path Construction (n.d.), <a href="https://docs.atlas.mongodb.com/reference/atlas-search/path-construction/#std-label-ref-path">https://docs.atlas.mongodb.com/reference/atlas-search/path-construction/#std-label-ref-path</a></p><p>[7] B. Grey, SECS/GEM series: Protocol Layer (2018), <a href="https://www.cimetrix.com/blog/secs-gem-series-protocol-layer">https://www.cimetrix.com/blog/secs-gem-series-protocol-layer</a></p><p>[8] I. Lamont, 4 ways to bring legacy manufacturing equipment into the IoT age (2019), <a href="https://www.hpe.com/us/en/insights/articles/4-ways-to-bring-legacy-manufacturing-equipment-into-the-iot-age-1903.html">https://www.hpe.com/us/en/insights/articles/4-ways-to-bring-legacy-manufacturing-equipment-into-the-iot-age-1903.html</a></p><p>[9] MongoDB, Query for Null or Missing Fields (n.d.), <a href="https://docs.mongodb.com/manual/tutorial/query-for-null-fields/">https://docs.mongodb.com/manual/tutorial/query-for-null-fields/</a></p><p>[10]MongoDB, Set Validation Rules for Your Schema (n.d.), <a href="https://docs.mongodb.com/compass/current/validation/">https://docs.mongodb.com/compass/current/validation/</a></p><p>[11]CDQ, Data Governance (n.d.), <a href="https://www.cc-cdq.ch/data-governance">https://www.cc-cdq.ch/data-governance</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ee34ff39cd7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/fuzzy-data-structures-curse-or-blessing-5ee34ff39cd7">Fuzzy data structures — curse or blessing?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>