<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andy Petrella on Medium]]></title>
        <description><![CDATA[Stories by Andy Petrella on Medium]]></description>
        <link>https://medium.com/@andy-petrella?source=rss-b7a7af8873c9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*J3kYFaWj32FJRJE1BteKxw.png</url>
            <title>Stories by Andy Petrella on Medium</title>
            <link>https://medium.com/@andy-petrella?source=rss-b7a7af8873c9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:29:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@andy-petrella/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Datastrophes: Il Buono, il Brutto, il Cattivo]]></title>
            <link>https://andy-petrella.medium.com/datastrophes-il-buono-il-brutto-il-cattivo-c0bad03e8dcb?source=rss-b7a7af8873c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0bad03e8dcb</guid>
            <category><![CDATA[ai-management]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-intelligence]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Andy Petrella]]></dc:creator>
            <pubDate>Mon, 01 Mar 2021 20:38:42 GMT</pubDate>
            <atom:updated>2021-03-02T13:57:21.736Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, I want to introduce you to something I had to deal with as either an engineer, a scientist, a leader, or a manager: <em>Datastrophes</em>.</p><p>Nevertheless, because most events have an opportunistic component, I will not only cover the odd-part of datastrophes but also what and how you could benefit from them.</p><blockquote>Here are the key takeaways in this article:</blockquote><blockquote>- The success of a data project depends on a series of assumptions to be valid and robust.</blockquote><blockquote>- A datastrophe is an impactful event occurring in a (managed) data system.</blockquote><blockquote>- A datastrophe always has a good side!</blockquote><h3>Data projects: stacking up assumptions</h3><p>There is a lot of literature about the phases of a data project; although there is no global consensus yet, I will use as an illustrative example the <a href="https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining">CRISP-DM</a> (originally dedicated to Data Mining):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*HX8tnnV2V2QuV4pkqJr3pw.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining">CRISP-DM</a></figcaption></figure><blockquote>Note: In a (big) data science approach, we may swap the two first boxes, but as you can see, that wouldn’t change the process dramatically.</blockquote><p>In each of these phases, assumptions are made by the different actors in order to create feasibility constraints. Here are some examples:</p><ul><li>Business understanding: students generally don’t subscribe easily to health insurance.</li><li>Data understanding: the data received for the surveying company represents students adequately.</li><li>Data preparation: the age of students is always greater than 18.</li><li>Data modeling: the distribution of the students’ income is bimodal.</li><li>Evaluation: testing the conversion rate is enough.</li><li>Deployment: everything is gonna be fine (sorry for the troll).</li></ul><p>Assumptions are generally drawn from experience acquired on the field (or gut feeling), otherwise derived manually (or not) from data (e.g. dimension reduction, feature selection).</p><p>In a way, they are part of the knowledge existing or acquired by the actors along the process.</p><p>Of course, it is fine to make assumptions, however, they come with downsides; e.g. they might not be robust enough to hold in case of events like, well, COVID-19 for instance.</p><p>Let’s take a couple of examples to describe further what assumptions can be made and their consequences.</p><h3>The case of SQL</h3><p>A simple case to consider is a SQL query, you are making the assumption that its model will remain unchanged, and anyway, you are not responsible for it… so why bother?</p><p>Or maybe, in the event it would change, you assume “someone” will let you know.</p><p>Until the first alteris applied, your code blows, and you’re on the hook for days to find out what to do until production is back up.</p><p>What happens the next time? Well, as a matter of fact, you still don’t bother! Of course, it is still not your responsibility, yet, you don’t feel comfortable because you depend on something you don’t manage.</p><p>However, if you are accountable for the results you produce there are a few best practices or habits you will develop, such as:</p><ul><li>Logging the schema: to find the cause rapidly or, using a monitoring tool to alert you.</li><li>Testing the schema stability: and stop the application with a warning message, resulting in a comprehensible ticket.</li><li>Reviewing the schema of the table periodically (or ping your DBA): to clear up your mind.</li></ul><p>Consequently, you start taking some responsibility for it, therefore you dedicate time to it. Imagine the consequences when projects are piling up and this dedication is spotted.</p><h3>The case of “AI”</h3><p>Considering now a more advanced case, most “AI”s are making an assumption that the world is close <em>enough</em> to the world described by the data used during its conception. From those, numerous opaque assumptions are derived and embedded in their prediction capabilities.</p><p>It is somewhat similar to the assumption made with the SQL query, with a significant difference that you have little (if not none) idea of what those assumptions are!</p><p>I won’t dig this much, as this is not the subject of this post, however, you’ll find at the end of this post a series of articles corroborating it.</p><p>Those assumptions are continuously used after the deployment of the project, which implies that “the health of your business depends on their robustness”.</p><p>Therefore, you can feel how uncomfortable it is for the project team to “let it go” live.</p><h3>Datastrophe, a definition</h3><blockquote>A datastrophe is a catastrophe with data. <br>A catastrophe is the denouement of a drama. <br>A datastrophe is the denouement of a <a href="https://www.dama.org/cpages/home">DAMA</a> (data management).</blockquote><p>Managing data is a wide subject, often misleadingly reduced to data governance only. Data management is the sum of all needed methods and processes to ensure that data is used appropriately, generating value and not risks. I wrote a short report on that subject that has been edited by O’Reilly, <a href="https://learning.oreilly.com/library/view/what-is-data/9781492090328/">check it here</a>.</p><p>We can’t reduce management to the only resolution of conflicts and issues, but datastrophes are literally the events supporting the need for better data management. Because issues aren’t showing up <em>in</em> data, but <em>what</em> used it.</p><p>Above, I explained how data projects (projects using data) rely on superheroes, the “assumptions”. My point is their supervillains are the datastrophes.</p><p>Nevertheless, in the current unconventional era, it is not anymore clear cut that the superheroes are the good team (e.g. <a href="https://en.wikipedia.org/wiki/The_Boys_(2019_TV_series)">The Boys</a>), and the supervillains are the bad one (e.g. <a href="https://en.wikipedia.org/wiki/Suicide_Squad_(film)">Suicide Squad</a>). That is the question…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*HsB8NOjNrqFenHfsWN376Q.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Question_(character)">DC-comics — Question</a></figcaption></figure><p>Therefore, following the movie analogy, I tend to say, like most <a href="https://www.quora.com/How-many-sides-does-a-coin-have">popular coins</a>, that datastrophes have 3 sides: The Good (Il Buono), The Bad (Il Cattivo), and The Ugly (Il Brutto).</p><blockquote>And yes the characters in the title are ordered differently per language. Heh, what did you assume? In fact, I always wondered how many non-Italian speakers assume that Brutto means Bad…</blockquote><p>Actually, each datastrophe will have its own 3-sides combo, this is why I’ll dedicate future posts to detail them individually.</p><p>However, please keep in mind now and forever that when a datastrophe shows up, it is simultaneously:</p><ul><li>Bad for you: you’ll undoubtedly be pointed out as guilty or responsible even though you may not feel that way,</li><li>Ugly for the group: data intelligence results are used to draw strategic or tactical moves resulting in disastrous events,</li><li>Good for both: please focus on this one, because there is always a light you can seek:</li></ul><ol><li>You can both learn from it and anticipate them in the future,</li><li>You can discover the opportunities opened by the new learning that those assumptions aren’t (always) right.</li></ol><p>By the way, without commenting, I’ll let you look at the below picture… it may dismantle some of your, well, assumptions about the characters ;-).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*iw0k4YIG771JAx1V" /><figcaption>Check on <a href="https://public.tableau.com/profile/filippo.mastroianni#!/vizhome/TheGoodtheBadandtheUgly/Dashboard1">this fantastic Tableau</a> on the movie created by Filippo Mastroianni (@FilMastroianni)</figcaption></figure><p>There are other names that were given, depending on which sides of the coin you look at, such as:</p><ul><li>Data issues: data monitoring.</li><li>Data downtime: data observability.</li><li>Dark data: coined by Gartner, means unused data.</li><li>Dark data (my preferred) coined by <a href="https://www.amazon.com/Dark-Data-What-Dont-Matters/dp/069118237X">David J. Hand</a> who lists 14th different DD-types:</li></ul><ol><li>Data We Know Are Missing.</li><li>Data We Don’t Know Are Missing.</li><li>Choosing Just Some Cases.</li><li>Self-Selection is a variant of DD-Type 3.</li><li>Missing What Matters.</li><li>Data Which Might Have Been.</li><li>Changes with Time.</li><li>Definitions of Data.</li><li>Summaries of Data.</li><li>Measurement Error and Uncertainty.</li><li>Feedback and Gaming.</li><li>Information Asymmetry.</li><li>Intentionally Darkened Data.</li><li>Fabricated and Synthetic Data.</li></ol><h3>What’s next?</h3><p>Throughout this article, you have learned what datastrophes are, how they relate to assumptions (made by humans or not), and why their management is crucial to ensure the viability of any data projects.</p><p>In the following posts, I will introduce examples of datastrophes, with their definition, their context of occurrence, and how they can be handled with a dedicated management system (such as a <a href="https://andy-petrella.medium.com/what-and-why-data-intelligence-management-aeba1ec5a242">Data Intelligence Management platform</a>). So… stay tuned ;-).</p><p><em>Are you going to leave yourself without enhanced monitoring capabilities of your data applications, reports, and AI? If your answer’s “no”, don’t hesitate to reach out to </em><a href="https://www.linkedin.com/in/andypetrella/"><em>me</em></a><em> or on our website </em><a href="https://kensu.io/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=evangelization&amp;utm_content=datastrophe"><em>Kensu.io</em></a><em>.</em></p><h3>References</h3><ul><li><a href="https://www.kdnuggets.com/2021/02/machine-learning-assumptions.html">https://www.kdnuggets.com/2021/02/machine-learning-assumptions.html</a></li><li><a href="https://www.quora.com/In-machine-learning-what-are-some-important-assumptions-that-people-tend-to-forget-when-fitting-a-specific-model">https://www.quora.com/In-machine-learning-what-are-some-important-assumptions-that-people-tend-to-forget-when-fitting-a-specific-model</a></li><li><a href="https://www.quora.com/In-Machine-Learning-why-do-we-often-apply-the-assumption-of-Independent-and-Identically-Distributed-records">https://www.quora.com/In-Machine-Learning-why-do-we-often-apply-the-assumption-of-Independent-and-Identically-Distributed-records</a></li><li><a href="https://towardsdatascience.com/back-to-basics-assumptions-of-common-machine-learning-models-e43c02325535">https://towardsdatascience.com/back-to-basics-assumptions-of-common-machine-learning-models-e43c02325535</a></li><li><a href="https://jrodthoughts.medium.com/assumptions-dimensionality-and-some-of-the-original-motivations-behind-deep-learning-a4df2f0f1689">https://jrodthoughts.medium.com/assumptions-dimensionality-and-some-of-the-original-motivations-behind-deep-learning-a4df2f0f1689</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0bad03e8dcb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What and Why Data Intelligence Management?]]></title>
            <link>https://andy-petrella.medium.com/what-and-why-data-intelligence-management-aeba1ec5a242?source=rss-b7a7af8873c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeba1ec5a242</guid>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <dc:creator><![CDATA[Andy Petrella]]></dc:creator>
            <pubDate>Mon, 15 Feb 2021 19:27:49 GMT</pubDate>
            <atom:updated>2021-03-02T13:55:40.966Z</atom:updated>
            <content:encoded><![CDATA[<p>When we think about enterprise-level management, a lot of tools and systems are available for different key positions in organizations:</p><ul><li>ITSM for CIO, CTO, Head of Engineering;</li><li>CRM for Head of Sales, Head of Marketing;</li><li>ERP for COO, CFO, Management.</li></ul><p>Albeit CDO, CDMO, Head of Data, Head of Analytics will leverage Data Management, the critical aspects of the data lifecycle won’t be intrinsically covered:</p><p>Managing the value of data throughout its various usages.</p><p>This is what I’ll cover here as the first entry of this series of blogs about <strong>Data Intelligence Management</strong>.</p><blockquote>Here are the key takeaways in this article:</blockquote><blockquote>1. The only way the value of data can grow is with smart data applications.</blockquote><blockquote>2. Manage your data application to overcome the limitations of Data Management.</blockquote><blockquote>3. ML&amp;AI (Machine Learning &amp; Artificial Intelligence) projects won’t be sustainable without a tailored intelligence management system.</blockquote><h3>Data is an asset</h3><p>Like any asset, data takes value along a processing line. Consider oil, for example, its value is only increased with refinements.</p><p>And like any other assets, especially nowadays, data can be bought, or resold (e.g. this article lists ~120 data brokers).</p><p>Furthermore, as any value generation process works, the better it is, the more value you generate…</p><p>But what means “better”? Well, simply, smarter! For which, a definition is presented below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*RRTNjpUpCK-bSD13" /><figcaption>Smart definition</figcaption></figure><p>This makes clear that in our super fast and highly competitive era, you have to differentiate your company on the market by the level of intelligence you put on the data.</p><h3>Data Intelligence generates the value</h3><p>After all… data is nothing but the result of applications — I mean data is never naturally:</p><ul><li>Created.</li><li>Modified.</li><li>Deleted.</li></ul><p>Hence, you are responsible to increase the value of your data by creating applications. Obviously, those applications will only increase the value of the data by adding a smart twist. The kinds of smart twists have evolved over the last few decades, from ad-hoc business rules to automated AI.</p><p>And that is <strong>Data Intelligence</strong> which aims to increase the value of data with smart applications that can be, potentially, automated, …</p><p>Examples of such applications are:</p><p>▹ Analytical tools (Excel, …);</p><p>▹ Artificial Intelligence models (Tensorflow, …);</p><p>▹ BI reports (Tableau, …);</p><p>▹ Data Services (API, …).</p><p>The smarter, the more automated, and the more trusted your applications are, the better will be your position compared to your competition.</p><p>That said, let’s consider their trustworthiness in the coming section.</p><h3>The responsibility of data applications</h3><p>To create continuous value with data, all applications must be executed autonomously (say, “in production”).</p><p>Therefore automation is key to success.</p><p>One of the biggest challenges that come with that is the trustworthiness of the automated results. You might be tempted to call this “data quality”, however, this is reductive, because you would discard the biggest information needed to assess it: its context.</p><p>Let’s take an example if you are considering a CSV file generated automatically that will create marketing campaigns from your CRM using an extractor tool that has been parameterized to extract several segments automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8hAOEpMoLUVq41HMSRcJ5A.png" /><figcaption>Data application example: segments from CRM</figcaption></figure><p>You will trust that file as long as you trust the parameters for the extractor are accurate, and that the data in the CRM is satisfying its constraints.</p><p>In other terms, the perceived quality of the results depends on the quality of the extraction and the data. This is somewhat disturbing because the data in CRM is outside your perimeter of influence, you have no control, you are not responsible for it. Note that, this is true whether or not silos exist, it is not because you can access some data, or you know it exists, that you are responsible for it. Nevertheless, how you will be using the data is undoubtedly your responsibility?</p><p>So you have two choices:</p><ol><li>Accept that if you get garbage, you create garbage (<em>Garbage in, Garbage out</em>).</li><li>Take some of the responsibilities to cover your case, and anticipate issues (<em>Garbage in… no out … or less out).</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JXoCfSTAYLwygJW5HVMHlw.png" /><figcaption>Garbage in — Garbage out</figcaption></figure><p>If you are worried about the quality of your results, and you want to be proactive, you have again two options:</p><ol><li>Scan periodically your results to assess the quality;</li><li>Report metrics from your extractor.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iSv3MjwTj4rLZhF6" /><figcaption>Data Intelligence Manager</figcaption></figure><p>Whilst the first option is part of Data Management (e.g. data monitoring, or data observability), the second option is part of <strong>Data Intelligence Management</strong>.</p><p>In that matter, if you are willing to do “Results Management”, you must first consider Data Intelligence Management.</p><h3>Data Intelligence Management a must-have for AI/ML</h3><p>Taking a data management approach to managing your results, or the value generated in your data, has several advantages that won’t be discussed here. Nevertheless, there are crucial limitations in this approach — it doesn’t work for AI/ML projects.</p><p>If you want to be smarter than your competition, you are probably already investing in AI/ML. These are the most promising Data Intelligence applications of today, yet you must find a way to manage their results.</p><p>The results of an AI/ML are called a model, such model is actually itself an artifact used to generate other results (prediction of churn, the recommendation of an article, …).</p><p>Such that the generation of those results is, by design, done without the intervention of a human (somewhat automatically) as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oqFsSHibyjlWr0zo" /><figcaption>AI requires Data Intelligence Management</figcaption></figure><p>To trust a model, therefore, you must trust its robustness which is highly dependent on what I call <strong><em>Datastrophes</em></strong> (more on this topic later in the series). This is why, in AI/ML, it is important to know when the model must be recreated.</p><p>Consequently, to do AI/ML in an efficient, sustainable, and scalable way Data Intelligence Management is a prerequisite.</p><p>What’s next?</p><p>Throughout this article, you have learned what data intelligence is and why it is a key element of your data management strategy.</p><p>In the coming weeks, I’ll cover how this approach will help your team in saving time and energy considering Datastrophes.</p><p><em>Interested in enhancing the Intelligence of your data? Don’t hesitate to reach </em><a href="https://www.linkedin.com/in/andypetrella/"><em>me</em></a><em> out to or on our website </em><a href="https://kensu.io/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=evangelization&amp;utm_content=dim"><em>Kensu.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeba1ec5a242" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>