<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fadi Maali on Medium]]></title>
        <description><![CDATA[Stories by Fadi Maali on Medium]]></description>
        <link>https://medium.com/@fsheer?source=rss-b7bf245b68cb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*p7N0fLMLmwxkuBEV.jpg</url>
            <title>Stories by Fadi Maali on Medium</title>
            <link>https://medium.com/@fsheer?source=rss-b7bf245b68cb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:43:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fsheer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Catalogs, the Luxury of Choice]]></title>
            <link>https://zendesk.engineering/data-catalogs-the-luxury-of-choice-c161cd91e713?source=rss-b7bf245b68cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/c161cd91e713</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[dataops]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-operations]]></category>
            <category><![CDATA[data-catalog]]></category>
            <dc:creator><![CDATA[Fadi Maali]]></dc:creator>
            <pubDate>Mon, 31 Aug 2020 04:07:36 GMT</pubDate>
            <atom:updated>2020-08-31T04:07:36.412Z</atom:updated>
            <content:encoded><![CDATA[<p>The growing use of data across all organisations stresses the importance of making the available data easy to find, and use. Over the last few years, this resulted in a proliferation of tools to build data catalogs. Data catalogs collect metadata about existing datasets and help users easily find, understand, and trust their organisation’s data.</p><p>Many companies built their own data catalog software, and made them available; examples include <a href="https://engineering.linkedin.com/blog/2019/data-hub">LinkedIn’s DataHub</a>, <a href="https://marquezproject.github.io/marquez/">WeWork’s Marquez</a>, <a href="https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9">Lyft’s Amundsen</a>, <a href="https://medium.com/airbnb-engineering/democratizing-data-at-airbnb-852d76c51770">Airbnb’s Data Portal</a>, <a href="https://cloud.google.com/data-catalog">Google’s Data Catalog</a>, <a href="https://eng.uber.com/databook/">Uber’s Databook</a>, <a href="https://netflixtechblog.com/metacat-making-big-data-discoverable-and-meaningful-at-netflix-56fb36a53520">Netflix’s Metacat</a>, and <a href="https://atlas.apache.org/#/">Hortonworks’s Atlas</a> (an <a href="https://www.apache.org/">Apache</a> project now). While this emphasis on organising metadata is very welcome, the large number of options might make it hard to choose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1iAZCtlmJfP1ZEjNsnTtZA.jpeg" /></figure><p>Zendesk’s Enterprise Data and Analytics team’s journey with data catalogs began in late 2018. In this post, I report on the process we went through in choosing a tool to build Zendesk’s internal data catalog. It should be noted that the goal here is by no means a comparison of those continuously evolving tools, but rather sharing our thoughts (circa 2018), and reflecting on our decisions.</p><h3>The Case for Data Catalogs</h3><p>While the term “data catalog” might be overloaded, the use in this post refers to an organised collection of metadata that aims to facilitate discovery, understanding, and usage of an organisation’s datasets (see <a href="https://medium.com/ataccama/essential-features-of-data-catalogs-22bcdcf0d106">this post</a> for a list of functionalities and features of data catalogs).</p><p>Data catalogs are mainly used in the context of data warehousing. They became increasingly relevant as data warehouses transitioned from (expensive, dearly-protected, meticulously-managed) monoliths to an suite of tools managing data flows in multiple formats from various sources via different routes (aka “<a href="https://en.wikipedia.org/wiki/Data_lake">data lakes</a>” or “<a href="https://dsf.berkeley.edu/jmh/papers/madskills-032009.pdf">MAD data warehouses</a>”).</p><p>In comparison to traditional metadata catalogs (think <a href="https://dev.mysql.com/doc/refman/8.0/en/information-schema.html">MySQL INFORMATION_SCHEMA</a>), the core values of data catalogs are in:</p><ul><li>Not being tied to a single system or data source.</li><li>Having human consumption as a first class focus: data catalogs do not focus on capturing information essential for operations such as schemas and physical locations, instead focussing on functionality such as keyword search, dataset ranking, lineage search, and visualisation.</li></ul><h3>The Choice</h3><p>Our journey with data catalogs in the Enterprise Data and Analytics team in Zendesk started late 2018. The main goal will probably sound familiar:<br><strong><em>To build a central catalog of metadata about all datasets in Zendesk. Users can utilise the catalog to find, understand, and monitor datasets.</em></strong></p><p>For each dataset, regardless of where it lives, we wanted to track:</p><ul><li>Core metadata such as name, description, creation date, and ownership.</li><li>Labels and categories.</li><li>Schema (for structured datasets).</li><li>History of schema changes.</li><li>Lineage and relationships to other datasets (upstream and downstream datasets).</li><li>Related resources (Wiki pages, example scripts, example queries).</li></ul><p>Our goal was also to collect metadata and lineage at the column level. This is challenging given that most existing tools focus on a broader dataset-level metadata and lineage.</p><p>In addition to support dataset discovery, we also wanted to use the catalog to support data operations. Example use cases we had in mind here are:</p><ul><li>Given a failing/delayed data pipeline, assess the effect on downstream SLA-bound products.</li><li>Given a data quality issues, facilitate tracing the issue back to its source (going upstream) and assessing impact on other datasets (going downstream).</li><li>When deprecating a column or a dataset, identify teams that use that column/dataset and need to be notified.</li><li>Given a dataset, show available related metrics about data quality, freshness, and other aspects of the dataset.</li></ul><p>When looking at available tools for data catalogs, we identified additional selection criteria:</p><ul><li>A flexible data model that allows user-defined types. We wanted to explicitly define our data products (dashboards, reports, important datasets), represent lineage, and support columns as first-class entities.</li><li>A graph querying ability; essential for querying lineage relationships.</li><li>Support for versions; essential for capturing history and precisely describe datasets relationships.</li></ul><p>We ultimately selected <a href="https://atlas.apache.org/#/">Apache Atlas</a>:</p><ul><li>It has a flexible and well-designed <a href="https://atlas.apache.org/#/TypeSystem">data model</a> that supports subtypes and has an underlying graph database.</li><li>It exposes a GUI and a <a href="http://atlas.apache.org/#/SearchAdvance">structured query API</a>.</li></ul><h3>Current status</h3><p>Atlas’ flexible data model serves us well. We’ve defined our own data model to represent BigQuery tables and columns, Airflow DAGs, view definitions, Tableau dashboards, and even Google Sheets. Atlas doesn’t support versioning, but we were able to do so in our own data model.</p><p>We collect data periodically via scheduled Kubernetes Jobs. Currently we collect:</p><ol><li>Metadata and schema information of tables and columns on BigQuery.</li><li>Views lineage obtained by parsing SQL definition of views (we used <a href="https://github.com/pyparsing">PyParsing</a> and <a href="https://github.com/pyparsing/pyparsing/blob/master/examples/bigquery_view_parser.py">contributed the parser </a>that was written by Michael Smedberg).</li><li><a href="https://airflow.apache.org/">Airflow</a> tasks lineage to capture input/output of all Airflow tasks that run on BigQuery.</li></ol><p>However, <a href="http://atlas.apache.org/#/SearchAdvance">the querying capability of Atlas</a> proved very limited both in terms of performance and expressivity. We ended up getting a daily export of the lineage data in <a href="https://www.w3.org/TR/rdf11-concepts/">RDF format</a> and serving lineage queries using <a href="https://www.w3.org/TR/sparql11-query/">SPARQL 1.1</a> on top of the RDF graph. We use the <a href="https://rdflib.readthedocs.io/en/stable/">rdflib Python library</a>.</p><p>We decided to make the catalog a read-only medium for users. The catalog a reflects captured metadata using established processes and tools. This eliminates the risk of conflicting with existing processes, and of metadata drift across systems.</p><p>The data catalog is used frequently to support our common data operations work. When a data pipeline fails, lineage information can quickly tell us wether this pipeline is essential for some important downstream data product. This helps us organise our response (decide for example if it is necessary to wake relevant people up in the middle of the night). Lineage information also proved essential when deprecating datasets.</p><h3>Time to Reflect</h3><p>The data catalog has been available to analysts, and the data operation team in Zendesk for about a year. Looking back, here are three main ideas that spring to mind…</p><p><strong><em>The generic data model:</em></strong></p><p>While the generic flexible data model was essential for our needs, <a href="https://en.wikipedia.org/wiki/There_ain%27t_no_such_thing_as_a_free_lunch">there is no free lunch</a>. This flexibility had at least two costs:</p><ul><li>Generic UI: Atlas provides a nice GUI with ability to<a href="https://atlas.apache.org/#/SearchBasic"> build structured query using the GUI</a>. However, since the interface is built for generic entity types, it is hard to use without good knowledge of the underlying data model. Tasks such as searching for tables that have specific columns are very hard to achieve.</li><li>Complicated code for common tasks. For example, checking if a dataset definition has changed required looking at its basic information, its list of columns, and its partitioning definition. Since all these are user-defined types, we had to write our own logic.</li></ul><p>Interestingly, most existing data catalog systems don’t support user defined types. Google catalog supports <a href="https://cloud.google.com/data-catalog/docs/concepts/overview#tags">structured tags</a> and recently <a href="https://cloud.google.com/data-catalog/docs/how-to/custom-entries">added limited support for user defined types</a>.</p><p><strong><em>Adoption</em></strong>:</p><p>As mentioned before, it was straightforward and extremely useful to fit the catalog in data operations workflows. However, we haven’t (yet!) had such success with other use cases (i.e. facilitating discovery and usage of data). Analysts and data users have their own workflows and tools. At the moment, the catalog is a separate tool that is consulted occasionally, but is not part of a regular workflow. The limited UI contributes to this. To encourage usage, we are investigating mirroring this metadata on Google Data Catalog so that it is closer to the actual datasets.</p><p><strong><em>A read-only catalog:</em></strong></p><p>As stated before, we decided to make the catalog a read-only medium for users. Users who want to update metadata have to do that in the source system and not in the catalog itself. This eliminated the risk of conflicting with existing processes, and of metadata drift across systems. However, this arguably limited the role the catalog could play in improving the quality of metadata, and in improving existing practices and processes.</p><p>Choosing a tool for building a data catalog is an important decision… The good news is that there are plenty of good options. The goals and importance of data catalogs are easy to explain and to resonate with people. However, defining the role of the catalog and how it should fit within people’s workflows is the more challenging decision that needs to be addressed early on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c161cd91e713" width="1" height="1" alt=""><hr><p><a href="https://zendesk.engineering/data-catalogs-the-luxury-of-choice-c161cd91e713">Data Catalogs, the Luxury of Choice</a> was originally published in <a href="https://zendesk.engineering/">Zendesk Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t use graphs for equivalence relations]]></title>
            <link>https://medium.com/@fsheer/dont-use-graphs-for-equivalence-relations-639696aaf049?source=rss-b7bf245b68cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/639696aaf049</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[data-structures]]></category>
            <dc:creator><![CDATA[Fadi Maali]]></dc:creator>
            <pubDate>Sun, 17 Sep 2017 20:27:39 GMT</pubDate>
            <atom:updated>2017-09-17T20:32:00.073Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Long story short…</em></h4><p>Don’t use <a href="https://en.wikipedia.org/wiki/Graph_(abstract_data_type)">graphs</a> to represent <a href="https://en.wikipedia.org/wiki/Equivalence_relation">equivalence relations</a>. Use <a href="https://en.wikipedia.org/wiki/Disjoint-set_data_structure">Disjoint-set data structure</a>. Using graphs is not wrong, but it is like <a href="http://www.phrases.org.uk/meanings/sledgehammer-to-crack-a-nut.html">using a sledgehammer to crack a nut</a>.</p><h4><em>Long story long…</em></h4><h4>Graphs are not good for equivalence relations.</h4><p>Equivalence relations are reflexive, symmetric, and transitive. Consequently, they end up partitioning the instances space into disjoint sets, known as equivalence classes. If this is represented as a graph, whenever you have an edge between <em>u</em> and <em>v</em>, an edge between <em>v</em> and <em>w</em>, there will be an edge between <em>u</em> and <em>w</em> (due to transitivity). The graph becomes a set of disconnected <a href="https://en.wikipedia.org/wiki/Complete_graph">complete graphs</a>. Graph navigation, the main advantage to using graphs, doesn’t make sense anymore. Any two nodes are either directly connected (i.e., they are part of the same equivalence class) or they are unreachable to each other (i.e., they are in different equivalence classes).</p><p><a href="https://en.wikipedia.org/wiki/Disjoint-set_data_structure">Disjoint-set data structure</a> gives time and space efficient implementation for finding if two instances are equivalent and for merging two equivalent sets. This makes it ideal for both building equivalence classes and exposing (using) them. I guess you might need an extra hashing data structure for quick access to an instance given some property (such as ID).</p><h4>Where did it come from?</h4><ul><li>I was giving an introduction lecture on graphs, I used this example to motivate the wide applicability of graphs… I went something like this:</li></ul><blockquote>If you have a set of countries, you can represent each country as a node and link each two countries if they share a border. The result is a graph. Relationships don’t have to be physical ones, you can link any two countries that speak the same language.</blockquote><p>The second example looked weird when visualised and somehow felt wrong.</p><ul><li>I was attending a talk at work about cross-device graphs… same story.</li><li>I did a search for on <a href="https://www.google.ie/search?q=&quot;cross-device+graph&quot;">Google for “cross-device graph”</a>… got about 5k results. Probably some of them might indeed need a graph, but my guess is that most don’t.</li></ul><p>The problem sounds common, I wrote this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=639696aaf049" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>