<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Esin Kirill on Medium]]></title>
        <description><![CDATA[Stories by Esin Kirill on Medium]]></description>
        <link>https://medium.com/@ya.esin.k?source=rss-7d5fb60726b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7rsGavooSXqjHi8yeWpkZQ.png</url>
            <title>Stories by Esin Kirill on Medium</title>
            <link>https://medium.com/@ya.esin.k?source=rss-7d5fb60726b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:15:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ya.esin.k/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How We Built Data Catalog]]></title>
            <link>https://medium.com/@ya.esin.k/how-we-built-data-catalog-38f25803a796?source=rss-7d5fb60726b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/38f25803a796</guid>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[mkdocs]]></category>
            <dc:creator><![CDATA[Esin Kirill]]></dc:creator>
            <pubDate>Wed, 18 Nov 2026 16:47:02 GMT</pubDate>
            <atom:updated>2026-11-18T16:47:02.632Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi everyone!<br>In this article, I want to share our experience of building and implementing a Data Catalog at our company.</p><p>Here is a quick overview. At the moment, it contains information about:</p><ul><li>700+ analytics events</li><li>1,000+ tables</li><li>100+ DAGs</li><li>50+ dashboards</li><li>around 100 other entities and pages</li></ul><p>If you are not particularly interested in the backstory, you can skip straight to <a href="#data-catalog">the result</a> or <a href="#how-it-works">how it works</a>.</p><h3>How We Got Here</h3><p>When an analytics system is small, it usually does not need a Data Catalog. Analysts remember where to find the right table, how a data mart is built, and what a particular event means.</p><p>But the system grows. New sources, tables, events, DAGs, and dashboards appear, along with an increasing number of connections between them. At some point, even the analysts start getting lost: everyone knows their own area well, but the big picture disappears. And then there are other departments and business teams asking what something is, where to find it, when it updates, and so on.</p><p>This is usually when the need emerges for a shared data “map” that reflects the current state of entities, their descriptions, and their dependencies.</p><p>Our starting point was:</p><ul><li>An in-house analytics event service with its own documentation</li><li>ClickHouse as the primary analytical database</li><li>Airflow as the orchestrator</li><li>Metabase as the main BI tool</li></ul><p>The analytics event service had — and still has — its own “documentation”: one large Markdown file generated automatically. As the number of events grew, the file eventually stopped rendering and opening properly. Anyone who wanted to read it had to open the source in GitLab or clone the project and view the file in an IDE.</p><p>Descriptions of tables, ETL processes, and dashboards lived in Confluence. Naturally, all of this was maintained manually and was not always up to date.<br>Sometimes people did not know that a data mart covering their use case already existed, so they built another one alongside it. Information had to be searched for across repositories, Confluence, chats, and other places.<br>In short, we realized it was time to build a Data Catalog.</p><h3>Looking for a Solution</h3><p>The first thing we did was look at the existing options. We discussed and compared many of them:</p><ul><li>DataHub</li><li>OpenMetadata</li><li>Amundsen</li><li>and others</li></ul><p>This article is not a comparative review, so I will keep it brief. Some solutions were full-fledged data platforms with their own rules and constraints. Others would have required us to rework our internal tools and processes. Still others were expensive and, in my opinion, overloaded with features.<br>Some did not support ClickHouse or could not retrieve documentation from the services we already had.</p><p>At some point, I came across one article explaining how to build a documentation site with MkDocs step by step and publish it through CI. MkDocs builds a site from files in a public directory, while one of its popular themes, Material, provides site-wide search, navigation, tabs, and other useful features such as a dark theme. You can also add custom JavaScript, macros, and styles when needed. We decided to use this technology.</p><h3>Data Catalog</h3><p>First, I would like to show what we ended up with, and then explain how everything works under the hood.</p><h3>What It Looks Like</h3><p>Disclaimer</p><blockquote><em>All names and data related to tables, events, DAGs, services, dashboards, and other entities, as well as the catalog’s appearance, have been deliberately changed to avoid disclosing internal company information.</em></blockquote><p>The catalog’s home page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3f_OpY5VQCZBaA_Ec1E1tg.jpeg" /></figure><p>At the top, we added tabs for different entity types: metrics, analytics events, tables, Metabase dashboards, and Data Governance. Each tab is a separate section with its own navigation tree on the left. This means you can switch between events and tables with a single click instead of searching for the right section in a shared menu. The tabs are sticky and remain visible while scrolling through a long entity page. In Material, this can be enabled with two lines in mkdocs.yml: navigation.tabs and navigation.tabs.sticky.</p><p>We also added a metric glossary search to the home page. It is a custom JavaScript feature requested by the business team. It searches not only by name but also by synonyms, and the results display the metric type and a short definition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*svizRXHVCmHs1WFamPeWmg.jpeg" /></figure><p>Material’s site-wide search works separately. Its results include metrics, events, tables, DAGs, dashboards, and Data Governance pages. To avoid looking for a table among events and dashboards, we added filter chips for entity types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yok9qpx1HocwvoASRUq8lA.jpeg" /></figure><h3>Tables</h3><p>For tables and databases, each database has its own page listing the available tables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lVlMlEUWmk8VGemfVwqZmQ.jpeg" /></figure><p>Each table page displays its status, a description of the table and its fields, and other properties. We also show information about the ingestion process, its schedule, sources, DDL, change history, and lineage. We made the lineage graph interactive, so clicking any table opens its page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*agmPr5BDTt-xgqljiieMEw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G1PQO_qP8GDMRAYwjIUgdA.jpeg" /></figure><h3>Events</h3><p>Events follow a similar structure and are grouped by platform. Each event page shows its status, description, parameters, and parameter types. We also collect a change history for every event:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mN3XFT110mBbOFKPsQR82A.jpeg" /></figure><h3>Dashboards</h3><p>We did not stop at tables and events. We added dashboard pages as well. Material offers plenty of options for building a convenient interface. For example, you can add a button that opens the relevant resource directly, which is what we did here. The result is quite convenient and interactive:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6CqM3XQNmPisE01gf5kM9g.jpeg" /></figure><p>We also added pages for data sources, DAGs, and a general overview of the analytics platform and its services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y5odgAkQEzEGBTRVBF8beA.jpeg" /></figure><h3>How It Works</h3><p>The overall process looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vM8r_QZzmbJ5LPJbqmU2cA.png" /></figure><p>Let me start with the catalog itself and then look at where its data comes from.</p><h3>Data Catalog Repository</h3><p>We started with the repository structure. Raw metadata is stored separately from the site pages: metadata/ contains YAML and JSON entity files, with a separate directory for each entity type. We named the public directory from which MkDocs builds the site catalog/; this is where the finished Markdown pages go. We also created directories for future use: one for HTML page macros and another for scripts that convert metadata into pages.</p><p>Repository structure:</p><pre>metadata/          Entity YAML and JSON files<br>  tables/          Tables<br>  events/          Events<br>  lineage/         Graph and sources<br>  metrics/         Metrics, maintained manually<br>macros/            Macros for HTML pages<br>scripts/           YAML/JSON → Markdown and other transformations<br>catalog/           Finished pages used to build the catalog site<br>  assets/          CSS and JavaScript, images, and other assets<br>  tables/<br>    .pages         Page navigation is defined here<br>    index.md       Root file of the tables section<br>    raw.md         File for the raw database<br>    ...<br>  events/<br>    .pages<br>    index.md<br>    web.md<br>    ...<br>  metrics/<br>    .pages<br>    index.md<br>    monetary.md<br>    ...<br>mkdocs.yml         Main MkDocs configuration file</pre><h3>Configuring MkDocs</h3><p>The repository’s main file is mkdocs.yml. Its configuration is very straightforward. We only needed to specify the basic settings—the site name, author, and GitLab link, although even that is optional—define the public directory, configure the theme, and enable additional features:</p><pre>############<br>### MkDocs Configuration Fle<br>### docs: https://www.mkdocs.org/user-guide/configuration/<br>############<br><br>############<br>### Basic settings<br>############<br>site_name: Data Catalog<br>site_description: Demo Data Catalog with fictional data<br>site_author: Demo<br>copyright: Data Catalog<br>############<br>### Public directory<br>############<br>docs_dir: catalog<br>############<br>### Theme configuration<br>############<br>theme:<br>  name: material<br>  logo: assets/images/logo.svg<br>  favicon: assets/images/logo.svg<br>  features:<br>    - navigation.tabs<br>    - navigation.tabs.sticky<br>    ...<br>############<br># Additional CSS files<br>############<br>extra_css:<br>  - assets/css/extra.css<br>############<br># Additional JavaScript files<br>############<br>extra_javascript:<br>  - assets/javascripts/tableSort.js<br>  - assets/javascripts/searchFilter.js<br>############<br># MkDocs plugins<br>############<br>plugins:<br>  - search<br>  - macros<br>  - ...<br>############<br># Markdown extensions<br>############<br>markdown_extensions:<br>  - attr_list<br>  - def_list<br>  - admonition<br>  - md_in_html<br>  - ...</pre><p>From there, it all comes down to a couple of commands. Put at least an index.md file in the public directory—catalog/ in our case—and run:</p><pre>pip install mkdocs-material   # MkDocs is installed together with the theme<br>mkdocs serve                  # Local server at http://127.0.0.1:8000; pages are rebuilt on the fly<br>mkdocs build                  # Static site in the site/ directory, which is what we publish</pre><p>That is it: the first page of your site is ready. The next question is how to collect and store the metadata.</p><h3>Metadata</h3><p>First, we defined the entities we wanted to see in the Data Catalog:</p><ul><li>tables</li><li>analytics events</li><li>lineage</li></ul><p>Initially, we did not know what format they should take in the catalog. We tried putting them directly into the Data Catalog repository and performing the required transformations there, but quickly realized that this was not the best approach. So, like true analysts, we decided to collect everything in a database.</p><h3>Tables</h3><p>We collected ClickHouse tables and columns from system.tables and system.columns. Table and column descriptions can be stored as comments, but most analysts are more accustomed to documenting them in files rather than when creating a table. We are gradually moving toward comments.</p><h3>Analytics Events</h3><p>Next, we moved on to the event service. We made a small improvement: event documentation was now generated not only as an MD file — which no longer rendered in the UI — but also as JSON. All that remained was to retrieve this JSON, parse it, and distribute the data among the appropriate tables.</p><h3>Lineage</h3><p>Collecting lineage required a few minor changes to the Airflow repository. First, we created lightweight helpers to establish a consistent format:</p><pre>from airflow.datasets import Dataset<br><br>def clickhouse_table(conn_id: str, database: str, table: str) -&gt; Dataset:<br>    return Dataset(f&quot;clickhouse://{conn_id}/{database}/{table}&quot;)<br><br>def mongo_collection(conn_id: str, collection: str) -&gt; Dataset:<br>    return Dataset(f&quot;mongo://{conn_id}/{collection}&quot;)<br><br>def external_source(source_type: str, name: str) -&gt; Dataset:<br>    &quot;&quot;&quot;An arbitrary external source: API, SFTP, S3, Google Sheets, etc.&quot;&quot;&quot;<br>    return Dataset(f&quot;{source_type}://{name}&quot;)</pre><p>We then used inlets and outlets in tasks to define the relationships between sources and consumers:</p><pre>load_order_items = MongoToClickHouseOperator(<br>    ...<br>    inlets=[mongo_collection(&quot;shop-db&quot;, &quot;order_items&quot;)],<br>    outlets=[clickhouse_table(&quot;clickhouse&quot;, &quot;raw&quot;, &quot;order_items&quot;)],<br>)</pre><p>After that, we extracted the DAGs themselves, their tasks, and their relationships from DagBag:</p><pre>from airflow.models import DagBag<br><br>dagbag = DagBag(dag_folder=dags_path, include_examples=False)<br><br>for dag_id, dag in dagbag.dags.items():<br>    ...</pre><p>This gave us graphs with nodes and the processes connecting them.</p><p>In addition to DAGs and their tasks, we also had dependencies in views and materialized views. We handled them by parsing the definitions with regular expressions. The output was a set of graphs in the same format as the Airflow graphs, except that the views themselves acted as the “tasks.”</p><h3>Reference Tables and Changelog</h3><p>We stored metadata snapshots in raw tables using SCD Type 2, allowing us to inspect their state as of any given date. We then started building reference tables containing the current state. We do not delete anything from these reference tables as part of the standard process; instead, we mark entities as outdated or deleted.</p><p>We also began collecting a changelog for each entity in order to:</p><ol><li>retain its history and display it in the catalog — finding out when a particular parameter or column was added used to be a real pain;</li><li>update only the part of the catalog metadata that had changed instead of refreshing everything.</li></ol><h3>Moving Data into the Data Catalog</h3><p>At this point, we had to choose between two approaches. People edit the descriptions of some entities directly in the catalog, including table and column descriptions. For other entities, such as events and dashboards, descriptions already live in their source systems: the event service and Metabase. This resulted in two separate flows.</p><p>Events, dashboards, and lineage do not need a format suitable for manual editing because nobody edits them in the catalog. We put the incoming delta into metadata/ as JSON, rebuild the pages, and that is it.</p><p>Tables are different. People are more comfortable working with files, so we convert the table delta to YAML and store each table in its own file. For example, the metadata for raw.order_items is stored in metadata/tables/raw/order_items.yml:</p><pre>table:<br>  name: order_items<br>  database: raw<br>  object_type: table<br>  is_deleted: false<br>  is_actual: true<br>  deployment: demo-cluster<br>  description: &#39;Order line items: which products and quantities each order contains.&#39;<br>  owners:<br>  - name: demo.analyst<br>    role: analytics<br>columns:<br>- name: order_id<br>  type: String<br>  description: Order<br>  is_in_primary_key: true<br>- name: event_date<br>  type: Date<br>  description: Order date<br>  is_in_primary_key: true<br>- name: product_id<br>  type: String<br>  description: Product<br>  is_in_primary_key: true<br>- name: qty<br>  type: UInt32<br>  description: Quantity<br>  is_in_primary_key: false<br>- name: price<br>  type: Decimal(12, 2)<br>  description: Unit price at the time of the order<br>  is_in_primary_key: false<br>ingestion:<br>- type: dag<br>  name: raw_order_items_loader<br>  schedule: 0 4 * * *<br>  description: Order line items<br>sources:<br>- name: shop-db.order_items<br>  type: mongo<br>storage:<br>  engine: MergeTree<br>  partition_key: toYYYYMM(event_date)<br>  sorting_key: event_date, order_id, product_id<br>  primary_key: order_id, event_date, product_id<br>  sampling_key: &#39;&#39;<br>create_table_query: &quot;CREATE TABLE raw.order_items\n(\n    order_id String,\n    event_date\<br>  \ Date,\n    product_id String,\n    qty UInt32,\n    price Decimal(12, 2)\n)\n\<br>  ENGINE = MergeTree\nPARTITION BY toYYYYMM(event_date)\nORDER BY (event_date, order_id,\<br>  \ product_id)&quot;<br>changelog:<br>- date: 12.01.2026<br>  description: Table added to the demo catalog<br>  author: catalog-bot<br>- date: 02.03.2026<br>  description: Column descriptions clarified<br>  author: demo.analyst</pre><p>Why YAML? First, YAML and JSON are easy to convert into one another: the delta arrives from Airflow as JSON, and converting it to YAML is straightforward. Second, YAML is easier to read and more convenient for people to work with.</p><h3>Bidirectional Synchronization</h3><p>It is ideal when an entity’s complete description is stored in its source system: the catalog then only needs to display it. This is the case for events and dashboards, but not for tables. ClickHouse supports comments on tables and columns, but our analysts are not yet used to documenting tables in comments. For now, we therefore decided to describe tables in catalog files.</p><p>As a result, table metadata is updated in two places. People add descriptions in the Data Catalog, while the table itself changes in ClickHouse: columns appear, types and keys change, and the DDL evolves. To prevent one side from overwriting the other, we needed bidirectional synchronization. In addition to descriptions, we also added several manually maintained fields that make tables easier to navigate.</p><p>The resulting process is:</p><ol><li>First, we retrieve the latest description changes made by people in the Data Catalog.</li><li>Next, we collect metadata from the source systems: ClickHouse, Airflow, Metabase, and the analytics event service.</li><li>We merge everything, process it, and save it to the database.</li><li>We send a delta back to the Data Catalog containing the complete picture: both the latest schema and the manually maintained descriptions.</li></ol><p>We set priorities for table descriptions: use the description from the Data Catalog first; if none exists, use the ClickHouse comment.</p><p>We implemented this only for tables. All other entities are documented in their source systems: there are far fewer of them, and maintaining their descriptions there is easier.</p><h3>Vibe Coding</h3><p>We used that too. We wanted proper headings, collapsible blocks for DDL and long changelogs, badges, and links between pages. We wrote the scripts that render Markdown from YAML and JSON with the help of AI. We also used it to build the Quick Metric Search and several source-file validations.</p><h3>Instead of a Conclusion</h3><p>What we ended up with was more than just a documentation site: it became a working tool used by several departments. For analysts and business teams, the catalog became a single entry point for data. They can find the right table, understand an event, open a dashboard, and trace relationships between entities.</p><p>Based on our observations, site-wide search is the most popular feature. People do not browse the section tree; they search for the entity they need right away. Among the sections themselves, tables and Metabase dashboards are opened most often.</p><p>The practical benefits also became apparent quite quickly. When one of the database servers failed, we restored the lost objects without digging through old chats. When investigating data errors, we could see which entities they had propagated to. And when deciding whether another table was necessary, we could inspect its relationships instead of polling the entire team.</p><p>We started getting fewer questions like:</p><ul><li>Which table contains the current order statuses?</li><li>What does this event, column, or table mean?</li><li>When is this table updated?</li><li>And so on.</li></ul><p>However, a catalog does not create documentation or bring order to data on its own. New objects, columns, and relationships are imported automatically, but meaningful descriptions still have to be written by people. Maintaining their quality and keeping them up to date is an ongoing effort.</p><p>Our main takeaway is simple: a useful Data Catalog does not have to be a heavyweight platform. Sometimes it is enough to integrate metadata collection into existing processes and give people one clear place to search. We continue to develop the tool by adding new entities and relationships.</p><p>If you have followed a similar path — building your own catalog or implementing an off-the-shelf solution — I would be interested to compare experiences in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38f25803a796" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>