<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ada Draginda on Medium]]></title>
        <description><![CDATA[Stories by Ada Draginda on Medium]]></description>
        <link>https://medium.com/@ada.draginda?source=rss-b162a61f31de------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UI7SJF22-4JRIcZIs-rKrw.png</url>
            <title>Stories by Ada Draginda on Medium</title>
            <link>https://medium.com/@ada.draginda?source=rss-b162a61f31de------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:57:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ada.draginda/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automating Propagations with DataHub and DataHub-Tools]]></title>
            <link>https://medium.com/datahub-project/automating-propagations-with-datahub-and-datahub-tools-4df1351d7565?source=rss-b162a61f31de------2</link>
            <guid isPermaLink="false">https://medium.com/p/4df1351d7565</guid>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[datahub]]></category>
            <dc:creator><![CDATA[Ada Draginda]]></dc:creator>
            <pubDate>Tue, 21 Mar 2023 15:49:44 GMT</pubDate>
            <atom:updated>2025-02-28T21:24:50.466Z</atom:updated>
            <content:encoded><![CDATA[<p>The propagation of metadata, tags, and owners outside of a DataHub dataset ingestion can be made easy.</p><p>At <a href="https://www.notion.so/">Notion</a>, data powers our business decision-making and our product itself. Therefore, it is essential for us to (a) understand what data in our warehouse is business critical and (b) ensure our critical data is built off of reliable sources. Without this understanding, we cannot effectively respond to incidents or know where to focus our reliability efforts. This would mean our customers and internal stakeholders would lose faith in our data.</p><p>We rely on <a href="https://www.acryldata.io/">Acryl Datahub</a> to gain these insights and ensure our critical data is reliable. Datahub is the #1 open source data catalog for data discovery and modern governance. Acryl’s SaaS product takes DataHub to the next level through automation and emphasis on time-to-value. In this post, we will discuss an integration we have built on top of DataHub to make sure our business-critical datasets are always properly labeled so we are able to respond to incidents with the most up to date information.</p><h3>The Problem</h3><p>At <a href="https://www.notion.so/">Notion</a>, we use a complicated formula that calculates who should own datasets and how these datasets should be triaged should their workflows fail. When we excitedly signed up with Acryl Data, we were faced with the challenge of migrating away from our manual processes of tagging and annotating our tables within DataHub. We were able to automate using DataHub’s <a href="https://datahubproject.io/docs/api/graphql/overview/">robust API</a>, which solves the problem of managing entities outside of their ingestion. However, we were met with a lot of boilerplate, so we wrote and open-sourced a <a href="https://github.com/makenotion/datahub-tools">small Python package</a> that wraps the API’s endpoints.</p><p>Let’s look at a simplified example of how we used the package to solve the problem…</p><p>Say we have some metadata boolean flags that describe how a dataset is used:</p><ul><li><em>is_exposed_externally</em></li><li><em>is_used_to_drive_company_metrics</em></li></ul><p>We want to use those flags to decide the right triage priority and owners for a given table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2btUkdLv5Ew4jE_tdzU2LQ.png" /></figure><ul><li>If <em>is_exposed_externally</em> is TRUE, then assign the highest priority, “P0” and assign the data engineering team as the technical owner</li><li>If <em>is_used_to_drive_company_metrics</em> is TRUE, then it’s still important, but a bit less critical, so “P1” and our best owner would be the data science team.</li><li>Otherwise, we’ll give the lowest priority P2, and not assign an owner.</li></ul><p>But we also need to consider any downstream tables that have a higher priority so the first step is to propagate metadata by inspecting the lineage. The best place to modify metadata is during ingestion, so we’ll need a <a href="https://datahubproject.io/docs/actions/guides/developing-a-transformer/">transformer</a> to do that work for us.</p><p>A full example of how to add a metadata transformer can be found as an example in the <a href="https://github.com/makenotion/datahub-tools/tree/main/examples/transforming">datahub-tools</a> repository, but the heart of the work is summarized below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*eOliCaKNNuNR_q0C1-zb0g.png" /><figcaption><a href="https://pages.acryl.io/slack">https://pages.acryl.io/slack</a></figcaption></figure><h3>Part 1: Metadata Propagation</h3><p>A transformer is a custom class that is called during the ingest of your data into DataHub.</p><pre>class MyPropertiesResolver(AddDatasetPropertiesResolverBase):<br><br>    def get_properties_to_add(self, entity_urn: str) -&gt; dict[str, str]:<br>        # returns a dictionary of metadata that should be included during<br>        # ingest for the givent entity_run<br>        return {}</pre><p>Steps that we will need to take to fill out this class:</p><ul><li>Get our resources (A resource is a table of data generated from SQL and is synonymous with a model or a DataHub dataset entity)</li><li>Propagate metadata from downstream resources</li><li>Provide a lookup that returns the metadata for a given DataHub entity URN.</li></ul><p>We will combine all of these steps into one function that we can call in the constructor of the class.</p><ol><li>Get Resources</li></ol><p>Our metadata are stored within our <a href="https://www.getdbt.com/">DBT</a> resources, so the first step is to fetch them all from a generated <em>manifest.json</em>. (For readers that are not familiar with DBT, the manifest contains all of the information on how to run the SQL for a given table, including all of its dependencies and metadata). Fortunately, the <a href="https://github.com/makenotion/datahub-tools/tree/main/examples/transforming">datahub-tools</a> package has a convenient function to do this work for us.</p><pre>from datahub_tools.dbt import extract_dbt_resources<br><br>resources_by_unique_id: dict[str, dict] = extract_dbt_resources(<br>    manifest_file=&#39;/path/to/your/manifest.json&#39;<br>)</pre><p>2. Propagate Metadata</p><p>Again, the <a href="https://github.com/makenotion/datahub-tools/tree/main/examples/transforming">datahub-tools</a> package can help significantly as it offers a convenience function that provides all downstream and upstream resources for a given resource.</p><pre>from datahub_tools.dbt import get_dbt_dependencies<br><br>dependencies: dict[str, ModelDependency] = get_dbt_dependencies(<br>    dbt_resources_by_unique_id=resources_by_unique_id<br>)</pre><p>We can now propagate the metadata along:</p><pre>for unique_id, resource in resources_by_unique_id.items():<br>    upstream_unique_ids: set[str] = dependencies[unique_id].get_all_upstream()<br>    # note that this is a custom metadata store that we added to our DBT<br>    resource_metadata = resource[&#39;config&#39;][&#39;notion_metadata&#39;]<br>    for upstream_unique_id in upstream_unique_ids:<br>        upstream_resource = resources_by_unique_id[upstream_unique_id]<br>        upstream_metadata = upstream_resource[&#39;config&#39;][&#39;notion_metadata&#39;]<br>        propagated_metadata = propagate(upstream_metadata, resource_metadata)<br>        upstream_resource[&#39;config&#39;][&#39;notion_metadata&#39;] = propagated_metadata</pre><p>Where the propagation logic is just combining the two dictionaries of booleans with or:</p><pre>propagate(left: dict[str, bool], right: dict[str, bool]) -&gt; dict[str, bool]:<br>    return {<br>        &#39;is_exposed_externally&#39;: <br>            left[&#39;is_exposed_externally&#39;] or right[&#39;is_exposed_externally&#39;],<br>        &#39;is_used_to_drive_company_metrics&#39;:<br>            left[&#39;is_used_to_drive_company_metrics&#39; <br>            or right[&#39;is_used_to_drive_company_metrics&#39;<br>    }</pre><p>This strategy of using or ensures that every resource has True set for a metadata entry if it or any of its downstream dependencies are set to True .</p><p>3. Map our Resources to DataHub Entities</p><p>We have our propagated metadata all within resources_by_unique_id but this maps a DBT unique id to the resource when DataHub uses the table name. (A DBT unique id is of the form<em> type.table_name</em> such as model.my_table whereas DataHub uses the table’s storage location, such as production.my_table ).</p><pre>metadata_by_table_name: dict[str, dict] = {<br>    x[&#39;name&#39;], x[&#39;config&#39;][&#39;notion_metadata&#39;] <br>    for x in resources_by_unique_id.values()<br>}</pre><p>Putting it all together into the Transformer class:</p><pre>import datahub.emitter.mce_builder as builder<br>from datahub.ingestion.transformer.add_dataset_properties import (<br>    AddDatasetPropertiesResolverBase,<br>)<br>from datahub_tools.dbt import (<br>    extract_dbt_resources, get_dbt_dependencies, ModelDependency,<br>)<br><br>class MyPropertiesResolver(AddDatasetPropertiesResolverBase):<br><br>    def __init__(self):<br>       super().__init__()<br>       resources_by_unique_id: dict[str, dict] = extract_dbt_resources(<br>           manifest_file=&#39;/path/to/your/manifest.json&#39;<br>       )<br>       dependencies: dict[str, ModelDependency] = get_dbt_dependencies(<br>           dbt_resources_by_unique_id=resources_by_unique_id<br>       )<br>       # propoagate metadata<br>       for unique_id, resource in resources_by_unique_id.items():<br>           upstream_unique_ids: set[str] = dependencies[unique_id].get_all_upstream()<br>           # note that this is a custom metadata store that we added to our DBT<br>           resource_metadata = resource[&#39;config&#39;][&#39;notion_metadata&#39;]<br>           for upstream_unique_id in upstream_unique_ids:<br>               upstream_resource = resources_by_unique_id[upstream_unique_id]<br>               upstream_metadata = upstream_resource[&#39;config&#39;][&#39;notion_metadata&#39;]<br>               propagated_metadata = propagate(upstream_metadata, resource_metadata)<br>               upstream_resource[&#39;config&#39;][&#39;notion_metadata&#39;] = propagated_metadata<br>    <br>       self.metadata_by_table_name: dict[str, dict] = {<br>           x[&#39;name&#39;], x[&#39;config&#39;][&#39;notion_metadata&#39;] <br>           for x in resources_by_unique_id.values()<br>       }<br><br>    def get_properties_to_add(self, entity_urn: str) -&gt; dict[str, str]:<br>        # returns a dictionary of metadata that should be included during<br>        # ingest for the givent entity_run<br>        dataset_key: DatasetKeyClass = builder.dataset_urn_to_key(entity_urn)<br>        return self.metadata_by_table_name.get(dataset_key, {})</pre><p>There is a bit of additional boilerplate required to add this transformer to your ingest, which can be found in the official <a href="https://datahubproject.io/docs/actions/guides/developing-a-transformer/">documentation</a>, but you can now transform your metadata entries during data ingest.</p><h3>Part 2: Tags and Owners</h3><p>Now that we have propagated and ingested metadata, we can calculate the appropriate owner and priority.</p><p>We first define how we will associate an owner and priority. In the interest of brevity, the calculation of the owner and priority were combined into one function (despite not being best practice to do so).</p><pre>import datahub.emitter.mce_builder as builder<br><br>def metadata_to_priority_and_owner(metadata: dict[str, str]) -&gt; tuple[str, str | None]:<br>   def _get_metadata(key: str) -&gt; bool | None:<br>       value = metadata.get(key)<br>       return value.lower().strip() == &quot;true&quot; if value else None<br><br>   is_exposed_externally = _get_metadata(&#39;is_exposed_externally&#39;)<br>   is_used_to_drive_company_metrics = _get_metadata(<br>       &#39;is_used_to_drive_company_metrics&#39;<br>   )<br><br>   if is_exposed_externally:<br>       priority_tag_urn = builder.make_tag_urn(&quot;P0&quot;)<br>       group_owner_urn = builder.make_group_urn(&quot;data_engineering&quot;)<br>   elif is_used_to_drive_company_metrics:<br>       priority_tag_urn = builder.make_tag_urn(&quot;P1&quot;)<br>       group_owner_urn = builder.make_group_urn(&quot;data_science&quot;)<br>   else:<br>       priority = builder.make_tag_urn(&quot;P2&quot;)<br>       group_owner_urn = None<br><br>   return priority_tag_urn, owner</pre><p>We now just have to fetch the metadata, pass it to the above function, and then set the owners and tags using the DataHub API. Fortunately, the <a href="https://github.com/makenotion/datahub-tools/tree/main/examples/transforming">datahub-tools</a> makes this process easier as well.</p><pre>from datahub_tools.client import get_datahub_entities, DHEntity<br><br>entities: list[DHEntity] = get_datahub_entities()</pre><p>The API allows batch setting/removing of tags/owners, and so we’ll group our changes as a list of entities that need each operation, reducing the overall number of calls that need to be made.</p><pre>import datahub.emitter.mce_builder as builder<br><br>add_tags: [str, list[DHEntity]] = {}<br>rem_tags: [str, list[DHEntity]] = {}<br>for dh_entity in entities:<br>   # priorities are P0, P1, or P2<br>   # owners are &quot;data_engineering&quot;, &quot;data_science&quot;, or None<br>   calculated_priority, calculated_owner = metadata_to_priority(dh_entity.metadata)<br>   existing_priority: str | None = dh_entity.get_priority()<br>   existing_owner: str | None = dh_entity.get_owner()<br><br>   # Remove priority if the new priority is None or is different<br>   # Add the priority if the existing priority is None or is different<br>   # (don&#39;t do anything if both are the same, including both being None)<br>   if calculated_priority != existing_priority:<br>       if existing_priority:<br>           existing_priority_urn = builder.make_tag_urn(existing_priority)<br>           entities = rem_tags.get(existing_priority_urn, [])<br>           entities.append(dh_entity)<br>           if existing_priority_urn not in rem_tags:<br>               rem_tags[existing_priority_urn] = entities<br><br>       if calculated_priority:<br>           new_priority_urn = builder.make_tag_urn(calculated_priority)<br>           entities = add_tags.get(new_priority_urn, [])<br>           entities.append(dh_entity)<br>           if new_priority_urn not in add_tags:<br>               add_tags[new_priority_urn] = entities<br><br>  # and similarly for owners<br>  ...</pre><p>Finally, we set/remove tags using functions from <a href="https://github.com/makenotion/datahub-tools/tree/main/examples/transforming">datahub-tools</a>.</p><pre>from datahub_tools.client import remove_tags, set_tags<br><br>for tag_urn, entities in rem_tags.items():<br>    remove_tags(tag_urns=tag_urn, resource_urns=[x.urn for x in entities])<br><br>for tag_urn, entities in add_tags.items():<br>    set_tags(tag_urns=tag_urn, resource_urns=[x.urn for x in entities])<br><br># and similaryly for owners, using the functions set_group_owner and <br># remove_owners from datahub_tools</pre><p>Above, we have shown how to inject metadata into our resources during the ingestion to DataHub. We also demonstrated how to pull that metadata back down, and then calculate then set owner and priority tags. To make all of it easier, there was frequent use of the open-source package <a href="https://github.com/makenotion/datahub-tools/tree/main/examples/transforming">datahub-tools</a>.</p><p>We hope you found this demonstration useful and best of luck in your data cataloging!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4df1351d7565" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datahub-project/automating-propagations-with-datahub-and-datahub-tools-4df1351d7565">Automating Propagations with DataHub and DataHub-Tools</a> was originally published in <a href="https://medium.com/datahub-project">DataHub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>