Summary

September highlights from the Amundsen community:

• Integrating Slack with Amundsen at Convoy
• Adding stats at a table level
• Improved support for showing nested columns
• Additional metadata for BigQuery
• Search revamp
• Extending badges browsing capabilities

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

Integrating Slack with Amundsen at Convoy

This past summer, George Pu joined Convoy’s Data Platform team as an intern to help surface common questions about data usage within Slack through Amundsen. The result? A usable and robust Slack integration. Learn more about the process he took and the team’s next steps: Integrating Slack with Amundsen for Ease of Data Discovery

Adding stats at a table level

Up until now, we have allowed adding stats (min, max, count distinct) on a column level. Moving forward, we now additionally allow adding stats on a table level, allowing us to share stats like “daily active users” or “query count” on a table basis.

See PR details for more info! Huge props to Allison Suarez Miranda!

Improved support for showing nested columns

We now support showing nested columns as individual rows in the list of columns instead of showing them in a modal.

See PR details for more info. Thanks to Daniel Won!

Additional metadata for BigQuery

We now show the column’s mode for BigQuery tables.

See PR details for more info. Thanks to Marwan Baghdad!

Search revamp

We’re looking to revamp search with a cleaner backend implementation, more robust queries, better customizability, and make use of modern v7 ES features.

This revamp would include deprecating Apache Atlas as a search backend. If you run Amundsen without Elasticsearch and instead rely purely on Atlas for search AND migrating to Elasticsearch would be prohibitive, please reach out to Dmitriy Kunitskiy. The goal is to improve the Amundsen search/ranking experience and remove a limited and likely-unused option of using Atlas in place of ES.

Check out the RFC for more info and let us know what you think!

Extending badges browsing capabilities — badges on landing page and browse view

We’re looking to introduce a “Popular/All badges” section below “Popular tags” on the landing page, as well as a browse section.

Check out the RFC for more info and let us know what you think!

Announcements

📣 No community meeting updates from last month due to my vacation. I took a week off and went to Orcas Island for a short trip before Labor Day!

Coming up next…

Next community meeting

• Date: Thursday, November 4, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/sahdiasw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen Monthly Update — September 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

Migrations are one of the most common initiatives that data teams own. But as everyone knows, they are complex beasts. They always take longer than they need to, many different teams are involved, and they’re super visible — not just to users, but execs, too. Everyone is anxious when a migration is underway. And, there’s always one ongoing or about to start.

Over my career, I have been a data engineer myself and have worked with hundreds of data engineers. One thing I’ve learned which correlates with the quality of data engineering teams is how well they do migrations.

I’ve found that how a data engineering team handles migrations can be a good proxy for the quality of a data engineering team. I’ll dive into the 3 crucial steps that I have seen great data engineering teams take to ensure a successful migration:

1. Determine the complexity and order of migration
2. Migrate and test level-1 tables
3. Provide tools for downstream owners and kick-off ripple migration

Each of these steps could be a blog post of its own, but for the sake of coherence I have tried to tackle them in one post.

Migrations are usually based on something that’s happening organizationally or externally to the company that’s pushing them to migrate. Migrations are exhaustive because success is tied to killing something rather than simply just moving something. In most migrations, getting 90% done isn’t good enough, even getting 100% done is not good enough, you really have to kill something old in order for a migration to be successful.

Here are some common reasons why data teams migrate, along with what success looks like in each of these cases:

1. Wholesale migration from one data warehouse to another, usually to reduce costs, consolidate, increase scalability or reliability. Success: no activity on the previous warehouse.
2. Migrate from one source of data to another, due to a decision made “upstream”. A common example is a company moving from Hubspot to Salesforce as their CRM and all the downstream sales analytics pipelines, dashboards and metrics need to be updated due to the change. Another common example is “decomp” or decomposition of upstream services from a monolith to a number of microservices. These microservices end up having their own databases, which leads to new decomposed sources of data that data warehouse ETL needs to be rebuilt on, and of course. Success: zero queries on old data.

It turns out that it’s much harder to kill something than to stand up something new. And, this is precisely why migrations are so hard and complex.

These are the 3 steps that I have seen great data engineering teams take when it comes to migrations.

Step 1 — Determine the complexity and order of migration

Complexity of migration

The first step to any migration is understanding how complex it is. Is it a 2 week, 2 month, or 2 year migration? The closest proxy to complexity of a migration is determined by two factors for the data source in question:

1. How many levels of data assets do you have?
2. How many data assets overall are built on the data source?

A data asset can be a table, dashboard, etc.

To look at it another way, if you had a lineage graph like the one below, these factors are:

1. What’s the depth of the graph?
2. How many nodes are there in the graph?

Order of migration

Great data engineering teams are thoughtful and intentional about the order in which they choose to migrate. In a lineage graph like above, you still have to choose which “branch” to migrate first.

When determining the order of which branch to choose first, here are the three most common categories that great teams think about:

1. Cost — we’re going to migrate the most costliest data first.
2. Usage Impact — we’re going to migrate the most used data sets last in order to reduce user impact.
3. Risk — we’re going to migrate the least riskiest data first. For example, let’s save core financial metrics to be migrated last since they get reported externally.

Migrations are easy, as long as you don’t have to worry about dependencies.

This brings me to the hardest part of migration. Once you have picked which branch to migrate, you have to determine how the existing data is being used.

Level-1 usage means that there are only leaf level consumers of the data like dashboards, ad hoc queries on the tables. Level-2 usage means that there are other tables built on the table that have dashboards and ad hoc queries being run on them.

An automated data catalog that captures lineage of your data can help you gain a good understanding of your levels of usage.

The data catalog that surfaces this lineage graph should be able to automatically capture these 3 kinds of relationships at a minimum:

1. Table level lineage
2. Dashboards/reports built on the tables
3. Names of users and number of ad hoc queries run by them, over the last 30 days

Nice to have (not must have):

• Column level lineage — this is hard to capture at a high level of granularity and accuracy and not needed for many migrations.
• More information about what process/ETL was used to generate the downstream tables.

There are other use-cases for column level lineage, like meeting regulatory needs or deprecating a column. However, for the larger migrations as described at the beginning of this post, I find that table level lineage is sufficient.

Step 2 — Migrate & test level-1 tables

In most organizations, you are going to have more than one level of usage. The first step is always to migrate these level-1 tables first. In some organizations, no real ETL/ELT is being done. In such cases, this step is a no-op.

There are two common cases here:

1. The upstream change is “absorbed,” or
2. The upstream change is “leaked.”

Testing

If the new tables are supposed to be equivalent to the old tables, it’s important to do data testing. A good data quality product will be able to profile the old data and the new data, highlighting the differences in row count, distribution, min, max, average, etc. You would often want to run the two jobs in parallel (at least 2 weeks) to ensure that the profiles are within some upper bound during that time period.

Step 3 — Provide tools for downstream owners and kick-off ripple migration

Migrations have a ripple effect. Great data engineering teams have a clear understanding of this and are able to build feedback loops in both directions to orchestrate the migration.

Once you have migrated level-1 tables, it’s time to notify downstream consumers of the data. There are 3 kinds of possible downstream consumers:

1. Dashboards built on the tables
2. Ad hoc queriers of the data
3. Derived downstream tables

For each category of downstream consumers, you are passing the responsibility of downstream migration to them. This means that:

• Downstream dashboard owners will need to migrate their dashboard to use the new source of data.
• Ad hoc queriers of the data will need to query the new sources of data instead of the old ones.
• Downstream table owners will need to migrate their ETL job to read from the new sources of data.

The last case for downstream table owners is the most interesting because either their table will absorb the change and no one further downstream of them needs to be notified. Or, their ETL job or table won’t be able to absorb the change and they will have triggered their own ripple effect, asking the consumers of their data to migrate.

Great data engineering teams provide tools for downstream consumers to know what their data is being used for, so they can perform their part of the migration.

Downstream tables and dashboards

Usually, data engineering teams will file a JIRA, which is integrated with their data catalog. This files tickets for downstream table or dashboard owners.

Dashboard owners update their dashboards, if needed. They often end up deprecating them since they no longer need them.

Downstream table owners have to do the same thing that the level-1 owners did. They either absorb or leak the change. If they absorb the change, they close the ticket. If they leak the change, they create tickets for their downstream consumers. They can use the same catalog and JIRA integration to create tickets for their downstream users.

Ad hoc queriers

While the JIRA ticket works well for downstream dashboard and table owners, it doesn’t work well for those who ad hoc query the data. Even if they were notified via a Slack message or an email message, a new data user can and will easily miss that detail.

It’s important to document the status of the migration so that those who mistakenly use the wrong data are politely redirected to the new one.

A good way to do that is to have a banner in your data catalog tool like so:

For example, if a new user stumbles upon an old table and its outdated documentation in their data catalog, they’ll be notified via banner that a new version of the table exists and redirected to use that one instead. Similar to how Amazon notifies users via banner about a new version of a product when it becomes available.

Taming the beast

Migrations don’t have to be the complex beasts that they always seem to be. There are numerous ways to ensure a smooth migration, they just involve a little extra love and care. The four steps above are the most common ways that I’ve seen data engineering teams tame the beast that is a migration, and at the heart of it lies the power of lineage and how it can transform your team’s approach. Because at the core of it, having the capability to quickly understand upstream and downstream dependencies in your data is one of the most crucial aspects of a successful migration.

To read more posts like this and stay in touch, follow me on Twitter or receive a monthly newsletter by subscribing here.

3 Steps for a Successful Data Migration was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

August highlights from the Amundsen community:

• Driving Data Governance Maturity with Amundsen at Revolut
• Amundsen at Devoted Health
• Using Column Level Lineage in Amundsen at Alvin
• Extending support for databuilder Atlas integration with table metadata models
• OpenLineage extractor for databuilder
• Generalized application model and adding Databricks application support
• Data quality integration

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

Driving Data Governance Maturity with Amundsen at Revolut

Dinan Amiendiartha, a Data Risk Manager at Revolut, joined us at our last community meeting to share how his team drove data governance maturity with Amundsen. Since January 2020, Revolut has expanded its global presence, seen a 4x increase in volume of data (and exponentially growing!), and rapidly developing new products for Retail. Due to this hyper growth, Revolut needed a solution to better manage its data and build greater trust. Dinan discussed why they chose Amundsen, dove into the details of their databuilder ingestion pipeline, how Amundsen is supporting their 4 data governance initiatives, and overall impact of Amundsen.

Check out Dinan’s presentation to hear all the details.💡

Amundsen at Devoted Health

Adam Boscarino, Manager of Data Engineering at Devoted Health, joined us at our last community meeting to share how his team has been using Amundsen at Devoted. The Devoted team was experiencing problems that many of us have heard time and time again. They had a growing data platform with increasing pipelines and tables, but no source of truth for documentation and no data lineage. They were finding it difficult to understand their own documentation and how tables were populated. There was too much tribal knowledge. They needed a single source of truth. Adam discusses why they chose Amundsen, dives deeper into their infrastructure and customization, highlights their dbt integration, and wraps up with impact and future work.

Check out Adam’s presentation to learn more.💡

Using Column Level Lineage in Amundsen at Alvin

Oisin Coveney, a Software Engineer at Alvin, joined us at our last community meeting to share the development process of enabling column level lineage in Amundsen. He shared how they see lineage at Alvin and then dove into the development process of how they developed a transformer for Amundsen and mapped Alvin to enable column level lineage.

Check out Oisin’s presentation to learn more.💡

Extending support for databuilder Atlas integration with table metadata models

We have extended support for the databuilder Atlas integration with table metadata models. 🎉

What does this mean?

• You can use databuilder extractors for a majority of table-related metadata to populate data in Atlas
• This is a backwards-compatible feature (although you might want to test it in your dev/acc Atlas before running this on prod)
• This change bridges the gap in terms of Neo4j vs. Atlas feature coverage, leaving just Column Features support to be introduced
• You can still use Atlas tools for metadata ingestion (ilke hive hook or your custom scripts) in parallel to databuilder ingestions to load the metadata from other databases
• You can use sample_data_loader_atlas.py to kickstart your Atlas test instance with the same sample data that is used for Neo4j or RDS

Huge shoutout to Mariusz Gorski!

OpenLineage extractor for databuilder

We now have an OpenLineage extractor for databuilder! This means you can extract table lineage information from OpenLineage events.

See PR details for more info. Props to Dominik Choma!

Generalized application model and added Databricks application support

Amundsen has had first-class citizen support for Airflow from Day 1. However, many companies don’t use Airflow, so we have now generalized the application model. Specifically, we have added support for Databricks application, for those companies that use Databricks’ ETL jobs.

See PR details for more info. Props to Jack Roof!

Data quality integration

We are starting to integrate existing Data Quality projects into Amundsen. The first step includes updating the frontend to display the status of the table. Specifically, we are working on display table status as: number of checks, number passing, and an optional timestamp.

Check out the open RFC and let us know what you think! 👀

Announcements

📣 We hosted office hours last month aimed for those just getting started with Amundsen. Check out the recording here.

📣 We are not hosting a September community meeting due to Mark Grover’s vacation earlier this month. Our next meeting will be on Thursday, October 7. See you next month!

Coming up next…

Next community meeting

• Date: Thursday, October 7, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/diuoeeiw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen Monthly Update — August 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the first paragraph of a post I had written earlier this month, I referred to data engineers as producers of data. Someone immediately replied something to the extent of, “You lost me at the first sentence. Data Engineers can’t be data owners.”

Clearly, I ruffled some feathers, and I have done the same plenty in the past whenever I talked about ownership of data.

Let’s say your company uses Segment to send events from its website to its data warehouse. Each Segment event, by default, creates a table in the warehouse. You, the data engineer, have created a derived view that aggregates all of these events into one table to make it much more accessible and consumable. A few days later, an analyst comes to you and says, “This view no longer has up-to-date data for any event.” And then asks the dreaded question: “Who’s the owner?”

In this case, the answer is somewhat obvious — the person who created this derived view, i.e. you — the data engineer.

But what if the question being asked was slightly different:

• What action on the website triggers this event to happen? OR,
• What does a particular item in the payload for place_order event mean?

It’s obvious that data engineers can’t be held responsible for answering such questions.

It turns out that who the data owner is depends on the question being asked.

Here are some common situations where someone could be asking the ownership question:

• The data is stale. Who should triage and fix it?
• The data is “wrong.” Some fields are nulls or malformed.
• What does this particular field mean?
• Can this data be used for a new ML model that will show a different price to different people on the website?

Ultimately, ownership of data can be divided into 3 categories:

1. Delivery owner — Ensures that this particular data gets delivered on time, on a promised delivery SLA. Usually, this is a data engineer or analytics engineer responsible for developing and maintaining the pipeline that produces this data.
2. Domain owner — What does this particular value in a field (or column) mean? When does this particular event get triggered? Usually, this is a product engineer who created the event or the analysts and data scientists who use this data most frequently and understand the physical reality that the data represents.
3. Policy owner — Ensures that this data gets used according to the classification and terms associated with it. Sometimes you acquire data from a source that should not be used for a certain category of use-cases. For example, YouTube is permitted to show ads to kids, but not permitted to personalize them. Therefore, the personalization data can’t be used if the subject is a child. The person making these calls is usually not an engineer or data scientist, but someone on the Policy or Privacy team at the company.

Aside — Quality owner: This may depend on organization, but I’ve found there to be a general resistance to take ownership of end-to-end quality of a data set. Why? Because data engineers don’t consider themselves owners of the data that’s produced from the upstream application and can’t be held responsible for hunting down a website bug that impacts the data in the warehouse. The product engineers don’t have enough context about how data gets joined and transformed downstream to own the final derived data artifact as their output. This may change as decentralized data management (AKA data mesh) is deployed more broadly.

There’s a tendency in some organizations to build “shared ownership.” This fails to work well in practice. The only times I’ve seen it work well is when there’s a shared understanding of different facets of ownership within this shared owner group and the group is able to redirect to the right person within the group based on each question.

In practice, the data engineer plays the role that a triage nurse would play in an Emergency Room. An issue (patient) comes in, they triage to see what’s going on. Sometimes it’s a problem they can fix, so they fix it and resolve the issue (injury). In other cases, they redirect to the appropriate owner (a different health practitioner). As you can see, the issue (or question being asked) informs who the owner will be.

Lastly, ownership does not need to be articulated at this level of granularity for all of your data. Data and organizations are ever-changing and any information around ownership is bound to become stale. What’s important is to have high quality granular ownership for the most important 20% of your data and have good defaults for the remaining 80%.

Organizations typically use a data catalog to document ownership information. For the most important 20% data within your organization:

Metadata and related owners

For the remaining 80% of data in your organization — automation. Frequent users are a good proxy for domain owners. And, the person who last changed the ETL code or gets alerted when the ETL job fails is a good proxy for the delivery owner.

So, the next time someone asks you about ownership, ask yourself: Are you looking for the delivery, domain, or policy owner?

To read more posts like this and stay in touch, follow me on Twitter or get a monthly newsletter with content like this by subscribing here.

Thanks to Chris Riccomini for reviewing a draft of this post.

Defining Data Ownership was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

July highlights from the Amundsen community:

• ML features discovery in Amundsen
• MySQL support in Amundsen
• Compatibility with Elasticsearch 7
• Salesforce & Oracle extractors
• Adding support for nested data types in Delta
• Proposal to make programmatic descriptions editable
• How-to: Setup OIDC Authentication in Amundsen

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

ML Features Discovery in Amundsen

We have exciting news for all the data scientists and ML modelers out there! Allison Suarez Miranda, a Software Engineer on the Amundsen team at Lyft, joined us at our July community meeting to speak about ML features discovery in Amundsen. She talked about why we’re starting with features, development of the first iteration, and a sneak peek into future work that we have in store. You can discover features, like the way you discover data in Amundsen!

Check out Allison’s presentation to hear all the details.💡

MySQL Support in Amundsen

We have the initial support to use MySQL as the backend metadata store. 🎉 At our July community meeting, Xuan Shen, who’s on the Data Infra team at WePay, spoke about adding MySQL support in Amundsen. This will allow you to use existing MySQL infrastructure as Amundsen’s backend without having to rely on a graph database. Xuan dove into Amundsen RDS and how MySQL works in databuilder and metadata service.

Check out Xuan’s presentation to learn more.💡

Compatibility with Elasticsearch 7

Amundsen is now compatible with Elasticsearch 7! This change is fully backward compatible. If you haven’t upgraded to ES.7, don’t worry. Amundsen works with both ES.6 and ES.7. 😀

Huge shoutout to Verdan Mahmood & Mariusz Gorski! You can see more details about the implementation here: https://github.com/amundsen-io/amundsen/pull/1386

Salesforce & Oracle Extractors

Salesforce, Oracle, who’s next? We’ve got two new extractors in Amundsen. 🙌

Salesforce extractor — provides support for extracting basic Salesforce object metadata from Salesforce into Amundsen.

See PR details and docs for more info. Props to Ben Rifkind!

Oracle extractor — extracts table and column metadata (including database, schema, table name, table description, column name, and column description) from the Oracle database into Amundsen.

See PR details and docs for more info. Props to Sávio Teles!

Adding support for nested data types in Delta

We’re adding support for nested datatypes in Delta! Amundsen has had support for extracting, displaying and even searching for nested data for other data sources (like BigQuery). Thanks to Jack Roof at Samsara, we recently added support for extracting and displaying nested data coming from Databricks Delta.

Check out the PR details 👉 here & here.

Proposal to make programmatic descriptions editable

We’re proposing editable programmatic descriptions.💥 Programmatic descriptions are a useful way for us to store and display business specific metadata about tables in Amundsen. This new feature will allow us to customize machine generated metadata.

Check out the open RFC and let us know what you think! 👀

How-to: Setup OIDC Authentication in Amundsen

Verdan Mahmood wrote a step-by-step guide on how to enable OIDC for your Amundsen installation. Check it out and let him know what you think!

Announcements

📣 We’d like to welcome 3 new maintainers to Amundsen! Maintainers are hugely important to our community — they make significant contributions to the project and it signifies community growth. 🌱 This means we’ll be able to develop more and support an even greater amount of users. Please welcome Dmitriy Kunitskiy from Lyft, Grant Seward from Stemma, and Dominik Choma from ING as new Amundsen maintainers! Don’t hesitate to lean on them for questions and support.

📣 We’ve done a bit of Slack housekeeping over the last month. What does this mean for you? Going forward, we’ll be using:

Updated Channels

#announcements — for all announcements for the project. Only maintainers can post on this channel (renamed from #general-no-use).

#amundsen-general-questions — for any general questions you may have about the project (renamed from #amundsen).

#troubleshoot — for specific debugging questions (no changes here).

New Channels (consider joining!)

#introductions — for introducing ourselves. All new members will be added here.

#randomness — because we’re all multifaceted people and sometimes things don’t fit in an existing bucket. 😁

Coming up next…

Next community meeting

• Date: Thursday, September 2, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/hsusdihw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen Monthly Update — July 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

You notice that data producers (data engineers, product engineers, analytics engineers) and data consumers (data analysts, data scientists, business users) don’t know what data exists, how to use it, and what is trustworthy. You see a ton of questions on slack and a real hit on the productivity of the data team.

You get a big fancy data catalog — it has lots of bells and whistles but months go by and no one uses it. You show it in your all-hands meeting, you send email blasts, you even reprimand people when they talk about data stuff in the Slack channel, pushing them to have this conversation within the data catalog.

But at the end of the day, the usage never sticks. Your data catalog never gets adopted.

I have seen this story play out so many times. Here are the top 2 reasons why, despite all good intentions, many data catalogs end up failing:

1. Catalog ghost town — lack of descriptions and metadata

In order to derive value from most data catalogs, you need to populate it with valuable information. This information includes descriptions, tags, primary keys, foreign keys, common ways to query the data, most common FAQs, etc. You get the idea. The hard part is getting your co-workers to enter this context.

Even if you somehow convinced others to add documentation, once it’s added, it quickly becomes out-of-date. Because data evolves quickly, not only does it need to be added once, it needs constant upkeep.

When upkeep doesn’t happen, well, your data catalog doesn’t help — it hurts. This story is the single most common story of data catalog failure.

I have seen the three best antidotes to this:

Automate as much as possible. Try to get as much metadata as possible through automation. There’s no reason why someone needs to populate foreign keys, primary keys, common ways to query the data for every single data set. All of that information is already buried and up-to-date in the existing usage patterns and logs. Extract the most common filters, join conditions, upstream dependencies, and downstream consumers. More on how to do that in a future blog post.

Extract documentation from “the flow”. Documentation is best obtained when it’s in the flow of the user. When a new data set is being created, you need to enforce documentation to be added then. Sometimes it’s a process change, but quite often it’s technical — I have seen quite a few successful deploys where you put in documentation checks in your CI/CD and break the build if the appropriate documentation isn’t supplied.

Curate the top 20%. There’s no denying that some things require a human to curate. Take your most commonly viewed metrics, dashboards, and tables and document them. The pre-requisite to doing this is realizing that you can’t boil the ocean and having an understanding of what are your most commonly queried tables and most commonly viewed dashboards.

2. Your catalog is too broad and not deep enough — leading to fragmentation

Often in understanding and trusting data, you need to answer and do the following:

• What are the most common conversations about this data?
• What knowledge base/wikis exist about this data?
• Query the data to explore it further

More recently, catalogs have become quite bulky and cause fragmentation in the user’s golden path:

• Conversations — You can have them on Slack or in your data catalog
• Knowledge base — You can write wiki articles in Confluence or in your data catalog
• Querying — You can query in your BI tool (Tableau, Looker, Mode, Snowflake UI) or you can query data via your data catalog.

This fragmentation is the worst possible thing. Why?

• The users now have to figure out which tool to choose for what.
• The data team now has to maintain the above two options. Take querying, for example. You now have to implement RBAC in your BI tool and your data catalog’s querying tool. Like implementing RBAC in just one tool wasn’t hard enough.

Ultimately, this fragmentation leads to enough friction, both individually and organizationally, for the data catalog to fail. Your catalog ends up doing a crappy job at too many things.

The antidote to this problem — if you have a catalog product that’s too bulky, choose your golden path thoughtfully and disable duplicate features amongst your products so that there’s one suggested option. For example, have one place for people to have conversations, either in your catalog or in Slack. Discourage the other option.

Summary

Data catalogs fail for two reasons:

1. Catalog Ghost town — not enough descriptions & metadata.
2. Catalog too broad, and not deep enough — fragmentation in the user’s golden path.

To avoid these problems, regardless of your tool choice, automate as much metadata as possible, curate only on the most impactful data, and be laser-focused on your user’s golden path.

To read more posts like this and stay in touch, follow me on Twitter or get a monthly newsletter with content like this by subscribing here.

Top 2 Reasons Why Data Catalogs Fail was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

June highlights from the Amundsen community:

• How Convoy got 80% of its tech org to use Amundsen
• Query Intelligence: View common query patterns in Amundsen
• Unifying shared dependencies — making life a lot easier for Amundsen devs and installers
• Amundsen now uses Mode discovery API
• Initial support to use MySQL as backend metadata store
• See popular dashboards in Amundsen
• Lineage graph visualization

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

How Convoy got 80% of its tech org to use Amundsen

Chad Sanderson and Daniel Dicker, from the Data Platform team at Convoy, joined us at our June community meeting and spoke about their success in implementing and onboarding their data consumers to Amundsen. They dove into the gaps that led them to make investments in Amundsen, their data discovery goals, implementation journey, and where they’re looking to take Amundsen into the future.

Gaps in Data Discovery

In early 2020, the team at Convoy started experiencing issues with data discovery. It was time-consuming to share information about what was in the data warehouse and the overhead of updating documentation kept increasing. Lots of conversations were happening in Slack and information became siloed, with institutional knowledge steadily becoming the norm. It was difficult for a single person to have a holistic view of the Convoy data model. With Convoy constantly growing and changing, the team knew they had to find a solution that would scale and allow a better data discovery experience.

Data Discovery Goals

Chad, Daniel, and their team had 4 main goals as they searched for a solution:

1. Lower the barrier to entry for adding informative, useful information. They wanted to make this as easy as possible for data consumers.
2. Improve search functionality to enable users to quickly find relevant information.
3. Add more useful context around the data: Who is using tables? Who are the owners of each table? What queries are being run against tables?
4. Strong foundation to build additional features as their data discoverability tool grew and changed with the business.

Choosing Amundsen

Chad, Daniel, and their team chose Amundsen for three main reasons: feature set, familiar technologies, and growing community. Since its implementation, the catalog has been a great success — 80% of the Product and Engineering team has used it over the past year, with over 4,500 searches per month.

Check out Chad and Daniel’s presentation to hear more about how they rolled out Amundsen to the broader team and upcoming plans for their data catalog.

Query Intelligence: View common query patterns in Amundsen

We also heard from Grant Seward, founding engineer at Stemma, at our June community meeting. He gave a presentation about query intelligence and how Amundsen is able to show common query patterns now.

As we think about the data discovery lifecycle, we’ve all had some form of these kinds of questions pop into our minds: What data exists? How do I join these tables together? What is the right way to form these queries? Are there any “gotchas” that exist in the data?

Grant gives a framework on how data discovery provides comprehensive answers to six fundamental questions:

What? Where? Who? When? Why? How?

However, Amundsen has had limited coverage for the question, “How?”

Query intelligence aims to augment our understanding of how data can be used by providing historical examples, which can then be utilized as a starting point during the research phase or referenced during development.

Since queries are by far the most widely used way in which we interact with data, by indexing query components, we can answer questions such as:

• What tables are most commonly joined to this table and how?
• Are there any filters (where clauses) that are always applied when accessing this table?
• What real-world queries are being used on this table?
• What queries are relevant now, given the recent queries that have been executed?

The answers to these questions form the building blocks that provide context about the data prior to any hands-on exploration that is done. This intelligence will answer the “How?” question, helping you discover how to use your data.

Check out Grant’s presentation to see a live demo and learn how Amundsen indexes query components.

Developer Experience — Unifying shared dependencies

We’ve merged refactoring dependency management in monorepo. If you’re building Amundsen packages/docker image from monorepo, this change will likely affect you.

Problem

Prior to this change, Python packages were pretty much the same for each proxy. This made it hard to sync across all repositories, and as a result, each proxy was running its own version of the dependency. We were having a lot of issues resulting in amundsen-common mismatch between FE and Metadata.

Solution

By unifying dependencies shared across multiple services (for both core and testing dependencies), all packages will be located in one place, which will keep dependencies similar across versions. This will make it possible to open a single PR that will update multiple packages.

What can you expect from this change?

• Installation from source — when installing from the source, use “pip3 install -e .” to ensure dependencies are properly pulled
• Building docker images — we moved docker images to root directory of Amundsen repo
• Makefile actions were adjusted to this change — you can do “make install_deps” OR “make image” to install from the source, OR build docker image respectively. The same goes for all docker-compose files available in monorepo.

Huge shoutout to Mariusz Gorski and the community for this refactor! You can see more details of the implementation here.

Migration to Mode discovery API

The Amundsen Mode dashboard integration has now been migrated over to the newly performant Mode discover API. 🎉 This will greatly improve the experience for those using the Mode dashboard integration.

Big thanks to Junda Yang from Brex and Neha Hystad from Mode for the support.

Initial support to use MySQL as backend metadata store

Thanks to WePay’s Xuan Shen’s contributions, Amundsen now has the initial support to use MySQL as the backend metadata store! This will allow you to use existing MySQL infrastructure as Amundsen’s backend without having to rely on a graph database. See more details here.

Xuan Shen gave a detailed presentation at our July community meeting. We’ll share more details in our next monthly update. ✨

Popular Dashboards

You can now see popular dashboards in Amundsen. You have always been able to see popular tables — we gave the same love to dashboards. BTW, popular here means most queried/most viewed dashboards, not popular based on Amundsen activity.

Thanks to Verdan Mahmood for his contributions here. 🙌

Lineage Graph Visualization

As part of Amundsen’s commitment to building in lineage exploration, we had two additional iterations on the graphical visualization UI for table lineage. We brought in atlas proxy support as well as multiple stability improvements following direct community feedback. Our next steps are to enter a brief period of aggregating community use cases and shaping the future of the lineage feature UX.

Huge props to Boyan Bonev for spearheading these UI/UX iterations!

Announcements

📣 We’d like to welcome Junda Yang, Amundsen’s latest committer (Junda’s GitHub). Junda was an early engineering member on the Data Tools team at Lyft working on Amundsen. He’s recently done a fantastic job integrating Amundsen with Mode dashboard and its latest discovery API. Looking forward to more contributions from Junda in the future! Amundsen is a Linux Foundation project, with an open governance model. Adding new committers like Junda, and growing our dev community, is an important, ongoing part of our project.

📣 We have deprecated the amundsen-atlas-types repository. Up until now, we maintained a separate package for creating proper entity types in Atlas in order to facilitate integration between Atlas and Amundsen. Moving forward, we are still keeping the approach of creating those entity types, BUT, we are moving the code directly to Amundsen repo. Moving forward, amundsen databuilder repo should be used instead of amundsen-atlas-types. Please see the tutorial for more details.

📣 We have a new version of flaskoidc. The new version available on PyPi is 1.0.2 and is backward incompatible. With this new version, we’ve moved away from flask-oidc and now rely on Authlib, there’s no need to maintain a separate clients_secrets.json file anymore. Configuration is cleaner and requires a few environment variables to enable e2e OIDC support. The access token is not being refreshed at the moment — this will be fixed in future releases. More details on how to configure this new version on github.

Coming up next…

ML Feature Discovery in Amundsen

A few months ago, we presented the first version of ML feature discovery in which Mariusz Strzelecki, from GetInData, had built V1 of what ML feature discovery and integration with Feast looks like. More recently during our July community meeting, Allison Suarez Miranda, from Lyft, shared what the next version of ML feature discovery in Amundsen looks like.

Next community meeting

• Date: Thursday, August 5, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/huuaiauw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen monthly update — June 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are only 2 goals that matter when it comes to measuring the success of a data catalog: 1) adoption, and 2) customer satisfaction. If you nail these two, you are successful.

I’m the co-creator of the leading open-source data catalog, Amundsen, which is used by 35+ companies including Instacart, Square, Brex, Asana, and many more. In this post, I share key learnings from Lyft, other Amundsen adopters, and Stemma customers on what makes a data catalog install successful.

There are learnings that we have incorporated in Stemma, but this article captures learnings that haven’t been captured in the product yet. These learnings focus on how to launch the product, how to land it for great adoption, and how to measure success.

1. Prioritize a persona and its use-cases

There are many user personas and use cases for a data catalog. Successful installs prioritize which personas and use cases to focus on first. Here’s a simplified view¹ of the most common personas and use-cases for a data catalog. It’s less important which persona you start with first, but more important that you start with a specific target group of users.

2. Launch in phases

In this section, I’ll dive deeper into best practices for launching your data catalog.

Step 1: Identify a small set of tables to get alpha user feedback on.

• This set can be the most commonly used tables within the company (often referred to as “core” tables) or one domain within the company like marketing, growth or finance, etc.
• More often, I have seen core tables being the chosen set, partly because they are the most impactful, but also because there’s often a central data team responsible for maintaining them.

Step 2: Populate MVP metadata on these tables.

• This is where most data catalogs fail. In order for users to get value out of them, descriptions, tags, owners, etc. need to be curated. However, this isn’t sustainable without having an army of data stewards, and this documentation quickly becomes out of date. This is the single biggest reason why data catalogs fail. Avoid this pitfall by choosing an automated data catalog for the majority of data and curate only the most impactful data.
• Where you must, for tribal knowledge, it helps to do a “docs jam session” with a group of data producers and consumers. You can even offer a reward (like a gift card) for those that put in the most docs!

Step 3: Alpha launch to 5–20 alpha users.

• It’s best for alpha users to be ultra-vocal users. These will be from the prioritized persona you chose earlier. These users will become the data catalog’s avid supporters when you launch to a broader audience.
• Incorporate feedback and iterate. Some types of feedback are super valuable here, like when someone says, “Oh, we already have this metadata in this spreadsheet — we should pull that in here, too.”

Step 4: Beta launch to all users of the prioritized persona.

• It’s important to focus your beta launch on your prioritized users (data consumers, for example). One common mistake is to dilute the focus of your launch by opening up to all personas. That doesn’t mean that you should lock out other personas from the data catalog, it just means that you sequence which personas to focus on first.
• Graduate to GA if you can meet success metrics targets. More on that in a later section on measuring success.

3. Land for great adoption

In order to get great adoption, here are a few best practices I have seen work:

• Update Slack channel headers where people ask each other questions. Product features can be super helpful here — for example, if your catalog has Slack integration and can link these conversations to the catalog automatically.
• Embed into new hire training. Tagging data sets per domain (marketing, growth, etc.) can help new hires quickly onboard to their domains. If you have existing training, showcase the catalog as an entry point. At Lyft, we had all tech new hires instrument a metric during onboarding. They used Lyft’s data catalog for discovering and understanding the right data for that task.
• Linkages with other products. Create links between various data tools. For example, auto-populate a link between Airflow DAG that populates a table and the table page in the data catalog (and vice-versa). Another impactful link is between the table page in the data catalog and a link to the code that is used to generate the table.
• Showcase the catalog at a group or company meeting. Deliver a short 5-minute demo at an all-hands meeting that targets persona users. Educate, answer questions, and thank your alpha users — it’s super impactful by creating more awareness and an opportunity to learn.

4. Measure success

Like I said earlier, adoption and customer satisfaction are the only two goals that matter. I dig further into what specific metric definitions to use for each of them:

1. Adoption:

• WAUs: I’d suggest starting off with Weekly Active Users (WAUs) instead of Daily Active Users or Monthly Active Users. Common usage frequency is weekly, not daily or monthly.‍
• Target Penetration rate: 80%. A great penetration rate is 80% within your target persona.

2. Customer Satisfaction (CSAT):

• Measure out of band periodically. In my experience, out of band (not in product), CSAT feedback measured periodically (every 3 or 6 months) is better than getting feedback within the data catalog product. I have learned that when feedback is measured in the product, the most recent experience can tarnish the feedback shared by the user.

There are a few other metrics that companies often consider: documentation quality, search quality, etc. However, my recommendation is to stick to the core metrics at the onset. As your data catalog matures and you ingest more metadata into your data catalog over time, you can use those specific metrics to track the impact of those various improvements.

I hope this step-by-step guide helps inform you and your team as you navigate your data catalog install and makes your data catalog successful. The right data catalog can greatly reduce the overhead of curation. However, the above steps still play a huge role in ensuring your success, regardless of the data catalog you choose.

Want to learn more about Stemma’s fully managed data catalog? Check out the demo and get started on stemma.ai.

How to Make Your Data Catalog Successful was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why Snap chose Amundsen, dbt integration, lineage graph viz, mono repo, and more!

Summary

May highlights from the Amundsen community:

• Why Snap chose Amundsen
• dbt ♥ Amundsen
• Lineage graph viz
• One repo to rule them all
• ML feature discovery in Amundsen
• Answer how to best use this table — with queries in Amundsen
• Sydney DataEng meetup

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

Why Snap chose Amundsen

This month, we heard from Justin Jordan, a developer on Snap’s analytics engineering team, about why his team chose Amundsen and how they use it to solve their problems with data discovery, specifically metrics discovery metadata.

Problem

Due to high growth, Snap ran into problems related to metrics discovery. With the number of metrics accelerating every year, it became difficult to know what metrics existed and how to use them. Challenging questions arose: Who owns the metrics? Where are they surfaced? What do they mean? Where is their data stored?

Solution

Justin and his team chose Amundsen to build Snap’s metric catalog. This catalog allows teams to search for metrics and definitions. This is a huge step forward — now teams can see all relevant technical and business metadata for their metrics. They can manage the entire metric lifecycle with one source of truth.

Check out Justin’s presentation below to hear him explain why Amundsen was chosen and what’s next for his team.

dbt ♥ Amundsen

dbt has enabled thousands of analysts and analytics engineers to transform data in their data warehouse. Many Amundsen users are also heavy dbt users. To that effect, we have built a deep integration with dbt ingesting rich metadata like lineage, tags, and descriptions from dbt into Amundsen.

Thanks to contributions from Grant Seward, Amundsen now integrates with dbt’s core metadata. There’s still a lot to be done here. Join Amundsen Slack — we’d love your feedback.

Lineage Graph Viz

Amundsen already has support for lineage. However, the default way of viewing lineage thus far was a list-based view, which includes downstream and upstream tabs. Amundsen has been working on a graphical view of lineage. Thanks to Knowl Baek for designs and to Verdan Mahmood for building the first beta version of the graph viz.

Amundsen lineage currently supports Neo4j proxy and support for Atlas proxy is coming soon.

We also want to highlight a common question from the community — how do you enable lineage on the front end? Verdan provides the answer in our community meeting recording here.

One repo to rule them all

As you know, Amundsen has a microservice-based architecture. For more flexibility, we started with one code repo for each of the four services — frontend service, metadata service, search service, and databuilder (which is a library not a service, but you get the idea). We learned from community and developer feedback that having different code repositories was a big pain, especially when changes needed to be coordinated across repositories.

We heard you!

We decided to move Amundsen code to one repo. If you develop on Amundsen, you should switch all your development to the new main branch. If you deploy Amundsen, this change doesn’t have much impact on you since the Python packages remain unchanged.

ML feature discovery in Amundsen

It’s happening! For a long time, many of us have wanted to integrate our ML projects in Amundsen. Feature engineering is one of the most time-consuming steps for ML engineers, and Amundsen is here to help. Allison Suarez Miranda and Knowl Baek proposed an RFC for indexing ML projects in Amundsen. The goal is for ML features to be indexed as first-class citizens just like tables, dashboards, and people — providing easy discoverability and ultimately, greater model re-usage and overall productivity.

Answer how to best use this table — with queries in Amundsen

Another exciting proposal for our roadmap, this time from Grant Seward — the ability to index queries and their individual components. Queries are one of the most common ways that Amundsen users interact with data. There is a significant amount of latent information that exists within these queries, which can be useful for users to discover. What does indexing queries mean for you? This means Amundsen will be able to answer key questions such as:

• What tables are most commonly joined to this table and how?
• Are there any filters (where clauses) that are always applied when accessing this table?
• What real-world queries are being used on this table?
• What queries are relevant now, given the recent queries that have been executed?

This will help you discover how to use your data.

Sydney DataEng meetup

Peter Hanssens presented at the Sydney DataEng meetup! He spoke about Amundsen/Neptune/CDK. See video below

Coming up next…

Shared dependencies across microservices

We’re continuing to work on making Amundsen easy to install and develop. After monorepo, we are working on unifying shared dependencies across various microservices. The goal is to avoid any potential mismatch between libraries shared between frontend, metadata, and search services. You can read more details here.

Dashboard support in Atlas metadata proxy

Amundsen was the first data catalog to add support for dashboards — reducing the duplication of analytical work as well as easing with change management. We currently support Tableau, Mode, Redash, and Superset. Dashboards only worked when you used the neo4j backend, we are now working on making dashboards work for Amundsen users who use the Atlas backend.

Next community meeting

• Date: Thursday, July 1, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/odsidaaw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen Monthly Update — May 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, we are excited to announce the launch of Stemma — a fully managed data catalog, powered by Amundsen, the leading open-source data catalog with the largest community and broadest adoption. We raised $4.8M in seed funding led by Sequoia to bring the power of the leading open-source data catalog to every organization.

The problem: Too much data, too little trust

Over the last decade companies have first captured more data and then made it accessible to more and more people within the company.

Everyone has access to data but few know what exists, what’s trustworthy, and how to use it.

This leads to a huge productivity loss and a big risk for the company. This problem doesn’t just impact the organization. It’s deeply visceral to data consumers like data analysts, data scientists, and business users and data producers like product and data engineers. It impacts them every day.

Analysts and Data Scientists deliver inaccurate reports and models because they inadvertently use the wrong source or incorrect logic. Even worse, the data keeps changing underneath them. Data gets delayed, deprecated or completely shut off and analysts and data scientists are the last ones to find out.

Data Engineers, on the other hand, are constantly bogged down with keeping everyone informed about the current status and upcoming changes to data. Data owners don’t know exactly what and who a change will impact, so they spray and pray. They blast their users with blanket emails that no one reads, let alone remembers.

First attempt: Gossip Protocols

The first attempt to bring sense to this lack of trust is the natural human response — gossip protocols of Slack and shoulder-tapping.

You create a Slack channel #ask-analytics where users ask questions like “What is the source of truth for X?”. It may take a few days to get an answer, but over time you get repeat questions. You wish people searched the Slack channel before they asked the same question over and over again.

It gets worse. Data evolves. And now you wish people would never search the Slack channel, lest they find out-of-date info. You’re thinking about limiting the retention to a couple of weeks.

But it gets worse. Wrong data leads to wrong conclusions: two different departments show two different forecasts for gross shipments during a board meeting, and suddenly everyone realizes this isn’t working, and something must be done!

Second attempt: Curated data catalog

The second attempt is to curate and document information about data — descriptions, experts, dependencies, update frequency, foreign keys, sample queries, and the list goes on. You document this either in a wiki or buy a full-blown product that’s simply just a data-aware wiki.

Sometimes you try to get an army of volunteers to enter documentation into this wiki. If you are lucky, you get the first set of docs in, but it starts rotting the day it’s written because the writing documentation requires the user to leave their existing flow.

Sometimes, you find someone (aka data steward) whose job it is to ensure this documentation is entered and remains up-to-date. But this doesn’t work because data stewards, while super valuable, don’t have context on all the uses of data, so they end up relying on data experts, bringing us back to the same problem as the army of volunteers.

Curation doesn’t work.

Amundsen — the leading open-source data catalog

After experiencing the faults of gossip protocol and curated data catalogs, I co-created Lyft’s data catalog, Amundsen, to solve the challenge of trust in data through automation. Amundsen is widely adopted at Lyft — used by 750 users every week with 75% of Data Analysts, Data Scientists, and Data Engineers using it every week. To this day, Amundsen is the highest-ranked Data & Analytics product at Lyft.

Amundsen is the leading open-source data catalog with the largest community and adoption. It’s used by 35+ companies today including Square, Instacart, ING, Brex, Asana, iRobot, and many others. You can join Amundsen’s growing community on Slack and read more about the project here.

Today, we are excited to launch Stemma’s product — bringing the power of Amundsen and more to all of you.

Stemma — bringing the power of Amundsen to you

Stemma is building on top of Amundsen and adds value in two ways: a) enterprise management — super easy deployment with enterprise-grade security. b) intelligence — automated documentation like common filter and join conditions, related Slack conversations, and a personalized experience based on the user’s role and activity.

We have a lot of exciting stuff coming up and we are going to make Stemma and Amundsen better in a myriad of ways. More on that in a later blog post. Subscribe to our blog to stay in touch. We’re deeply dedicated to making Amundsen the de-facto choice for anyone who needs a modern data catalog, and to empower organizations to harness its power through the press of a button with Stemma.

Dorian and I started Stemma in 2020 to help bring the power of automated data catalog to the market. Today, we are also super excited to announce our seed funding from Sequoia.

Learn more, see our demo, and get started at stemma.ai.

Stemma: Helping you trust your data was originally published in Stemma on Medium, where people are continuing the conversation by highlighting and responding to this story.