Summary

September highlights from the Amundsen community:

• Integrating Slack with Amundsen at Convoy
• Adding stats at a table level
• Improved support for showing nested columns
• Additional metadata for BigQuery
• Search revamp
• Extending badges browsing capabilities

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

Integrating Slack with Amundsen at Convoy

This past summer, George Pu joined Convoy’s Data Platform team as an intern to help surface common questions about data usage within Slack through Amundsen. The result? A usable and robust Slack integration. Learn more about the process he took and the team’s next steps: Integrating Slack with Amundsen for Ease of Data Discovery

Adding stats at a table level

Up until now, we have allowed adding stats (min, max, count distinct) on a column level. Moving forward, we now additionally allow adding stats on a table level, allowing us to share stats like “daily active users” or “query count” on a table basis.

See PR details for more info! Huge props to Allison Suarez Miranda!

Improved support for showing nested columns

We now support showing nested columns as individual rows in the list of columns instead of showing them in a modal.

See PR details for more info. Thanks to Daniel Won!

Additional metadata for BigQuery

We now show the column’s mode for BigQuery tables.

See PR details for more info. Thanks to Marwan Baghdad!

Search revamp

We’re looking to revamp search with a cleaner backend implementation, more robust queries, better customizability, and make use of modern v7 ES features.

This revamp would include deprecating Apache Atlas as a search backend. If you run Amundsen without Elasticsearch and instead rely purely on Atlas for search AND migrating to Elasticsearch would be prohibitive, please reach out to Dmitriy Kunitskiy. The goal is to improve the Amundsen search/ranking experience and remove a limited and likely-unused option of using Atlas in place of ES.

Check out the RFC for more info and let us know what you think!

Extending badges browsing capabilities — badges on landing page and browse view

We’re looking to introduce a “Popular/All badges” section below “Popular tags” on the landing page, as well as a browse section.

Check out the RFC for more info and let us know what you think!

Announcements

📣 No community meeting updates from last month due to my vacation. I took a week off and went to Orcas Island for a short trip before Labor Day!

Coming up next…

Next community meeting

• Date: Thursday, November 4, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/sahdiasw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen Monthly Update — September 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

August highlights from the Amundsen community:

• Driving Data Governance Maturity with Amundsen at Revolut
• Amundsen at Devoted Health
• Using Column Level Lineage in Amundsen at Alvin
• Extending support for databuilder Atlas integration with table metadata models
• OpenLineage extractor for databuilder
• Generalized application model and adding Databricks application support
• Data quality integration

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

Driving Data Governance Maturity with Amundsen at Revolut

Dinan Amiendiartha, a Data Risk Manager at Revolut, joined us at our last community meeting to share how his team drove data governance maturity with Amundsen. Since January 2020, Revolut has expanded its global presence, seen a 4x increase in volume of data (and exponentially growing!), and rapidly developing new products for Retail. Due to this hyper growth, Revolut needed a solution to better manage its data and build greater trust. Dinan discussed why they chose Amundsen, dove into the details of their databuilder ingestion pipeline, how Amundsen is supporting their 4 data governance initiatives, and overall impact of Amundsen.

Check out Dinan’s presentation to hear all the details.💡

Amundsen at Devoted Health

Adam Boscarino, Manager of Data Engineering at Devoted Health, joined us at our last community meeting to share how his team has been using Amundsen at Devoted. The Devoted team was experiencing problems that many of us have heard time and time again. They had a growing data platform with increasing pipelines and tables, but no source of truth for documentation and no data lineage. They were finding it difficult to understand their own documentation and how tables were populated. There was too much tribal knowledge. They needed a single source of truth. Adam discusses why they chose Amundsen, dives deeper into their infrastructure and customization, highlights their dbt integration, and wraps up with impact and future work.

Check out Adam’s presentation to learn more.💡

Using Column Level Lineage in Amundsen at Alvin

Oisin Coveney, a Software Engineer at Alvin, joined us at our last community meeting to share the development process of enabling column level lineage in Amundsen. He shared how they see lineage at Alvin and then dove into the development process of how they developed a transformer for Amundsen and mapped Alvin to enable column level lineage.

Check out Oisin’s presentation to learn more.💡

Extending support for databuilder Atlas integration with table metadata models

We have extended support for the databuilder Atlas integration with table metadata models. 🎉

What does this mean?

• You can use databuilder extractors for a majority of table-related metadata to populate data in Atlas
• This is a backwards-compatible feature (although you might want to test it in your dev/acc Atlas before running this on prod)
• This change bridges the gap in terms of Neo4j vs. Atlas feature coverage, leaving just Column Features support to be introduced
• You can still use Atlas tools for metadata ingestion (ilke hive hook or your custom scripts) in parallel to databuilder ingestions to load the metadata from other databases
• You can use sample_data_loader_atlas.py to kickstart your Atlas test instance with the same sample data that is used for Neo4j or RDS

Huge shoutout to Mariusz Gorski!

OpenLineage extractor for databuilder

We now have an OpenLineage extractor for databuilder! This means you can extract table lineage information from OpenLineage events.

See PR details for more info. Props to Dominik Choma!

Generalized application model and added Databricks application support

Amundsen has had first-class citizen support for Airflow from Day 1. However, many companies don’t use Airflow, so we have now generalized the application model. Specifically, we have added support for Databricks application, for those companies that use Databricks’ ETL jobs.

See PR details for more info. Props to Jack Roof!

Data quality integration

We are starting to integrate existing Data Quality projects into Amundsen. The first step includes updating the frontend to display the status of the table. Specifically, we are working on display table status as: number of checks, number passing, and an optional timestamp.

Check out the open RFC and let us know what you think! 👀

Announcements

📣 We hosted office hours last month aimed for those just getting started with Amundsen. Check out the recording here.

📣 We are not hosting a September community meeting due to Mark Grover’s vacation earlier this month. Our next meeting will be on Thursday, October 7. See you next month!

Coming up next…

Next community meeting

• Date: Thursday, October 7, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/diuoeeiw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen Monthly Update — August 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

July highlights from the Amundsen community:

• ML features discovery in Amundsen
• MySQL support in Amundsen
• Compatibility with Elasticsearch 7
• Salesforce & Oracle extractors
• Adding support for nested data types in Delta
• Proposal to make programmatic descriptions editable
• How-to: Setup OIDC Authentication in Amundsen

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

ML Features Discovery in Amundsen

We have exciting news for all the data scientists and ML modelers out there! Allison Suarez Miranda, a Software Engineer on the Amundsen team at Lyft, joined us at our July community meeting to speak about ML features discovery in Amundsen. She talked about why we’re starting with features, development of the first iteration, and a sneak peek into future work that we have in store. You can discover features, like the way you discover data in Amundsen!

Check out Allison’s presentation to hear all the details.💡

MySQL Support in Amundsen

We have the initial support to use MySQL as the backend metadata store. 🎉 At our July community meeting, Xuan Shen, who’s on the Data Infra team at WePay, spoke about adding MySQL support in Amundsen. This will allow you to use existing MySQL infrastructure as Amundsen’s backend without having to rely on a graph database. Xuan dove into Amundsen RDS and how MySQL works in databuilder and metadata service.

Check out Xuan’s presentation to learn more.💡

Compatibility with Elasticsearch 7

Amundsen is now compatible with Elasticsearch 7! This change is fully backward compatible. If you haven’t upgraded to ES.7, don’t worry. Amundsen works with both ES.6 and ES.7. 😀

Huge shoutout to Verdan Mahmood & Mariusz Gorski! You can see more details about the implementation here: https://github.com/amundsen-io/amundsen/pull/1386

Salesforce & Oracle Extractors

Salesforce, Oracle, who’s next? We’ve got two new extractors in Amundsen. 🙌

Salesforce extractor — provides support for extracting basic Salesforce object metadata from Salesforce into Amundsen.

See PR details and docs for more info. Props to Ben Rifkind!

Oracle extractor — extracts table and column metadata (including database, schema, table name, table description, column name, and column description) from the Oracle database into Amundsen.

See PR details and docs for more info. Props to Sávio Teles!

Adding support for nested data types in Delta

We’re adding support for nested datatypes in Delta! Amundsen has had support for extracting, displaying and even searching for nested data for other data sources (like BigQuery). Thanks to Jack Roof at Samsara, we recently added support for extracting and displaying nested data coming from Databricks Delta.

Check out the PR details 👉 here & here.

Proposal to make programmatic descriptions editable

We’re proposing editable programmatic descriptions.💥 Programmatic descriptions are a useful way for us to store and display business specific metadata about tables in Amundsen. This new feature will allow us to customize machine generated metadata.

Check out the open RFC and let us know what you think! 👀

How-to: Setup OIDC Authentication in Amundsen

Verdan Mahmood wrote a step-by-step guide on how to enable OIDC for your Amundsen installation. Check it out and let him know what you think!

Announcements

📣 We’d like to welcome 3 new maintainers to Amundsen! Maintainers are hugely important to our community — they make significant contributions to the project and it signifies community growth. 🌱 This means we’ll be able to develop more and support an even greater amount of users. Please welcome Dmitriy Kunitskiy from Lyft, Grant Seward from Stemma, and Dominik Choma from ING as new Amundsen maintainers! Don’t hesitate to lean on them for questions and support.

📣 We’ve done a bit of Slack housekeeping over the last month. What does this mean for you? Going forward, we’ll be using:

Updated Channels

#announcements — for all announcements for the project. Only maintainers can post on this channel (renamed from #general-no-use).

#amundsen-general-questions — for any general questions you may have about the project (renamed from #amundsen).

#troubleshoot — for specific debugging questions (no changes here).

New Channels (consider joining!)

#introductions — for introducing ourselves. All new members will be added here.

#randomness — because we’re all multifaceted people and sometimes things don’t fit in an existing bucket. 😁

Coming up next…

Next community meeting

• Date: Thursday, September 2, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/hsusdihw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen Monthly Update — July 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

June highlights from the Amundsen community:

• How Convoy got 80% of its tech org to use Amundsen
• Query Intelligence: View common query patterns in Amundsen
• Unifying shared dependencies — making life a lot easier for Amundsen devs and installers
• Amundsen now uses Mode discovery API
• Initial support to use MySQL as backend metadata store
• See popular dashboards in Amundsen
• Lineage graph visualization

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

How Convoy got 80% of its tech org to use Amundsen

Chad Sanderson and Daniel Dicker, from the Data Platform team at Convoy, joined us at our June community meeting and spoke about their success in implementing and onboarding their data consumers to Amundsen. They dove into the gaps that led them to make investments in Amundsen, their data discovery goals, implementation journey, and where they’re looking to take Amundsen into the future.

Gaps in Data Discovery

In early 2020, the team at Convoy started experiencing issues with data discovery. It was time-consuming to share information about what was in the data warehouse and the overhead of updating documentation kept increasing. Lots of conversations were happening in Slack and information became siloed, with institutional knowledge steadily becoming the norm. It was difficult for a single person to have a holistic view of the Convoy data model. With Convoy constantly growing and changing, the team knew they had to find a solution that would scale and allow a better data discovery experience.

Data Discovery Goals

Chad, Daniel, and their team had 4 main goals as they searched for a solution:

1. Lower the barrier to entry for adding informative, useful information. They wanted to make this as easy as possible for data consumers.
2. Improve search functionality to enable users to quickly find relevant information.
3. Add more useful context around the data: Who is using tables? Who are the owners of each table? What queries are being run against tables?
4. Strong foundation to build additional features as their data discoverability tool grew and changed with the business.

Choosing Amundsen

Chad, Daniel, and their team chose Amundsen for three main reasons: feature set, familiar technologies, and growing community. Since its implementation, the catalog has been a great success — 80% of the Product and Engineering team has used it over the past year, with over 4,500 searches per month.

Check out Chad and Daniel’s presentation to hear more about how they rolled out Amundsen to the broader team and upcoming plans for their data catalog.

Query Intelligence: View common query patterns in Amundsen

We also heard from Grant Seward, founding engineer at Stemma, at our June community meeting. He gave a presentation about query intelligence and how Amundsen is able to show common query patterns now.

As we think about the data discovery lifecycle, we’ve all had some form of these kinds of questions pop into our minds: What data exists? How do I join these tables together? What is the right way to form these queries? Are there any “gotchas” that exist in the data?

Grant gives a framework on how data discovery provides comprehensive answers to six fundamental questions:

What? Where? Who? When? Why? How?

However, Amundsen has had limited coverage for the question, “How?”

Query intelligence aims to augment our understanding of how data can be used by providing historical examples, which can then be utilized as a starting point during the research phase or referenced during development.

Since queries are by far the most widely used way in which we interact with data, by indexing query components, we can answer questions such as:

• What tables are most commonly joined to this table and how?
• Are there any filters (where clauses) that are always applied when accessing this table?
• What real-world queries are being used on this table?
• What queries are relevant now, given the recent queries that have been executed?

The answers to these questions form the building blocks that provide context about the data prior to any hands-on exploration that is done. This intelligence will answer the “How?” question, helping you discover how to use your data.

Check out Grant’s presentation to see a live demo and learn how Amundsen indexes query components.

Developer Experience — Unifying shared dependencies

We’ve merged refactoring dependency management in monorepo. If you’re building Amundsen packages/docker image from monorepo, this change will likely affect you.

Problem

Prior to this change, Python packages were pretty much the same for each proxy. This made it hard to sync across all repositories, and as a result, each proxy was running its own version of the dependency. We were having a lot of issues resulting in amundsen-common mismatch between FE and Metadata.

Solution

By unifying dependencies shared across multiple services (for both core and testing dependencies), all packages will be located in one place, which will keep dependencies similar across versions. This will make it possible to open a single PR that will update multiple packages.

What can you expect from this change?

• Installation from source — when installing from the source, use “pip3 install -e .” to ensure dependencies are properly pulled
• Building docker images — we moved docker images to root directory of Amundsen repo
• Makefile actions were adjusted to this change — you can do “make install_deps” OR “make image” to install from the source, OR build docker image respectively. The same goes for all docker-compose files available in monorepo.

Huge shoutout to Mariusz Gorski and the community for this refactor! You can see more details of the implementation here.

Migration to Mode discovery API

The Amundsen Mode dashboard integration has now been migrated over to the newly performant Mode discover API. 🎉 This will greatly improve the experience for those using the Mode dashboard integration.

Big thanks to Junda Yang from Brex and Neha Hystad from Mode for the support.

Initial support to use MySQL as backend metadata store

Thanks to WePay’s Xuan Shen’s contributions, Amundsen now has the initial support to use MySQL as the backend metadata store! This will allow you to use existing MySQL infrastructure as Amundsen’s backend without having to rely on a graph database. See more details here.

Xuan Shen gave a detailed presentation at our July community meeting. We’ll share more details in our next monthly update. ✨

Popular Dashboards

You can now see popular dashboards in Amundsen. You have always been able to see popular tables — we gave the same love to dashboards. BTW, popular here means most queried/most viewed dashboards, not popular based on Amundsen activity.

Thanks to Verdan Mahmood for his contributions here. 🙌

Lineage Graph Visualization

As part of Amundsen’s commitment to building in lineage exploration, we had two additional iterations on the graphical visualization UI for table lineage. We brought in atlas proxy support as well as multiple stability improvements following direct community feedback. Our next steps are to enter a brief period of aggregating community use cases and shaping the future of the lineage feature UX.

Huge props to Boyan Bonev for spearheading these UI/UX iterations!

Announcements

📣 We’d like to welcome Junda Yang, Amundsen’s latest committer (Junda’s GitHub). Junda was an early engineering member on the Data Tools team at Lyft working on Amundsen. He’s recently done a fantastic job integrating Amundsen with Mode dashboard and its latest discovery API. Looking forward to more contributions from Junda in the future! Amundsen is a Linux Foundation project, with an open governance model. Adding new committers like Junda, and growing our dev community, is an important, ongoing part of our project.

📣 We have deprecated the amundsen-atlas-types repository. Up until now, we maintained a separate package for creating proper entity types in Atlas in order to facilitate integration between Atlas and Amundsen. Moving forward, we are still keeping the approach of creating those entity types, BUT, we are moving the code directly to Amundsen repo. Moving forward, amundsen databuilder repo should be used instead of amundsen-atlas-types. Please see the tutorial for more details.

📣 We have a new version of flaskoidc. The new version available on PyPi is 1.0.2 and is backward incompatible. With this new version, we’ve moved away from flask-oidc and now rely on Authlib, there’s no need to maintain a separate clients_secrets.json file anymore. Configuration is cleaner and requires a few environment variables to enable e2e OIDC support. The access token is not being refreshed at the moment — this will be fixed in future releases. More details on how to configure this new version on github.

Coming up next…

ML Feature Discovery in Amundsen

A few months ago, we presented the first version of ML feature discovery in which Mariusz Strzelecki, from GetInData, had built V1 of what ML feature discovery and integration with Feast looks like. More recently during our July community meeting, Allison Suarez Miranda, from Lyft, shared what the next version of ML feature discovery in Amundsen looks like.

Next community meeting

• Date: Thursday, August 5, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/huuaiauw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen monthly update — June 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why Snap chose Amundsen, dbt integration, lineage graph viz, mono repo, and more!

Summary

May highlights from the Amundsen community:

• Why Snap chose Amundsen
• dbt ♥ Amundsen
• Lineage graph viz
• One repo to rule them all
• ML feature discovery in Amundsen
• Answer how to best use this table — with queries in Amundsen
• Sydney DataEng meetup

All that and more details below!

Check out last month’s highlights here.

Don’t forget: Join our Slack community at slack.amundsen.io. We can’t wait to meet you!

Why Snap chose Amundsen

This month, we heard from Justin Jordan, a developer on Snap’s analytics engineering team, about why his team chose Amundsen and how they use it to solve their problems with data discovery, specifically metrics discovery metadata.

Problem

Due to high growth, Snap ran into problems related to metrics discovery. With the number of metrics accelerating every year, it became difficult to know what metrics existed and how to use them. Challenging questions arose: Who owns the metrics? Where are they surfaced? What do they mean? Where is their data stored?

Solution

Justin and his team chose Amundsen to build Snap’s metric catalog. This catalog allows teams to search for metrics and definitions. This is a huge step forward — now teams can see all relevant technical and business metadata for their metrics. They can manage the entire metric lifecycle with one source of truth.

Check out Justin’s presentation below to hear him explain why Amundsen was chosen and what’s next for his team.

dbt ♥ Amundsen

dbt has enabled thousands of analysts and analytics engineers to transform data in their data warehouse. Many Amundsen users are also heavy dbt users. To that effect, we have built a deep integration with dbt ingesting rich metadata like lineage, tags, and descriptions from dbt into Amundsen.

Thanks to contributions from Grant Seward, Amundsen now integrates with dbt’s core metadata. There’s still a lot to be done here. Join Amundsen Slack — we’d love your feedback.

Lineage Graph Viz

Amundsen already has support for lineage. However, the default way of viewing lineage thus far was a list-based view, which includes downstream and upstream tabs. Amundsen has been working on a graphical view of lineage. Thanks to Knowl Baek for designs and to Verdan Mahmood for building the first beta version of the graph viz.

Amundsen lineage currently supports Neo4j proxy and support for Atlas proxy is coming soon.

We also want to highlight a common question from the community — how do you enable lineage on the front end? Verdan provides the answer in our community meeting recording here.

One repo to rule them all

As you know, Amundsen has a microservice-based architecture. For more flexibility, we started with one code repo for each of the four services — frontend service, metadata service, search service, and databuilder (which is a library not a service, but you get the idea). We learned from community and developer feedback that having different code repositories was a big pain, especially when changes needed to be coordinated across repositories.

We heard you!

We decided to move Amundsen code to one repo. If you develop on Amundsen, you should switch all your development to the new main branch. If you deploy Amundsen, this change doesn’t have much impact on you since the Python packages remain unchanged.

ML feature discovery in Amundsen

It’s happening! For a long time, many of us have wanted to integrate our ML projects in Amundsen. Feature engineering is one of the most time-consuming steps for ML engineers, and Amundsen is here to help. Allison Suarez Miranda and Knowl Baek proposed an RFC for indexing ML projects in Amundsen. The goal is for ML features to be indexed as first-class citizens just like tables, dashboards, and people — providing easy discoverability and ultimately, greater model re-usage and overall productivity.

Answer how to best use this table — with queries in Amundsen

Another exciting proposal for our roadmap, this time from Grant Seward — the ability to index queries and their individual components. Queries are one of the most common ways that Amundsen users interact with data. There is a significant amount of latent information that exists within these queries, which can be useful for users to discover. What does indexing queries mean for you? This means Amundsen will be able to answer key questions such as:

• What tables are most commonly joined to this table and how?
• Are there any filters (where clauses) that are always applied when accessing this table?
• What real-world queries are being used on this table?
• What queries are relevant now, given the recent queries that have been executed?

This will help you discover how to use your data.

Sydney DataEng meetup

Peter Hanssens presented at the Sydney DataEng meetup! He spoke about Amundsen/Neptune/CDK. See video below

Coming up next…

Shared dependencies across microservices

We’re continuing to work on making Amundsen easy to install and develop. After monorepo, we are working on unifying shared dependencies across various microservices. The goal is to avoid any potential mismatch between libraries shared between frontend, metadata, and search services. You can read more details here.

Dashboard support in Atlas metadata proxy

Amundsen was the first data catalog to add support for dashboards — reducing the duplication of analytical work as well as easing with change management. We currently support Tableau, Mode, Redash, and Superset. Dashboards only worked when you used the neo4j backend, we are now working on making dashboards work for Amundsen users who use the Atlas backend.

Next community meeting

• Date: Thursday, July 1, 9am Pacific, 12pm Eastern, 6pm Central Europe
• Add to your calendar: https://evt.to/odsidaaw

Join us on Slack: slack.amundsen.io

Subscribe for periodic updates: Medium & Twitter

Curated with ❤ by Stemma

Amundsen Monthly Update — May 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.

April 2021 — Data lineage, AWS Neptune support, Amundsen in top 10 data OSS projects, and more!

Summary

A quick summary for April in Amundsen:

• BYOL: Bring your own lineage! We improved data lineage adding native support for ingesting, storing and serving table and column level lineage natively in Amundsen
• Use AWS Neptune as Amundsen backend
• Deep-dive on ML feature discovery with Feast integration
• Use Redash client to preview data in Amundsen
• Amundsen + Atlas integration gets a new look!
• Amundsen rated in the top 10 most popular data open-source projects!

All that and more details below!

Don’t forget: Join our Slack at slack.amundsen.io

Lineage in Amundsen

Amundsen has had support for lineage for a while but it required having a separate lineage service to serve lineage relationships. We have added native lineage support in Amundsen so you can ingest lineage metadata (both at table and column level) straight into the graph backend. This means if you already have lineage relationships in your organization, you can very easily ingest them into Amundsen. Once ingested, this is what the UI will look like. Check out the model and sample scripts!

Note: You still have to do the work of parsing lineage yourself. More on that in the next update!

AWS Neptune as Amundsen backend

We added support for AWS Neptune as a backend for Amundsen graph. This adds a third option for those deploying Amundsen to the already existing two — Neo4j and Apache Atlas.

Hear the real story behind adding AWS Neptune support in the April community meeting. Check out the code here.

Not sure which backend to use?

Neo4j is the default, the most commonly used, and the most tested backend. Read more about backend choice in our FAQs.

ML Feature discovery (aka Feast integration)

Feast + Amundsen integration has existed since late 2020. We did a deep-dive into how it works in the April community meeting.

Here’s a related blog post from Mariusz:

Machine Learning Features discovery with Feast and Amundsen

Check out the code here.

Atlas client gets a new look!

Amundsen integrates with Atlas — making is super easy for those who already have an existing Atlas install to provide a great data discovery experience with Amundsen using the same metadata.

For a variety of reasons (which we won’t get into here), we were using a forked Python client for Atlas. As of April we are using the official Python client for Atlas! Check it out!

So, what does it mean for you? Well, it brings in performance improvements and supports linking both tags and badges in Amundsen with Atlas glossaries and classifications respectively. Some times to innovate through, we have to let go of the past. So, It is a breaking change for Atlas Proxy users. If you are one of those users, all of your Atlas classifications will be treated as badges, not tags.

Redash preview client

Amundsen allows a user to see preview of recent data from the table if they have permissions (configured in the BI tool). We have supported preview from Apache Superset, BigQuery and Dremio for a while. Last month, we added support for Redash preview client. So, if you are a Redash and Amundsen user, you can see data preview in Amundsen, using the Redash configured perms! Check out docs!

Amundsen: Top 10 Most Popular OSS data projects of 2021!

Data Council ran an OSS Data Tools community survey on most popular open-source projects of 2021!

Top 10 were: dbt, Airflow, Superset, Dagster, Trino, Prefect, Great Expectations, Spark, Amundsen, and Flink!

We are grateful to be paired with such great and impactful projects. You can read more about the survey in this April blog post from Pete Soderling.

Next up

In the next update, we will share more on the work done in May, including:

• Data Lineage improvements — including a graphical UI
• dbt integration
• Monorepo — one repo to rule them all!
• Why Snap chose to use Amundsen

Keep in touch!

Join us on Slack: slack.amundsen.io

Next community meeting: Add to your calendar: Apple Google Office 365 Thursday, June 3rd at 9am PT, 6pm Central Europe.

Subscribe for periodic updates: Medium or Twitter to receive periodic updates.

Curated with ❤ by Stemma

Amundsen Monthly Update — April 2021 was originally published in amundsen-io on Medium, where people are continuing the conversation by highlighting and responding to this story.