Databricks DABs CI/CD: 4 Ways to Automate Your Deployments

job_runs: the hidden DABs feature that solves everyday CI/CD problems.

Thanks to Kristof Molnar for helping write the article.

You can access the article on the SunnyData blog for free.

In CI/CD, I always had to run something after deploying a job and resources, like:

• Initial setting of some objects,
• DML operations to convert config JSON to UC Delta when JSON changes,
• Log something after deployment,
• Validate job run and roll back if failing.

This is now much easier because we have job_runs in DABs, which are perfect for this.

“We’re working towards supporting more and more CI/CD use cases natively with DABs, and this feature brings us one step closer to a fully featured Databricks CI/CD offering. “

Kristof Molnar, DABs Product Manager

Let’s look at some examples.

Initial setup

Some deployments need initial settings before normal processing starts. For that, initialize job_runs.

It gives us “run on initial deployment” behavior, but not a permanent exactly-once…