Databricks SDKs vs. CLI vs. REST APIs vs. Terraform provider vs. DABs

The previous blog post about Databricks Terraform provider vs. Databricks Asset Bundles (DABs) was quite successful, but it didn’t cover all possible application areas. So, there were requests for a follow-up post covering other tools, such as Databricks CLI, SDKs, and REST APIs, and when to use them compared to Databricks Terraform provider and DABs.

Databricks REST API

The Databricks REST API is the foundation for all other tools. All interactions with the Databricks Platform happen via it and you have full control over what you’re doing. But with the great power, you’re now responsible for handling all nuances of API usage:

• Authentication: Multiple authentication methods are supported, but, for example, you need to generate and renew OAuth tokens yourselves.
• Implementation details, like pagination in list API: different APIs use different pagination methods, and you need to understand all the details of each (note: the unification is in progress, but it takes time).
• Error handling: You need to retry the call when you get the HTTP 429 status code (rate limit) and some other situations, or stop processing if you get other, non-retryable errors.
• Some services, such as clusters, model serving, etc., are starting their objects, and you need to wait until they successfully start to declare the success. This could be done by continuous polling, but you shouldn’t overload APIs by polling too often and shouldn’t waste time polling rarely.

Databricks SDKs

Databricks provides a number of SDKs for different languages (officially supported are for Go, Python, and Java languages). All these SDKs are generated from the same source — API specification that describes the whole Databricks REST API surface. Having SDKs generated from the same source has a big advantage — SDKs get new functionality as soon as new/updated APIs are published. Another great thing is that APIs and their usage in different languages are quite similar to each other (taking into account language differences), so it’s easier to switch between different languages.

SDKs solve all the problems described above by providing:

• Authentication — you can authenticate using all supported authentication methods (PATs, Databricks/Azure/GCP user-to-machine and machine-to-machine OAuth, …). You can provide authentication parameters either explicitly when creating an API client, via environment variables, or have a mix of them. SDKs also support using information from configuration profiles defined in Databricks CLI’s configuration file. And when you’re running it from a Databricks Notebook, you don’t even need to specify any authentication parameters — everything will be configured automatically.
• Abstracting away implementation details, such as pagination implementation, you just call Clusters.ListAll and don't worry about what pagination method is used by the specific API.
• Handling retries and errors — SDKs automatically retry the call if it hits rate limits or other conditions that allow the action to be tried again.
• Providing auxiliary methods, such as GetByName to get an object by its name or WaitGetClusterRunning to wait for a cluster creation - all these methods are generated automatically for most services. But SDKs also include manually written auxiliary methods, such as Clusters.SelectNodeType or Clusters.that, SelectSparkVersion allow the building of cloud-agnostic code (similar to this Terraform example).

In general, the use of Databricks SDKs is very simple:

• you create an instance of workspace or account client
• you use methods of specific service exposed by the client — clusters, jobs, etc.

Here is a simple example of listing all jobs in the workspace using Python SDK (authentication parameters will be taken from the notebook environment or environment variables):

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
job_list = w.jobs.list(expand_tasks=False)

More complex examples could be found in documentation and in Databricks Labs Sandbox repository.

Databricks CLI

Databricks CLI is built on top of Databricks Go SDK and provides an easy-to-use interface to interact with Databricks Platform from the command line (on both workspace and account levels). CLI is also a home for Databricks Asset Bundles that greatly simplify deployment and promotion of the code and other assets to the Databricks Platform.

Get Alex Ott’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Because it’s built on top of Go SDK, it inherits all its capabilities but provides an easier-to-use interface to perform specific tasks — create or start clusters, list jobs, etc. That’s ideal for one-time use, or for scripting. But we still need to take care of providing the right payload, such as JSON-encoded cluster or job specification, etc., the same as for the corresponding REST APIs. These payloads could be quite complex, and not so portable if we talk about references to cluster policies, instance pools, DLT pipelines, or other “external” references. For such cases, it’s better to use DABs or the Databricks Terraform provider to define the environment consisting of multiple objects and deploy them in the right order, with references, etc.

One great part of Databricks CLI is the ability to define a configuration profile — a named entity describing a specific environment — primarily these are authentication parameters, like, host, token, etc., but it’s possible to specify other configurations as well. After the profile is defined we can easily use that configuration by specifying only its name, without the need to specify all parameters together. I.e., it’s easy to export workspace objects (notebooks, workspace files, etc.) from one workspace and import them into another using the following command:

databricks --profile ws1 workspace export-dir -o '/Users/...' local-dir && \
    databricks --profile ws2 workspace import-dir -o local-dir "/..."

Profiles can also be used by SDKs and Terraform providers, making it easy to reuse the same code by specifying environment variables to specify which profile should be used instead of hardcoding configuration in the code or specifying multiple environment variables.

When to use what?

To decide what tool to use I typically ask myself a very simple question — what I want to achieve?

• If I want to define some “environment” (especially complex, consisting of multiple objects, like, a job with multiple tasks of different types), and keep its configuration up to date — then use DABs or Terraform. These tools will take care of tracking what objects are already created, what configuration they have, etc., and make changes if necessary to bring them to the desired state. DABs provide an additional functionality on top of it, like, starting a DLT pipeline or job and wait for its execution (it’s not available in Terraform by default).
• If you need to perform some action — use Databricks CLI or SDKs for the language of your choice. (These actions are typically stateless):
• CLI is ideal for one-time actions, like, start cluster, list jobs, etc. As soon as you need to implement more complex logic, you will start to chain CLI calls using shell, and it will become an unsupported mess (believe me, I wrote and supported huge shell scripts ;-)
• SDKs are ideal for implementing complex logic — you can use the full power of selected programming language with abstractions provided by SDKs. With SDKs, it’s easy to implement custom tasks, i.e., find all scheduled/triggered jobs, and pause/unpause them, or deactivate/reactivate all non-admin users/service principals in the workspace, etc. See more examples in the Databricks Labs Sandbox repository.

The direct use of the Databricks REST APIs should be the last resort due to the need to handle authentication, retries, implementation details (i.e., pagination), etc. yourself. Although there are still cases when you can select to use REST APIs:

• There is no SDK for your language. The best approach would be to raise a request to add one for your language of choice, and use REST API directly until it’s available. With correct design of your programs, you can easily swap your direct implementation with SDKs.
• SDKs don’t provide the necessary functionality yet — typically this happens with APIs that are in the private preview, so the API specification isn’t updated yet. In this case, you can still use CLI and SDKs — they provide a raw interface to REST APIs, handling things like authentication and error handling/retries:
• For CLI the raw interface is available as databricks api commands ().
• SDKs provide an low-level API client that is used by both workspace and account-level clients under the hood. For example, Python SDK has the ApiClient class that could be used to call an arbitrary REST API.

Conclusion

I hope that this blog post will help you identify and start using the right tool for your Databricks automation journey. I would be really grateful for your feedback!

Crosspost from Blogger…