Add Hamilton's post about using Databricks SQL Serverless

Author: rtyler

_posts/2022-06-28-databricks-serverless.md

@@ -0,0 +1,58 @@
+----
+layout: post
+title: "Accelerating Looker with Databricks SQL Serverless"
+tags:
+- looker
+- databricks
+- featured
+team: Core Platform
+author: hamiltonh
+----
+
+We recently migrated Looker to a Databricks SQL Serverless, improving our
+infrastructure cost and reducing the footprint of infrastructure we need to
+worry about! “Databricks SQL” which provides a single load balanced Endpoint
+for executing Spark SQL queries across multiple Spark clusters behind the
+scenes. “Serverless” is an evolution of that concept, rather than running a SQL
+Endpoint in our AWS infrastructure, the entirety of execution happens on the
+Databricks side. With a much simpler and faster interface, queries executed in
+Looker now return results much faster to our users than ever before!
+
+When we originally provisioned our “Databricks SQL” endpoints, we worked
+together with our colleagues at Databricks to ensure [the terraform provider
+for Databricks](https://github.com/databricks/terraform-provider-databricks) is
+ready for production usage, which as of today is Generally Available. That
+original foundation in Terraform allowed us to more easily adopt SQL Serverless
+once it was made available to us.
+
+```hcl
+resource "databricks_sql_endpoint" "endpoint" {
+  name                      = "Looker Serverless"
+  # ...
+  enable_serverless_compute = true
+  # ...
+}
+```
+
+The feature was literally brand new so there were a few integration hurdles we
+had to work through with our colleagues at Databricks, but we got things up and
+running in short order. By adopting SQL Serverless, we could avoid setting up
+special networking, IAM roles, and other resources within our own AWS account,
+we can instead rely on pre-provisioned compute resources within Databricks' own
+infrastructure.  No more headache of ensuring all of the required infra is in
+place and setup correctly!
+
+The switch to Serverless reduced our infra configuration and management
+footprint, which by itself is an improvement. We also noticed a significant
+reduction in cold start times for the SQL Serverless Endpoint compared to the
+standard SQL Endpoint. The faster start-up times meant we could configure even
+lower auto-terminate times on the endpoint, savings us even more on
+unproductive and idle cluster costs.
+
+On the Looker side there really wasn’t any difference in the connection
+configuration other than a URL change. In the end, after some preparation work
+a simple 5 minute change in Looker, and a simple 5 minute change in Terraform
+switched everything over to Databricks SQL Serverless, and we were ready to
+rock! Our BI team is very happy with the performance, especially on cold start
+queries. Our CFO is happy about reducing infrastructure costs. And I’m happy
+about simpler infrastructure!