Orchestration Overhead - Lakeflow Jobs

Scenario

At The Data Collective we have a dynamic ingestion / orchestration framework that gives us a higher level of flexibility than simply using the natively available features: the option of running notebooks inside for-each loops without specifying the notebook path at design time.

This is hugely significant because it enables us to have a single set of orchestration jobs for all out ingestion patterns. Patterns are templatised notebooks to handle our most common operations, for example:

• Ingesting data from relational databases through a JDBC connection
• Using Auto Loader to ingest files
• Run any notebook
• Run a declarative pipeline

These various patterns operate with varying sets of parameters (or without a single parameter) and having totally different executor jobs for each type would make the solution unnecessarily complicated.

In a nutshell the extra layer that is executed by the for-each loop is a simple translator / executor that reads from the metadata what notebook is to be executed and runs it with the right parameters utilising

dbutils.notebook.run

How to avoid the Overhead?

That extra layer presents a fairly high overhead that is the price for keeping the flexibility. Is it a good enough deal in terms of performance? Well, that is the question and for us in most cases it is good enough, yet as engineers we strive to minimize that and can’t help revisiting this performance challenge regularly out of the sheer quest towards enhancing our codebase. The possibility is always there that the Databricks platform has evolved and offers any new functionality to utilise towards this direction, given the rate of churning out new features. It’s hard to keep up with those!

However the focus is not on orchestration and the need for this extra layer is still there. As a result we have tried various ideas to work our way around it if possible. Over time these have come up:

• Using .py scripts instead of notebooks
• Experimenting different types of clusters
• Regular emails to Databricks, highlighting the issue and asking about future performance achievements

No — low-hanging — fruits yet:

• .py scripts didn’t make a difference (I am actually relieved because switching to them would have meant losing many of the good troubleshooting features notebooks provide)
• serverless clusters are marginally faster but overall it’s negligible
• no real solution offered by Databricks in focused email conversations

Detailed Analysis

To be able to test the before-mentioned various approaches a mini module was necessary with the following components:

• our usual orchestration objects (jobs and the translator / executor)
• a custom metadata to run a few streams in parallel
• the “test” pattern which is an oversimplified notebook that accepts a few parameters through widgets and does a print of them

First in the jobs definitions I changed from using a job cluster to serverless and in a subsequent step I converted the notebook to a simple .py script to grasp for any possible low-hanging fruit. The harvest was not abundant.

Get István’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Still, the setup gave a good baseline to analyse and quantify the sentiment: “there is too much overhead with orchestration” - how much is too much?

Using Genie code for the detailed analysis the below breakdown of the result were presented:

Overhead in figures

Clearly this is outside the influence of the code developer: no matter how much we enhance that part, it would have almost zero impact. Even if it was possible to make the code run in an instant, 39 seconds instead of 42 wouldn’t get me a standing ovation at any user-group meeting. Maybe some would see the relevance of 42 (many of us do like Douglas Adams) …

Possible Option

Out of curiosity I couldn’t help myself following the chat session with a prompt to see what the Genie assistant would recommend: it wasn’t programmed for loyalty as its suggestion was to ditch the natively available ForEach task and achieve the same with custom code, harnessing

ThreadPoolExecutor

In this case the loop is executed in a single notebook instead of the Lakeflow Jobs For Each task. The result?
7 seconds

And the code is fairly simple:

DIY For Each

We only have it as a Proof-of-Concept at this point because otherwise Lakeflow Jobs is a good product and the UI makes it incredibly good to understand, analyse and troubleshoot processes.

This monitoring feature is important in both dev/test and production style environments but in a different way:

• during development having printouts gives good visibility of what’s happening and provides a good overview if things work as expected
• in production debug printouts mean an unnecessary overhead most of the time (that is if your framework operates efficiently ;o)) but when there is an unexpected happening, having the option to quickly and efficiently located where it happened and what it is - that’s crucial

Conclusion

It is slightly counter-intuitive that using custom code such a decrease in execution time is possible. I wonder if such an approach can be integrated natively by Databricks into Lakeflow Jobs. That would mean hitting two birds with one stone: performance and monitoring/troubleshooting with a single tool. Awesome outcome. Fingers crossed it will become a reality soon!