Azure Custom Application Logging
Part 2 ADF First Use Case
Let’s consider a use case for an ETL pipeline in ADF that needs to copy and deliver some incoming data files landed in an Azure storage account container. Then copy that validated data into the Azure Data Lake in near real-time. This pattern will also be capturing status, metadata, and metrics into our new custom app log table in Log Analytics; which can then be aggregated and published on an Azure Portal or PowerBI dashboard.
Azure Data Factory is offered as a standalone service in Azure Cloud and as an embedded (bound) service in Microsoft Fabric. Microsoft encourages you to try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting.
Previously, I would use Azure ADF standalone for most low-code data copy activities as described in this Microsoft documentation.
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview
Microsoft suggests these basic steps to copy data from a source to a sink using a standalone ADF service that runs the Copy activity performs these steps:
- Reads data from a source data store.
- Performs serialization/deserialization, compression/decompression, column mapping, and so on. It performs these operations based on the configuration of the input dataset, output dataset, and Copy activity.
- Writes data to the sink/destination data store.
However, we will add additional steps and activities to enhance this workflow:
1.b Include schema validation as an ingestion gate.
2.b Deserialization or decompression
Note: 2.b process step reveals ability to match column mapping. This is the step that allows for active analysis and sending process flow log and error messages. This is also the opportunity to generate any custom events as needed.
Our copy strategy uses more steps than the Microsoft Basic tutorial shows. Our pattern for the ADF pipeline calls a Databricks Notebook for a schema check as a “Quality Gate” before invoking the copy activity, or not.
Get B. Bragg’s stories in your inbox
Join Medium for free to get updates from this writer.
The actual schema validation code can be a Pyspark script that extracts the incoming file schema for comparison and processing with a “Rules Engine” module.
ADF pipelines can call a Databricks notebook to perform “compute” type logic and processing, which is the best practice for performance efficiency.
We now have an operational quality gate ADF/Databricks pipeline strategy.
However, how do we respond, collect, and gather process flow logging or error logging?
While in the Databricks context we have the REST API option to call an Azure Logic App, or use the Azure Monitor API to inject operational or error messages into the monitoring stream.
Employing the Logic App strategy allows the capture of status, metadata, and metrics into Logic App in order to write into the custom defined log table in Log Analytics.
The Logic App message schema can of course be a custom design by you that is lean and specific to your application, or you can adopt a template such as the “Common Alert Schema”.
Request Body for JSON Common Alert Schema
I did not use the common alert schema for my workflow logging, but created a separate Logic App with its own custom schema.
The Azure Logic App has an intelligent feature that will create the log table in Log Analytics Workspace if it does not exist; however, best practice would be to have table already mapped and created.
What initiates the ADF pipeline?
The traditional method was to poll for new files, or a scheduler configured according to a known receipt pattern.
The event driven strategy is the preferable method for “near real time” scenarios.
There are Blob storage triggers available for generating an event when a file is written into the landing zone container. (Select the “flush” event vs. “file created” event.) Microsoft has recently corrected this option inside ADF to avoid the previous “double” event triggering issue.
Remember, you can generate event triggers directly from the “Copy Activity” for any pipeline workflow inside the your ADF context before and after switching into the Databricks Notebook context. These features allow for as much (robust) or as little (Lean) workflow error or process logging as you need. It also scales very well.
