<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hosea Kidane on Medium]]></title>
        <description><![CDATA[Stories by Hosea Kidane on Medium]]></description>
        <link>https://medium.com/@hoseatk97?source=rss-feffbedbe778------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gtUZoGo0FxJjNn56jn2u6w.jpeg</url>
            <title>Stories by Hosea Kidane on Medium</title>
            <link>https://medium.com/@hoseatk97?source=rss-feffbedbe778------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:00:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hoseatk97/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Streaming with SQL on Databricks — Why you should and How to do it]]></title>
            <link>https://medium.com/@hoseatk97/streaming-with-sql-on-databricks-why-you-should-and-how-to-do-it-151ba75eac7b?source=rss-feffbedbe778------2</link>
            <guid isPermaLink="false">https://medium.com/p/151ba75eac7b</guid>
            <dc:creator><![CDATA[Hosea Kidane]]></dc:creator>
            <pubDate>Wed, 10 Jan 2024 16:02:24 GMT</pubDate>
            <atom:updated>2024-01-12T01:11:36.089Z</atom:updated>
            <content:encoded><![CDATA[<h3>Streaming with SQL on Databricks — Why You Should and How to Do It</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*_4vXsVnywKptAXib6Uqcng.png" /></figure><p>In this article, we’ll explore why and how to use SQL to simplify streaming pipeline development and management with Databricks’ Delta Live Tables (DLT).</p><p><a href="https://docs.databricks.com/en/delta-live-tables/index.html#what-is-delta-live-tables">Delta Live Tables (DLT)</a> is a managed service from Databricks designed for building and maintaining reliable data pipelines for batch and streaming use cases. It uses Spark as the underlying technology and abstracts the complexities of <a href="https://spark.apache.org/streaming/">Structured Streaming</a> to make streaming more user-friendly.</p><p>We will split our discussion into two parts. First, we’ll examine the benefits of streaming with DLT. In the second half, we’ll delve into practical examples, with code snippets from an end-to-end streaming project.</p><p>Let’s get right into it!</p><h3>Benefits of Streaming with Databricks Delta Live Tables</h3><h4>Democratization of Streaming Pipelines</h4><p>DLT makes Spark Structured Streaming more accessible and manageable for Data Engineers, Scientists, and Analysts by enhancing Spark SQL and tackling challenges like checkpointing and state management.</p><ul><li><strong>SQL-centric Approach:</strong> DLT enables users to define data pipelines using <a href="https://docs.databricks.com/en/delta-live-tables/sql-ref.html">SQL</a> queries, reducing the need to learn Python/Scala or a streaming-specific framework like Apache Flink.</li><li><strong>Low Latency Streaming:</strong> DLT allows reading directly from sources like Kafka, Kinesis, or Event Hubs. This minimizes architectural complexity and lowers <a href="https://www.databricks.com/blog/latency-goes-subsecond-apache-spark-structured-streaming">end-to-end latency</a>.</li><li><strong>Reduced operational complexity:</strong> DLT abstracts away the details of checkpoint and state management, ensuring fault tolerance without requiring the user to configure checkpoints manually.</li></ul><h4>Data Quality and Monitoring Features</h4><p>Another challenge with streaming data is its complexity in terms of debugging and testing. DLT provides mechanisms to address data quality issues, thereby helping decrease system downtime.</p><ul><li><strong>Constraints and Expectations</strong>: These features provide a way to define rules or conditions that the data must meet. For example, we can define a constraint that expects a particular column must always contain non-null values or that the values in a numeric column must fall within a specified range. If records violate our <a href="https://docs.databricks.com/en/delta-live-tables/expectations.html#what-are-delta-live-tables-expectations">expectations</a>, instead of always failing the pipeline, we can automatically drop those records, log them, or quarantine them for manual review while allowing the pipeline to operate uninterrupted.</li><li><strong>Lineage Visualization:</strong> DLT’s lineage graph provides a clear and intuitive representation of the data pipeline, showing how data moves from source to destination. This helps us quickly understand the overall structure and dependencies within the pipeline which helps to pinpoint where issues are occurring in the data stream. We can also see how changes in one part of the pipeline might affect other components, helping in making informed decisions about modifications or updates.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_i3wvXvXepkA3RTVOFzQKA.png" /></figure><h4>Enhanced Performance and Scalability</h4><p>DLT pipelines are powered by the Spark engine, which is known for its performance in handling large-scale data processing. However, there are additional key features and optimizations specific to DLT pipelines.</p><ul><li><a href="https://www.databricks.com/blog/2023/04/14/how-we-performed-etl-one-billion-records-under-1-delta-live-tables.html">Performance and Scalability:</a> Streaming data often experiences fluctuating input rates, with periods of intense activity followed by quieter times, making it difficult to predict and efficiently allocate compute resources. <a href="https://docs.databricks.com/en/delta-live-tables/auto-scaling.html">Enhanced Autoscaling</a> optimizes cluster utilization based on workload volume and improves upon traditional auto-scaling by effectively managing tough-to-predict streaming workloads.</li><li><strong>Automated Maintenance:</strong> DLT automatically conducts maintenance within 24 hours after a table update. This includes an <a href="https://docs.databricks.com/en/sql/language-manual/delta-optimize.html">OPTIMIZE</a> operation that organizes and compacts table data, followed by a <a href="https://docs.databricks.com/en/sql/language-manual/delta-vacuum.html">VACUUM</a> process to remove outdated table versions. This default sequence ensures tables are efficiently maintained without manual input and optimizes query performance and cost.</li></ul><p>I hope you’re convinced! In the next part, I will provide examples of how to perform common transformations and demonstrate streaming best practices using DLT SQL.</p><h3>DLT Streaming Examples</h3><h4>Ingestion</h4><p>Let’s start with ingestion! Databricks supports bringing in data directly from message buses into a “Bronze” or “Raw” table in Databricks with SQL. The following example is one way of ingesting with the <a href="https://docs.databricks.com/en/sql/language-manual/functions/read_kafka.html#read_kafka-table-valued-function">read_kafka</a> function.</p><pre>CREATE OR REFRESH STREAMING LIVE TABLE delta_stream1<br>AS<br>SELECT *<br>  FROM read_kafka(<br>    bootstrapServers =&gt; &#39;kafka_server:9092&#39;,<br>    subscribe =&gt; &#39;ride_events&#39;<br>  );</pre><h4>Interval Joins</h4><p>Now on to joins! Interval join in Spark SQL allows us to join two streaming tables based on a time constraint.</p><p>A regular join assumes that the data is already aligned, which is often not the case with data streams. Interval joins can handle this by joining records based on time intervals, ensuring that late-arriving data is still accurately joined with its corresponding records.</p><p>Performing a regular join on streaming data is computationally expensive and less efficient, as it would require maintaining large state information to track all possible join candidates. Interval joins are more resource-efficient because they limit the join to records within a specified time frame.</p><p>The following example shows how to implement a stream-stream interval join in DLT.</p><pre>CREATE OR REFRESH STREAMING LIVE TABLE joined_streams<br>PARTITIONED BY (City) <br>AS<br>SELECT ride_start_timestamp, ride_end_timestamp, a.ride_id, ride_length_estimate_mins, city<br>  FROM STREAM(delta_stream1) AS a<br>  JOIN STREAM(delta_stream2) AS b<br>      ON a.ride_id = b.ride_id<br>      AND ride_end_timestamp &lt;= ride_start_timestamp + INTERVAL 1 HOUR;</pre><h4>Setting Data Quality Expectations</h4><p>Expectations and constraints ensure that the data in our table meets certain conditions, such as expectations on specific columns (e.g., non-null, unique values, or specific data ranges).</p><p>This integrated approach means we don’t have to write separate validation scripts or maintain additional data quality tools.</p><pre>CREATE OR REFRESH STREAMING LIVE TABLE model_bias_calculation<br>(CONSTRAINT reasonable_difference EXPECT <br>(difference_in_minutes between -60 and 60) ON VIOLATION DROP ROW) <br>AS<br>SELECT *, (ride_length_estimate_mins - actual_ride_length) AS difference_in_minutes<br>  FROM STREAM(LIVE.joined_streams_ride_length);</pre><h4>Window Aggregations</h4><p>Windowed aggregations allow us to perform time-based aggregations (like counts, averages, and sums) over specific intervals, making it easier to extract insights from streaming data.</p><p>Windowed aggregations are essential for real-time analytics, where we need to continuously update our calculations based on the most recent data, like tracking real-time trends or monitoring in real-time.</p><p>In the following example, we can calculate the average value of a column over the past 30 minutes, updating every minute.</p><pre>CREATE OR REFRESH STREAMING LIVE TABLE average_bias_per_city_per_window <br>CONSTRAINT null_value EXPECT (city IS NOT NULL AND model_bias IS NOT NULL AND newest_timestamp IS NOT NULL)<br>AS<br>  SELECT<br>    city,<br>    AVG(difference_in_minutes) AS model_bias,<br>    window(ride_end_timestamp, &#39;30 minute&#39;, &#39;1 minute&#39;).<br>    end as newest_timestamp<br>  FROM<br>    STREAM(LIVE.model_bias_calculation) <br>  WATERMARK ride_end_timestamp<br>  DELAY OF INTERVAL 0 SECONDS<br>  GROUP BY<br>    window(ride_end_timestamp, &#39;30 minutes&#39;, &#39;1 minute&#39;),<br>    city;</pre><p>Get started with this <a href="https://docs.databricks.com/en/delta-live-tables/tutorial-sql.html">DLT tutorial</a> and check out my <a href="https://github.com/hoseakidane/DLT-feature-engineering-Kinesis">git repo</a> for an end-to-end streaming pipeline created using Kinesis and Databricks’ DLT. It contains a data generator notebook, transformations with DLT SQL, and a feature store notebook.</p><p>Thanks for reading!</p><p>If you have any feedback or just want to get in touch, feel free to respond in the comments below or reach out to me on <a href="https://www.linkedin.com/in/hosea-kidane/">Linkedin</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=151ba75eac7b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Feature Engineering (How-To-Guide & Technical Challenges)]]></title>
            <link>https://medium.com/@hoseatk97/real-time-feature-engineering-how-to-guide-technical-challenges-cec5cd60a15b?source=rss-feffbedbe778------2</link>
            <guid isPermaLink="false">https://medium.com/p/cec5cd60a15b</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[kinesis]]></category>
            <dc:creator><![CDATA[Hosea Kidane]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 10:44:01 GMT</pubDate>
            <atom:updated>2023-03-17T19:51:24.109Z</atom:updated>
            <content:encoded><![CDATA[<p>In a previous <a href="https://medium.com/@hoseatk97/real-time-feature-engineering-with-kinesis-databricks-architecture-689881117fe3">post</a>, I went over the architecture for my real-time feature engineering project.</p><p>This blog post is a comprehensive how-to guide that provides details on implementation and technical challenges that you may encounter in building out a feature engineering project. I will offer practical insights, share key learning moments, recommend ways to optimize, and provide some debugging tips to help you identify common pitfalls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*kWMJ8jqkQGqTbFhpitYhww.png" /></figure><h4>Quick Recap</h4><p>To simulate an API writing records to Kinesis, we will create two notebooks, each with a Spark job generating fake data as a data frame and writing to different Kinesis streams. Using a third notebook, we will read from Kinesis, join and transform the data, and write to Delta. Finally, we will register the feature for offline use in the Databricks Feature Store and create a dashboard for visualization.</p><ul><li><a href="#0435">How to create fake data in data frame format?</a></li><li><a href="#7380">How to write to Kinesis?</a></li><li><a href="#7380">How to read from Kinesis?</a></li><li><a href="#cea4">How to aggregate data from multiple streams?</a></li><li><a href="#69c6">Creating the feature and dashboard</a></li><li><a href="#3dd1">What’s next?</a></li></ul><p>Let’s get right to it!</p><h3>How to create fake data in data frame format?</h3><p>For clarification, it&#39;s not our responsibility to get the live data into Kinesis. But we do have to test our pipeline using some kind of data. With the code below, we can create fake records using a Databricks library called “<a href="https://databrickslabs.github.io/dbldatagen/public_docs/APIDOCS.html">dbldatagen</a>.”</p><pre># define dataframe schema for fake data <br>schema = StructType([StructField(&quot;rideid&quot;, IntegerType(), True),StructField(&quot;ride_length_estimate_mins&quot;, IntegerType(), True),StructField(&quot;city&quot;, StringType(), True),]) <br> <br># Create datagen object <br>ds = (dg.DataGenerator(spark, name=&quot;start_time_fake_data_stream&quot;, rows=100000, partitions=20).withColumn(&#39;ride_start_timestamp&#39;, &#39;timestamp&#39;, begin=&#39;2023-01-02 00:09:00&#39;, end=&quot;2023-12-31 00:00:00&quot;, interval=&quot;5 minute&quot;).withSchema(schema))<br><br># specify a list of values to randomize through for the city column  <br>ds2 = ds.withColumnSpec(&#39;ride_length_estimate_mins&#39;, minValue=10, maxValue=15, random=True).withColumnSpec(&#39;city&#39;, values=[&#39;SF&#39;, &#39;LA&#39;, &#39;NYC&#39;, &#39;Miami&#39;, &#39;Austin&#39;, &#39;Chicago&#39;, &#39;Detroit&#39;,])<br><br># turn the datagen object into a streaming dataframe <br>df = ds2.build(withStreaming=True, options={&#39;rowsPerSecond&#39;: 1})</pre><p>The first notebook will include the first stream with ride-picked-up data. It will look like this:</p><pre>{<br>    &quot;ride_start_timestamp&quot;: &quot;2023-07-10 19:30:00&quot;,<br>    &quot;city&quot;: &quot;NYC&quot;,<br>    &quot;ride_length_estimate_mins&quot;: &quot;10&quot;,<br>    &quot;rideid&quot;: &quot;50&quot;<br>}</pre><p>The second notebook will include the code for the second stream with ride-dropped-off data:</p><pre>{<br>    &quot;ride_end_timestamp&quot;: &quot;2023-07-10 19:37:00&quot;,<br>    &quot;rideid&quot;: &quot;50&quot;<br>}</pre><blockquote><strong>Key Learning Moment:</strong></blockquote><blockquote>Connecting Kinesis to Spark proved challenging because Kinesis is not a supported sink. The complexities became apparent when we needed to stream each line of the DataFrame as its own record. I used the forEach method to split up each row in the DataFrame and write to Kinesis using the Boto3 API. Due to time constraints and already being close to figuring it out, I ultimately stuck to using Kinesis. For those looking to replicate the project, Kafka or MSK are easier tools to write to.</blockquote><blockquote><strong>Debugging tip:</strong></blockquote><blockquote>Because of the distributed nature of how the ForEach function works, it might be difficult to understand the errors that arise here. If your Spark job fails, you can access the Spark UI by navigating to “Clusters” → “Cluster Name” → “Spark UI” → “Web Interfaces” → “Jobs” → “Failed Tasks” → “Task ID.” By examining the stack trace for specific worker nodes, you can locate clues about the error’s origin. Additionally, the Spark UI is helpful for identifying performance bottlenecks and optimizing Spark job execution while the job is running.</blockquote><h3>How do we write to Kinesis?</h3><p>We will need to convert the data frames into a format that Kinesis can understand through the process of serialization. JSON is the simplest serialization format but Avro and Protobuff can be even better options for high-volume data.</p><p>We can’t simply use a .writeStream method and have serialization taken care of, we will need to go through each step manually.</p><p>First, let’s create a new column called “json” that maps columns from the original data frame to their corresponding keys in a dictionary. This “json” column can then be passed to json.dumps function for serialization.</p><pre># convert df into dictionary before inputing into json function<br>df1 = df.withColumn(&#39;json&#39;, create_map(lit(&#39;ride_start_timestamp&#39;),\<br>                    col(&#39;ride_start_timestamp&#39;), lit(&#39;rideid&#39;),col(&#39;rideid&#39;),\<br>                    lit(&#39;ride_length_estimate_mins&#39;),col(&#39;ride_length_estimate_mins&#39;),\<br>                    lit(&#39;city&#39;),col(&#39;city&#39;)))</pre><blockquote><strong>Key Learning Moment:</strong></blockquote><blockquote>I initially implemented the following function to write the data frame rows to Kinesis.</blockquote><pre>def write_to_kinesis(row):<br>    <br>  STREAM_NAME = <br>  REGION_NAME = <br>  aws_access_key_id = <br>  aws_secret_key =<br><br>  payload = json.dumps(row[&#39;json&#39;])<br><br>  kinesis_client = boto3.client(&quot;kinesis&quot;, region_name=REGION_NAME,  aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_key)<br><br>  response = kinesis_client.put_record(StreamName=STREAM_NAME, Data=payload, PartitionKey=&quot;partitionkey&quot;)<br><br>stream = df1.writeStream.foreach(write_to_kinesis).outputMode(&quot;append&quot;).start()</pre><blockquote>It took a couple of minutes for the records to land in Kinesis which almost made me think the function wasn’t working. After getting a better understanding of boto3, I realized that the function was unnecesarly reinitializing the client for every record.</blockquote><p>To solve this problem, we can implement this KinesisSink class, which only initializes the client once and then runs the process method to send the data until the stream terminates. This is more efficient and eliminates our latency problem. The foreach command will loop over each row in the data frame, apply our KinesisSinkclass and append each row (key/value) into Kinesis as a single JSON record.</p><pre>class KinesisSink:<br>  <br>  kinesis_client = None<br>  STREAM_NAME = <br>  REGION_NAME = <br>  aws_access_key_id = <br>  aws_secret_key =<br>  <br>  def open(self, partition_id, epoch_id):<br>    <br>    self.kinesis_client = boto3.client(&quot;kinesis&quot;, region_name=REGION_NAME,  aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_key)<br>    return True<br>  <br>  def process(self, row):<br>    payload = json.dumps(row.asDict(), default=str)<br>    <br>    response = self.kinesis_client.put_record(StreamName=STREAM_NAME, Data=payload, PartitionKey=&quot;partitionkey&quot;)<br>    print(response)<br>  <br>  def close(self, error):<br>    print(&quot;Closed with error: %s&quot; % str(error))<br>  <br>stream = df1.writeStream.foreach(KinesisSink()).outputMode(&quot;append&quot;).start()</pre><blockquote><strong>Debugging tip:</strong></blockquote><blockquote>Go into each stream on the AWS console and click “Data Viewer” to ensure that the data is arriving in Kinesis correctly.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MXuluw-Vkn4Pf_dmgbTmSg.png" /></figure><blockquote><strong>Optimization:</strong></blockquote><blockquote>I’ve hard coded the credentials for the sake of time but its not good practice to do this in production. You can securely manage credentials on Databricks using <a href="https://www.databricks.com/blog/2018/06/04/securely-managing-credentials-in-databricks.html">Secret Management</a>.</blockquote><h3>How to read from Kinesis?</h3><p>Now that we’ve faked our data and loaded it into Kinesis, let’s bring it into Spark to perform our feature engineering work in a new notebook. Since Kinesis is a supported source, we can write the following code to load the data onto Spark. We will need to refactor the code below for each stream.</p><pre>kinesisDF1 = spark \<br>  .readStream \<br>  .format(&quot;kinesis&quot;) \<br>  .option(&quot;streamName&quot;, STREAM_NAME) \<br>  .option(&quot;initialPosition&quot;, &quot;TRIM_HORIZON&quot;) \<br>  .option(&quot;region&quot;, REGION_NAME) \<br>  .option(&quot;awsAccessKey&quot;, aws_access_key_id) \<br>  .option(&quot;awsSecretKey&quot;, aws_secret_key) \<br>  .load()</pre><p>Kinesis data is in byte format, we need a UDF to deserialize the input (‘data’) back to JSON format.</p><pre>my_udf = udf(lambda x: x.decode(),StringType())<br>newdf1 = kinesisDF1.withColumn(&#39;jsonString&#39;, my_udf(&#39;data&#39;))</pre><p>Next, this code is converting the JSON to a string type, and then uses the <a href="https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.functions.from_json.html">from_json</a> function to convert it into a structured data frame with individual columns per our schema.</p><pre>schema = StructType([<br>    StructField(&quot;ride_start_timestamp&quot;, TimestampType(), True),<br>    StructField(&quot;rideID&quot;, StringType(), True),<br>    StructField(&quot;ride_length_estimate_mins&quot;, StringType(), True),<br>    StructField(&quot;city&quot;, StringType(), True),<br>])<br><br>options = {&#39;mode&#39;: &#39;FAILFAST&#39;}<br>cleanDF1 = newdf1\<br>    .selectExpr(&#39;CAST(jsonString AS STRING)&#39;)\<br>    .select(from_json(&#39;jsonString&#39;, schema, options).alias(&#39;cleandata&#39;))\<br>    .select(&#39;cleandata.*&#39;)\</pre><blockquote><strong>Debugging Tip:</strong></blockquote><blockquote>For development, I highly recomend setting the options variable to FAILFAST to immediately stop processing and raise an error if there are any issues processing the JSON. For production, you can use DROPMALFORMED or use PERMISSIVE with a <a href="https://www.databricks.com/blog/2023/03/02/scalable-spark-structured-streaming-rest-api-destinations.html">dead letter queue</a> to handle corrupted records.</blockquote><h3>How to aggregate data from multiple streams?</h3><p>In this section, we will join the two streaming data frames and calculate the actual length of the ride. Then, using a <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#types-of-time-windows">sliding window</a>, we will analyze the difference between the estimated and actual ride lengths over the past 30 minutes. These results will be grouped by city. At this point, we will have a feature dataset that contains the average difference between the ETA estimation and the actual ride length over the past 30 minutes.</p><p><a href="https://www.databricks.com/blog/2022/08/22/feature-deep-dive-watermarking-apache-spark-structured-streaming.html">Watermarks</a> are particularly important in scenarios where data arrives out of order and we need all the data in place before transforming it. A watermark threshold specifies how long the system waits for late data. It&#39;s especially relevant in this use case where we have two independent streams starting at different times. I’ve opted for a shorter watermark threshold (10 seconds) so that the pipeline will process the data more quickly, reducing memory usage. However, this also increases the risk of incomplete or missing data.</p><pre> # add watermark on both streams <br>cleanDF1_wtmrk = cleanDF1.withWatermark(&#39;Ride_start_time_stamp&#39;, &#39;10 seconds&#39;)<br>cleanDF2_wtmrk = cleanDF2.withWatermark(&#39;Ride_end_time_stamp&#39;, &#39;10 seconds&#39;)</pre><p>After watermarking, we will join the streams on the rideid column.</p><pre>joined_df = cleanDF1_wtmrk.join(cleanDF2_wtmrk, on=&#39;rideid&#39;, how=&#39;inner&#39;)</pre><blockquote><strong>Optimization:</strong></blockquote><blockquote>When performing a join operation, you should add a time range condition to limit the time range of other events that each event can join against. With this, the engine will automatically calculate state limits and drop old events accordingly. This will limit the amount of data that needs to be processed and retained in memory, improving performance and reducing resource usage.</blockquote><p>Then we can add a new column to the joined data frame that records the difference between the ETA and the actual ride length, thus representing the model’s bias. (*include table output*)</p><pre>df4 = joined_df.withColumn(&#39;difference_in_minutes&#39;, joined_df[&#39;ride_length_estimate_mins&#39;] - joined_df[&#39;actual_ride_length&#39;])</pre><p>The final transformation utilizes a sliding window to analyze the model bias over the past 30 minutes, update every minute, and group the results by city. The “Ride_end_timestamp” column will be the key column here.</p><pre>windowed_df = df4\<br>  .withWatermark(&#39;ride_end_timestamp&#39;, &#39;20 seconds&#39;)\<br>  .groupBy(window(df4.Ride_end_time_stamp, &#39;30 minutes&#39;, &#39;1 minute&#39;), df4.city)\<br>  .avg()</pre><h3>Creating the feature and dashboard</h3><p>This section will cover writing the feature dataset to Delta. Delta will be the underlying table for the Databricks Offline Feature Store and for our visualization dashboard.</p><h4><a href="https://docs.databricks.com/dbfs/mounts.html"><strong>Mount</strong></a><strong> S3 bucket to DBFS</strong></h4><p>First, let’s mount using the following function. This will create a link between our workspace and cloud object storage (S3). Mounts work by creating a local alias under the /mnt directory that stores bucket information and security credentials in DPFS.</p><pre> def mount_to_dbfs():<br>  <br>   access_key = <br>   secret_key = <br>   encoded_secret_key = secret_key.replace(&quot;/&quot;, &quot;%2F&quot;)<br>   aws_bucket_name = &quot;checkpointbucket&quot;<br>   mount_name = &quot;hk-uber-bucket-checkpoint&quot;<br>  <br>  # Check if the mount exists<br>  mountExists = False<br>  for item in dbutils.fs.ls(&quot;/mnt/&quot;):<br>      if str(item.name) == r&quot;/mnt/{mount_name}&quot;:<br>          mountExists = True<br>      <br>      # drop if exists to refresh credentials<br>      if mountExists:<br>        dbutils.fs.unmount(r&quot;/mnt/{mount_name}&quot;)<br>        mountExists = False<br><br>      # Mount a bucket<br>      if not (mountExists):<br>          dbutils.fs.mount(f&quot;s3a://{access_key}:{encoded_secret_key}@{aws_bucket_name}&quot;, f&quot;/mnt/{mount_name}&quot;)<br>          )<br>          print(&quot;Bucket mounted&quot;)<br>      else:<br>          print(&quot;Bucket already mounted&quot;)<br><br>directory = mount_to_dbfs()</pre><h4>Write to Delta</h4><p>After <a href="https://docs.databricks.com/storage/amazon-s3.html">mounting</a> the s3 bucket to Databricks File System (DBFS), we can use the following code to stream to Delta.</p><pre>(windowed_df3.writeStream\<br>  .format(&quot;delta&quot;)\<br>  .outputMode(&quot;append&quot;)\<br>  .option(&quot;checkpointLocation&quot;, &#39;dbfs:/mnt/hk-uber-bucket-checkpoint&#39;)\<br>  .toTable(&#39;uberdataset&#39;)\<br>)</pre><ul><li>The <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#starting-streaming-queries">checkpoint</a> location is specified as an S3 bucket. If the stream terminates for any reason, the pipeline will refer to the checkpoint location to determine where to resume processing from. This ensures fault tolerance and exactly-once processing.</li><li>We will use the <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#handling-late-data-and-watermarking">‘append’ output mode</a> since we only need to output new rows to the Delta table. The engine maintains a partial count internally for each window while waiting for late-arriving data. Once the watermark threshold has passed, it will append the results to the table.</li></ul><blockquote><strong>Optimization:</strong></blockquote><blockquote>We can use <em>MERGE INTO</em>to update and insert data into Delta. The underlying data must be partiontioned appropriately or latency can be greatly increased. Check out this <a href="https://canadiandataguy.medium.com/using-spark-streaming-to-merge-upsert-data-into-a-delta-lake-6d34c827a892">article</a> and this <a href="https://www.youtube.com/watch?v=2Iy5S0Hf4XM&amp;t=220s&amp;ab_channel=AdvancingAnalytics">video</a> on how to implement.</blockquote><p>The Delta table can now be registered to the Databricks Feature Store for offline model training. The “city” will be the primary key and the “bias average” will be the value.</p><pre># register an exisiting Delta table as as a feature table<br><br>from databricks import feature_store<br>from pyspark.sql.functions import *<br><br>fs = feature_store.FeatureStoreClient()<br><br>fs.register_table(<br>  delta_table=&#39;uber_dataset2&#39;,<br>  primary_keys=&#39;city&#39;,<br>  description=&#39;ETA_bias_by_city&#39;<br>)</pre><blockquote><strong>Debugging tip:</strong></blockquote><blockquote>The “window” column cannot be written to feature store because its not a <a href="https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/feature-tables">supported data type</a>. Use a CTAS statement to create a new Delta table (uber_dataset2) by selecting the other columns and write that to feature store.</blockquote><blockquote><strong>Optimization:</strong></blockquote><blockquote>For online model inference, the feature can be streamed to one of Databricks’ online store providers such as DynamoDB. More details are available in the <a href="https://docs.databricks.com/machine-learning/feature-store/publish-features.html">Databricks documentation</a>. Another option would be to <a href="https://docs.databricks.com/machine-learning/feature-store/publish-features.html">publish</a> from the offline store to the online store.</blockquote><p>Then we can query the delta table using Databricks SQL with the following snippet:</p><pre>select first(avg_diff_btwn_estimate_and_actual) as Model_Error_Mins, city, max(Ride_end_time_stamp) as most_recent_timestamp <br>from hive_metastore.default.uberdataset2<br>group by city<br>order by most_recent_timestamp desc;</pre><p>And Ta-da! We have a fully functional real-time dashboard created using DB SQL Serverless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PdIGdMP_qVdUENopFpAGxw.png" /></figure><p>That’s it! We just went over how to implement, debug and optimize a Kinesis + Databricks streaming pipeline. But the work doesn’t stop here.</p><h3>What’s Next?</h3><p>To be production ready, one critical missing piece is observability. We need to answer questions like “is our pipeline still working?” and “are we producing the feature within our SLA?” This can be done with <a href="https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-incremental-data.html">Delta Live Tables</a>, which manages monitoring, alerting, data quality, and error handling with just a few changes to our code. I will be implementing DLT and the online feature store in a follow-up article. Stayed tuned!</p><p>If you have any questions, feel free to reach out to me on <a href="https://www.linkedin.com/in/hosea-kidane/">LinkedIn</a> or ask questions in the comment section!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cec5cd60a15b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Feature Engineering with Kinesis & Databricks (Architecture)]]></title>
            <link>https://medium.com/@hoseatk97/real-time-feature-engineering-with-kinesis-databricks-architecture-689881117fe3?source=rss-feffbedbe778------2</link>
            <guid isPermaLink="false">https://medium.com/p/689881117fe3</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[kinesis]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <dc:creator><![CDATA[Hosea Kidane]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 10:43:04 GMT</pubDate>
            <atom:updated>2023-03-17T18:48:21.124Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lFaxsxIEDTvPEwONwWwu6A.png" /></figure><p>This blog post will discuss the architecture, requirements, and tradeoffs for a real-time calibration feature built using Kinesis and Databricks. The end goal of this project is to enhance the accuracy of a hypothetical ETA predictor model utilized by popular ride-sharing companies like Uber and Lyft. Check out the implementation guide <a href="https://medium.com/@hoseatk97/real-time-feature-engineering-how-to-guide-technical-challenges-cec5cd60a15b">here</a>.</p><p>There are two reasons why I decided to undertake this project. First, it provided me with a great opportunity to gain hands-on experience working with Spark and AWS cloud infrastructure. Second, I am fascinated by the process of machine learning and wanted to understand how to seamlessly integrate multiple big data tools for feature development.</p><h3>ML Context</h3><p>We know that machine learning uses existing data to build a model for predicting future outcomes. The process of preparing raw data for machine learning is called feature engineering, and the outputs of this process are called features, which are the building blocks of the model.</p><p>Another important term is bias correction. “Bias” is the systematic error in a model that results in predictions consistently different from the true values. In this project, we will build a real-time feature that tracks the bias of a model.</p><p>Over time, inaccurate ETAs lead to reduced trust and satisfaction in the platform as well as higher churn rates. Providing accurate ETA predictions will greatly improve the end-user experience and increase customer retention.</p><h3>Requirements</h3><p>Our customer is a Data Scientist that has two kinesis streams consisting of rides picked-up and rides dropped-off data. He/She has requested a feature that calculates the ETA model’s bias, which contains the average difference between a model’s estimated ride length and the actual ride length over the past 30 minutes updated every minute, and grouped by region. They have the following stipulations:</p><ul><li>Offline persistence for model training available in under 30 minutes</li><li>Online persistence for real-time model training and inference available in under 5 minutes</li><li>Needs to handle about 3000 rides per second to account for peak times</li><li>Scalable feature retrieval infrastructure to handle thousands of concurrent requests with sub-1 second latency</li><li>A dashboard for feature visualization that allows Data Scientists to gain quick insights</li></ul><h3><strong>Architecture</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*kWMJ8jqkQGqTbFhpitYhww.png" /></figure><p>Our architecture for this use case is pretty simple. We will aggregate data from two Kinesis streams and transform it in Spark. Then we will write to Delta, which stores the data and checkpoint information in S3 and serve as the foundational layer for our Databricks Feature Store and DB SQL Dashboard.</p><h4>Pipeline Capacity &amp; Latency</h4><p>Writing to <a href="https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/delta-lake">Delta</a> is the main limiting factor for the speed of this streaming pipeline. Underlying the Databricks Offline Feature Store are Delta tables which seamlessly integrate both components so there won’t be any additional latency there.</p><p>We can then <a href="https://docs.databricks.com/machine-learning/feature-store/publish-features.html">publish</a> the feature to an online feature store through a provider like DyanmoDB. I’ve estimated that all of this can be completed end-to-end in under 2 minutes with room to spare. And with autoscaling clusters, our architecture can scale up and down to meet ride demand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*ctvW_dRJA2Z0IIZfTeOU3w.png" /></figure><h4>Feature Retrieval Latency</h4><p><a href="https://docs.databricks.com/machine-learning/feature-store/online-feature-stores.html">Databricks’ Online Feature Store</a> has typical latency of under 10 milliseconds depending on the provider. When the ETA model is used for online inference, it will automatically retrieve the feature from the feature store and calibrate the ETA model in real-time.</p><h4>Observability</h4><p>While out of scope for this project, observability through monitoring and alerting is crucial to building a reliable pipeline. This can be accomplished with <a href="https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-incremental-data.html">Delta Live Tables</a>. I’ll be implementing this at a later time.</p><h4><strong>Visualization</strong></h4><p>Finally, using DB SQL, we can query our Delta tables to create a dashboard for visualization. The dashboard can be updated as frequently as every second if necessary. With the serverless option, we get access to instant compute and leave the cluster management to Databricks. The end result is below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PdIGdMP_qVdUENopFpAGxw.png" /></figure><h3><strong>Tradeoffs</strong></h3><p>When making technical decisions, there are always tradeoffs to consider. Here are some of the main ones:</p><ul><li><strong>Batch vs Stream processing:</strong> I decided on stream processing because updates to the ETA model need to be real-time to account for random events such as accidents and traffic.</li><li><strong>Sliding window vs tumbling window</strong>: We will be creating the main transformation for our feature with a <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#types-of-time-windows">window operation</a>. Sliding windows are better suited for real-time data analysis while tumbling windows are better suited for summarizing data over fixed intervals. We want to ensure that errors were updated as soon as possible, rather than waiting for a window period to conclude.</li><li><strong>Short watermark threshold vs longer threshold: </strong>The <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#handling-late-data-and-watermarking">watermark threshold</a> in this context refers to the waiting period defined for late-arriving data.<strong> </strong>Shorter watermark thresholds reduce latency but can lead to incomplete data or data being discarded. I’ve chosen to reduce delays in the pipeline with a 10-second threshold.</li></ul><h3>What’s next?</h3><p>In a <a href="https://medium.com/@hoseatk97/real-time-feature-engineering-how-to-guide-technical-challenges-cec5cd60a15b">follow-up article</a>, I will discuss the implementation and technical challenges of this project.</p><p>Feel free to reach out to me on <a href="https://www.linkedin.com/in/hosea-kidane/">LinkedIn</a> or ask questions in the comment section!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=689881117fe3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>