<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kiran PS on Medium]]></title>
        <description><![CDATA[Stories by Kiran PS on Medium]]></description>
        <link>https://medium.com/@iamkiranps?source=rss-13e9dd87ccf3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*X_Zlz7ynVCC3FMjTJM6q6g.jpeg</url>
            <title>Stories by Kiran PS on Medium</title>
            <link>https://medium.com/@iamkiranps?source=rss-13e9dd87ccf3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:31:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@iamkiranps/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Analogy between Feature Engineering and Data Warehousing]]></title>
            <link>https://medium.com/@iamkiranps/analogy-between-feature-engineering-and-data-warehousing-0e22e8641aa2?source=rss-13e9dd87ccf3------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e22e8641aa2</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Kiran PS]]></dc:creator>
            <pubDate>Tue, 22 Jul 2025 21:26:32 GMT</pubDate>
            <atom:updated>2025-07-22T21:26:32.800Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zUGw7esfhoeAu6Ly" /><figcaption>Photo by <a href="https://unsplash.com/@pickawood?utm_source=medium&amp;utm_medium=referral">Pickawood</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>While learning about Feature Store, I found analogy between feature engineering and data warehousing. Even though they are not the same it helped me relate both concepts. [ Just thought of writing an article for my future reference and for readers, just in case if it helps ]</p><p>Drawing parallels between a feature store (commonly used in machine learning systems) and a data warehouse (DWH) helps make the concept more intuitive for people familiar with traditional analytics. Here’s how the analogy can be refined and expanded for greater clarity:</p><p>✅ Refined Analogy: Feature Table = Dimension Table, Training Dataset = Fact Table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*7ap4e4vuZlhUWa26zAwyzQ.png" /></figure><h3>🧠 Example: Retail Purchase Prediction</h3><p>Let’s say you’re building a model to predict whether a customer will buy a product.</p><h4>🟩 Feature Tables (Dimension Tables)</h4><ul><li>customer_features<br> → customer_id, age, loyalty_score, region, account_age_days</li><li>product_features<br> → product_id, category, price, brand</li></ul><h4>🟦 Training Dataset (Fact Table)</h4><ul><li>purchases_train<br> → customer_id, product_id, purchase_timestamp, label</li></ul><p>➡️ Join on customer_id to bring in features from the customer feature table. You may also join product features on product_id to bring category-level information.</p><h3><strong>💡 Bonus Layer: Role of Time</strong></h3><p>Traditional DWHs use slowly changing dimensions and snapshotting to maintain historical integrity.</p><p>Feature stores (like those in Databricks Feature Store or Feast) need point-in-time correctness, meaning, you must use the feature values as they existed at the time of the label to avoid data leakage. They ensure this via feature lookups with timestamps, which is a more automated and rigorous version of historical dimension joining.</p><p>🔄 Mapping Summary</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*97-EIF5-pw1qDgAIVrXzqA.png" /></figure><p>Just remember that feature stores require more time-awareness than typical dimension joins — in ML, when the feature was valid matters a lot (to prevent leaking future data into model training). You’re essentially doing dimension-style joins plus time-aware joins with data versioning — a supercharged DWH pattern designed for ML modeling.</p><h3>Why Feature Stores in Machine Learning Need More Time-Awareness Than Typical Dimension Joins</h3><h3>Key Differences: ML Feature Stores vs. Dimension Table Joins</h3><p>Traditional data warehouses:</p><ul><li><em>Dimension table joins</em> focus on merging descriptive attributes for entities (like customer demographics) with transactional data (fact tables).</li><li>The join typically brings in the “current” or most recent values of those attributes, unless careful snapshotting or slowly-changing dimension (SCD) techniques are used.</li><li>For most analytics/reporting, using the latest dimension value is acceptable.</li></ul><p>ML Feature Stores:</p><ul><li><em>Feature lookups</em> must return feature values as they existed at the exact point-in-time when the event (i.e., prediction target) occurred.</li><li>This prevents data leakage, a serious problem where future information “leaks” into model training, resulting in unrealistically optimistic performance and poor deployment results.</li></ul><h3>What Is Data Leakage and Why Does It Matter?</h3><ul><li>Data leakage happens if the model is trained on features containing information from after the label/event being predicted.</li><li>Example: If you predict churn in March, but join customer tenure as of July (after the customer already churned), you risk leaking future info.</li></ul><h3>How Feature Stores Enforce Time-Awareness</h3><ul><li>Point-in-time correctness: ML feature stores track historical values of features, versioned by timestamp.</li><li>You must retrieve the value for each feature <em>as it existed at the time of prediction</em>.</li><li>This ensures the model, during both training and serving, only “sees” data that would’ve been available at that moment.</li><li>Automated point-in-time join logic: Feature stores (like <a href="https://docs.databricks.com/aws/en/machine-learning/feature-store/time-series#create-a-training-set-with-a-time-series-feature-table">Databricks Feature Store</a>, <a href="https://docs.feast.dev/v0.35-branch/getting-started/concepts/point-in-time-joins">Feast</a>, <a href="https://www.tecton.ai/blog/time-travel-in-ml/#:~:text=That%20is%20exactly%20what%20Tecton,quality%2C%20accurate%20training%20data%20sets.">Tecton</a>) offer APIs to fetch the correct historical snapshot automatically, avoiding accidental leakage.</li><li>Backfilling: When generating a training dataset, the feature store reconstructs what feature values were at each label timestamp, supporting millions of examples efficiently.</li></ul><h3>Example: Churn Prediction</h3><ul><li>Fact Table (Training Data):</li><li>customer_id, event_timestamp (e.g., when churn was observed), churned</li><li>Feature Table (Customer Attributes Over Time):</li><li>customer_id, feature, feature_value, feature_valid_from, feature_valid_to</li></ul><p>ML Pipeline Must Join “As Of” Each Event’s Timestamp:</p><ul><li>When training, for each churn observation on 2024–06–01, only use feature values valid on or before 2024–06–01 for that customer.</li><li>This replicates real-world predictions and eliminates future information bias.</li></ul><h3>Key Requirements Comparison</h3><figure><img alt="Key Requirements Comparison" src="https://cdn-images-1.medium.com/max/751/1*MzSwZWzA4f_r-wsFKY7PQQ.png" /></figure><p><strong>Summary</strong><br>Feature stores require strict time-awareness to avoid leakage: only features available at the time of the label/event should be used during model training and serving. This is much stricter than typical analytics joins and is a foundational requirement for trustworthy ML pipelines.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e22e8641aa2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Framework to manage data quality rules for Delta Live Tables in Databricks]]></title>
            <link>https://medium.com/@iamkiranps/framework-to-manage-data-quality-rules-for-delta-live-tables-in-databricks-90ade86043fe?source=rss-13e9dd87ccf3------2</link>
            <guid isPermaLink="false">https://medium.com/p/90ade86043fe</guid>
            <category><![CDATA[databricks-certification]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-sql]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Kiran PS]]></dc:creator>
            <pubDate>Thu, 13 Feb 2025 23:18:25 GMT</pubDate>
            <atom:updated>2025-02-13T23:30:36.859Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FKq55T9Yb6maC5UL" /><figcaption>Photo by <a href="https://unsplash.com/@nate_dumlao?utm_source=medium&amp;utm_medium=referral">Nathan Dumlao</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article, I will discuss an effective approach to reuse data quality rules across all tables in a DLT pipeline. In a production scenario, when the number of quality rules increase, it is difficult to manage data exceptions in notebooks in the pipeline. At times developers need to update the rules in multiple notebooks which can cause mistake and also may miss to update in some places.</p><p>To tackle this scenario, it is better to maintain the data exceptions rules outside the main notebooks. One such approach is to maintain the data quality rules in a Delta table outside of the pipeline’s target schema.</p><p>To store data quality rules in a separate Delta table for reuse across multiple DLT pipelines, you can follow these steps:</p><ol><li>Create a Delta table to store the rules.</li></ol><pre>CREATE TABLE IF NOT EXISTS data_quality_rules (<br>  rule_id STRING,<br>  table_name STRING,<br>  column_name STRING,<br>  rule_type STRING,<br>  rule_expression STRING<br>)</pre><p>2. Insert rules into the table.</p><pre>INSERT INTO data_quality_rules VALUES<br>  (&#39;rule1&#39;, &#39;sales&#39;, &#39;order_date&#39;, &#39;expect&#39;, &quot;order_date &gt; &#39;2020-01-01&#39;&quot;),<br>  (&#39;rule2&#39;, &#39;customers&#39;, &#39;email&#39;, &#39;expect_or_drop&#39;, &quot;email LIKE &#39;%@%.%&#39;&quot;),<br>  (&#39;rule3&#39;, &#39;products&#39;, &#39;price&#39;, &#39;expect&#39;, &quot;price &gt; 0 AND price &lt; 10000&quot;)</pre><p>3. In your DLT pipeline, load and apply the rules.</p><pre>import dlt<br>from pyspark.sql.functions import col<br><br># Load rules from the Delta table<br>rules = spark.table(&quot;data_quality_rules&quot;)<br><br>@dlt.table<br>def apply_quality_rules(table_name):<br>    # Get rules for the specific table<br>    table_rules = rules.filter(col(&quot;table_name&quot;) == table_name)<br>    <br>    # Apply each rule<br>    for rule in table_rules.collect():<br>        if rule.rule_type == &quot;expect&quot;:<br>            dlt.expect(rule.rule_id, rule.rule_expression)<br>        elif rule.rule_type == &quot;expect_or_drop&quot;:<br>            dlt.expect_or_drop(rule.rule_id, rule.rule_expression)<br><br>    # Return the data with applied rules<br>    return dlt.read(table_name)</pre><p>This approach offers several advantages:</p><p>· Centralization: Storing data quality rules in a separate Delta table allows for centralized management and maintenance of these rules.</p><p>· Reusability: By keeping the rules outside the pipeline’s target schema, they can be easily reused across multiple tables and even multiple pipelines.</p><p>· Flexibility: Providing the schema name as a pipeline parameter allows for easy configuration and adaptation to different environments or use cases.</p><p>· Scalability: This method scales well as the number of tables and rules grows, avoiding repetition in individual DLT notebooks or files.</p><p>· Consistency: Centralizing the rules ensures that all tables in the pipeline are using the same set of quality checks, maintaining consistency across the data.</p><p>· Control: A data engineering lead in the team can take control of the schema of the rules table, so that other wont play with the table and tamper the rules.</p><p>By using this approach, the team can efficiently manage and apply data quality rules across their DLT pipeline, reducing redundancy and improving maintainability of their data quality checks.</p><p>References:</p><ul><li><a href="https://wtdanalytics.com/blog/managing-data-quality-with-delta-live-tables-in-databricks">Managing Data Quality with Delta Live Tables in Databricks</a></li><li><a href="https://docs.databricks.com/en/delta-live-tables/expectations.html">Manage data quality with pipeline expectations</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FwEW8ordPgkA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DwEW8ordPgkA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FwEW8ordPgkA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a6c49bfde46a2e3098ae5573289bb89/href">https://medium.com/media/5a6c49bfde46a2e3098ae5573289bb89/href</a></iframe><p>#databricks #dataengineering #dlt_pipeline #deltatables</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90ade86043fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving Hacker Rank SQL Questions using Pyspark]]></title>
            <link>https://medium.com/@iamkiranps/solving-hacker-rank-sql-questions-using-pyspark-fa7a9fce9081?source=rss-13e9dd87ccf3------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa7a9fce9081</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[pyspark-dataframes]]></category>
            <category><![CDATA[pyspark-training]]></category>
            <category><![CDATA[pyspark-joins]]></category>
            <dc:creator><![CDATA[Kiran PS]]></dc:creator>
            <pubDate>Sat, 09 Nov 2024 02:23:01 GMT</pubDate>
            <atom:updated>2024-11-09T02:23:59.362Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rOimxqm3W6Rs2BPv" /><figcaption>Photo by <a href="https://unsplash.com/@anastasiiachepinska?utm_source=medium&amp;utm_medium=referral">Anastasiia Chepinska</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Using the hacker rank SQL questions, I wrote some Python code, which I’m sharing in this post. This is just for Pyspark and SQL practice. For people working as Python developers and getting ready for in-person interviews, this post would be beneficial.</p><p>A fantastic place to practice SQL questions is <a href="https://www.hackerrank.com/">Hacker Rank</a>. To learn Pyspark, I have obtained data and answered questions from Hacker Rank. The notebook has been run using the Databricks Community free edition platform. This allowed me to experiment using Spark APIs and learn about the Databricks platform.</p><p>All the notebooks are shared on my github page <a href="https://github.com/iamkiranps/HackerRanks">here</a>. If you intend to use the CSV data for each question, you must download it from git and upload it to Databricks dbfs.</p><h3>⚡1 — List the Python objects in the Notebook</h3><p>It is common for you to lose track of the names of the Python objects you have create when you have a lot of them in your notebook. All of the Python objects in the notebook can be located by using the %who magic keyword.</p><figure><img alt="%who lists all the python objects in the notebook" src="https://cdn-images-1.medium.com/max/1014/1*UHg_19obxDAeyREHCWAKoA.png" /></figure><h3>⚡2 — Always define schema</h3><p>Defining the dataset’s schema at all times is great practice. It aids in your comprehension of the data you are dealing with and helps you spot any problems that may arise from the datatype. By repeatedly using the datatype classes assisted me to mind map the Spark APIs.</p><pre>## Create Schemas<br>hackers_schema = StructType([StructField(&#39;hacker_id&#39;, IntegerType(), False),<br>                            StructField(&#39;name&#39;, StringType(), False)])<br><br>submissions_schema = StructType([StructField(&#39;submission_date&#39;, DateType(), False),<br>                                 StructField(&#39;submission_id&#39;, IntegerType(), False),<br>                                 StructField(&#39;hacker_id&#39;, IntegerType(), False),<br>                                 StructField(&#39;score&#39;, IntegerType(), False)])</pre><h3>⚡3 — Use col() function</h3><p>It is a best practice to always use col() function to call a column. When running join queries, it fixes conflicts between datasets with the same column name.</p><h3>⚡4 — Use dataframe .alias() function</h3><p>Use the dataframe.alias() function when creating join queries to help differentiate the dataframe (particularly when a dataframe has a self join).</p><pre>functions_sub = functions_df.withColumn(&#39;rn&#39;, F.row_number().over(Window.orderBy(F.col(&#39;X&#39;), F.col(&#39;Y&#39;))))<br><br>results_df = functions_sub.alias(&#39;f1&#39;)\<br>    .join(functions_sub.alias(&#39;f2&#39;),\<br>        ((F.col(&#39;f1.rn&#39;) &lt; F.col(&#39;f2.rn&#39;)) &amp; \<br>            (F.col(&#39;f1.X&#39;)==F.col(&#39;f2.Y&#39;)) &amp; \<br>                (F.col(&#39;f1.Y&#39;)==F.col(&#39;f2.X&#39;))),\<br>            how=&quot;inner&quot;)\<br>                .orderBy(F.col(&#39;f1.X&#39;))\<br>                    .select(F.col(&#39;f1.X&#39;), F.col(&#39;f1.Y&#39;))<br><br>display(results_df)</pre><h3>⚡5 — Think SQL write Python</h3><p>It is simple to write the solution using a SQL query and then translate the logic into Pyspark APIs if you have a background in SQL, like I do. It would help you in code writing interviews if you could write in PySpark and think in SQL. It would also assist you in investigating SQL functions that are similar to those in Python.</p><h3>⚡6— SQL IN(), NOT IN() Clause</h3><p>Select .. Where IN(Select .. from table) filter condition is a commonly used clause in SQL.</p><pre>select start_date from projects <br>  where start_date not in (select end_date from projects)</pre><p>However, in Pyspark,.isin() takes a list of elements. We cannot pass the result of a dataframe (like SQL). One way to do this is to convert the dataframe to RDD, flatten it, collect the result, and pass it into.isin().</p><pre># Step 1: Get all end_dates<br>end_dates = projects.select(&quot;end_date&quot;).distinct()<br><br># Step 2: Select start_dates that are not in end_dates<br>result = projects.select(&quot;start_date&quot;)\<br>          .where(~F.col(&quot;start_date&quot;).isin(end_dates.rdd.flatMap(lambda x: x).collect()))</pre><p>The flatMap(lambda x: x).collect() part is used to convert the DataFrame column to a list. However,<strong> be cautious</strong> with collect() for large datasets as it brings all data to the driver node. For very large datasets, you might want to consider using a join operation instead.</p><p>Alternative way is to convert the above logic using outer join as below.</p><pre>-- select start_date from projects where start_date not in (select end_date from projects) -- This query can be written as below<br>select a.start_date, b.end_date <br>from projects a left outer join projects b <br>    on a.start_date = b.end_date <br>where b.end_date is null </pre><pre>## Pyspark equivalent of the above SQL<br>start_date_df = projects_df.alias(&#39;a&#39;)\<br>    .join(projects_df.alias(&#39;b&#39;), F.col(&#39;a.start_date&#39;) == F.col(&#39;b.end_date&#39;), how=&#39;left&#39;)\<br>        .where(F.col(&#39;b.end_date&#39;).isNull())\ <br>            .select(F.col(&#39;a.start_date&#39;))</pre><h3>⚡7— Corelated subquery using Pyspark python APIs</h3><p>In my learning experience, Pyspark does not support corelated subquery using Python APIs (it is supported using Pyspark spark.sql(“Select…”) API).</p><p>For example, the below SQL where exists query needs to be written using join() in the Pyspark Python API.</p><pre>with CTE as <br>(<br>select x as X1, y as Y1, row_number() over (order by X, Y) rn from functions_v<br>)<br>select X1, Y1 from CTE f1 where exists(select 1 from CTE as f2 where f1.rn &lt; f2.rn and f1.X1=f2.Y1 and f1.Y1=f2.X1)<br>order by X1</pre><pre>functions_sub = functions_df.withColumn(&#39;rn&#39;, F.row_number().over(Window.orderBy(F.col(&#39;X&#39;), F.col(&#39;Y&#39;))))<br><br>results_df = functions_sub.alias(&#39;f1&#39;)\<br>    .join(functions_sub.alias(&#39;f2&#39;),\<br>        ((F.col(&#39;f1.rn&#39;) &lt; F.col(&#39;f2.rn&#39;)) &amp; \<br>            (F.col(&#39;f1.X&#39;)==F.col(&#39;f2.Y&#39;)) &amp; \<br>                (F.col(&#39;f1.Y&#39;)==F.col(&#39;f2.X&#39;))),\<br>            how=&quot;inner&quot;)\<br>                .orderBy(F.col(&#39;f1.X&#39;))\<br>                    .select(F.col(&#39;f1.X&#39;), F.col(&#39;f1.Y&#39;))</pre><p>I have six question notebooks in my github as of November 8, 2024. I shall continue to write. I appreciate that you read the article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa7a9fce9081" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>