<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Palak Kharbanda on Medium]]></title>
        <description><![CDATA[Stories by Palak Kharbanda on Medium]]></description>
        <link>https://medium.com/@pkharbanda09?source=rss-a4529e4beae0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*go1qOXrDRScT7uGohaI7LQ.jpeg</url>
            <title>Stories by Palak Kharbanda on Medium</title>
            <link>https://medium.com/@pkharbanda09?source=rss-a4529e4beae0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:29:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pkharbanda09/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploring Soda Core for Data Quality in Databricks]]></title>
            <link>https://medium.com/@pkharbanda09/exploring-soda-core-for-data-quality-in-databricks-f856bab1910d?source=rss-a4529e4beae0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f856bab1910d</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[soda-cl]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[soda-core]]></category>
            <dc:creator><![CDATA[Palak Kharbanda]]></dc:creator>
            <pubDate>Sun, 02 Jun 2024 15:14:54 GMT</pubDate>
            <atom:updated>2024-06-03T08:24:44.791Z</atom:updated>
            <content:encoded><![CDATA[<p>Data quality is paramount in any data engineering workflows. As organisations increasingly depend on data-driven insights, the need for accurate, consistent, and reliable data becomes crucial.</p><p>In this blog post, we’ll explore how to use Soda Core in Databricks, and practical implementation steps.</p><h3>Soda Core</h3><p>Soda Core is a free, open-source Python library and command-line tool designed to ensure data quality. It uses the Soda Checks Language (SodaCL) to convert user-defined inputs into SQL queries that test for data quality. By preparing data quality checks, Soda Core scans datasets within a data source to ensure they meet predefined standards.</p><p>Soda Core allows data engineers to connect to data sources, define quality checks, and scan datasets to identify issues. It enables three primary tasks:</p><ol><li>Connecting to data sources</li><li>Defining checks to identify poor-quality data</li><li>Scanning datasets to run the defined checks</li></ol><h4>What is Soda CL?</h4><p><strong>Soda Checks Language (SodaCL)</strong> is a YAML-based language. It allows you to write data quality checks, which gets executed when scanning data within the data source.</p><h4>Setting Up Soda Core in Databricks</h4><p>Integrating Soda Core with Databricks can significantly enhance your data quality management.</p><p>To get started with Soda Core in Databricks, follow these steps:</p><p><strong>1. Install Soda Core</strong></p><p>First, you need to install the Soda Core library in your Databricks environment.</p><pre>pip install -i https://pypi.cloud.soda.io soda-core-spark-df==3.2.1</pre><pre>dbutils.library.restartPython()</pre><p><strong>2. Create Data &amp; Add Soda CL Checks</strong></p><p>A Soda Check is a test that library executes when it scans a dataset in your data source.<strong> </strong>It is a Python expression that checks metrics to see if they match the parameters that are defined for a measurement.</p><p>Create a dummy dataset that can be used for quality checks. Here, we have created datasets: customers_data, orders_data, orders_items_data</p><pre>customers_data = [<br>    (1, &quot;John Smith&quot;, &quot;john.smith@example.com&quot;),<br>    (2, &quot;Jane Brown&quot;, &quot;jane.brown@example.com&quot;),<br>    (3, &quot;Matt Brown&quot;, &quot;sam.brown@example.com&quot;),<br>    (3, &quot;James Smith&quot;, &quot;james.smith@example.com&quot;)<br>]<br><br>orders_data = [<br>    (101, 1, &quot;2024-01-15&quot;),<br>    (102, 2, &quot;2024-01-17&quot;),<br>    (104, 3, &quot;2024-01-20&quot;)<br>]<br><br>order_items_data = [<br>    (1001, 101, &quot;Product A&quot;, 2, 20.0),<br>    (1002, 102, &quot;Product B&quot;, 1, 50.0),<br>    (1003, 104, &quot;Product C&quot;, 3, 15.0)<br>]<br><br>customers_df = spark.createDataFrame(customers_data).toDF(&quot;customer_id&quot;, &quot;name&quot;, &quot;email&quot;)<br>orders_df = spark.createDataFrame(orders_data).toDF(&quot;order_id&quot;, &quot;customer_id&quot;, &quot;order_date&quot;)<br>order_items_df = spark.createDataFrame(order_items_data).toDF(&quot;order_item_id&quot;, &quot;order_id&quot;, &quot;product_name&quot;, &quot;quantity&quot;, &quot;price&quot;)</pre><pre>#Create a view that sodacl uses as dataset <br>customers_df.createOrReplaceTempView(&quot;customers_df&quot;)<br>orders_df.createOrReplaceTempView(&quot;orders_df&quot;)<br>order_items_df.createOrReplaceTempView(&quot;order_items_df&quot;)</pre><p>Then, define Soda CL checks for the data quality metrics. Below is an example :</p><ol><li>Basic validation checks: It contains row count, duplicate record checks, max length of a column, missing count checks and more.</li><li>Reference check: It validates that column contents match between datasets.</li><li>Schema check: It confirm the data types of the columns andverify the existence of columns in a dataset.</li></ol><pre># Define checks for datasets <br>checks = &quot;&quot;&quot;<br># Checks for basic validation<br>checks for customers_df:<br>    - row_count &gt; 0<br>    - missing_count(customer_id) = 0<br>    - duplicate_count(customer_id):<br>         fail: when &gt; 0<br>    - customer_id &gt;= 1:<br>        name: customer_id greater than equal to 1<br>        customer_id query: SELECT customer_id FROM customers_df<br>    - max_length(name) = 11    <br>    - missing_percent(customer_id) &lt; 5%:<br>        missing values: [&#39;0&#39;, &#39;null&#39;]<br># Check for referential integrity<br>checks for orders_df:<br>  - values in (customer_id) must exist in customers_df (customer_id)  <br># Checks for schema changes <br>checks for order_items_df:<br>  - schema:<br>      name: Confirm that the required columns are present<br>      fail:<br>        when required column missing: [order_item_id, order_id, product_name, quantity, price]<br>  - schema:<br>      name: Confirm that column type is correct<br>      fail:      <br>        when wrong column type:<br>          order_item_id : bigint<br>          order_id : bigint<br>          product_name : string<br>          quantity : bigint<br>          price : double</pre><p><strong>3. Load Data and Run Data Quality Checks</strong></p><p>A Soda scan executes the checks , once the checks are ready, you can run Soda CL checks within a Databricks notebook. Use the following code to execute the checks:</p><pre># import Scan from Soda Library<br>from soda.scan import Scan</pre><pre>scan = Scan()<br>scan.set_scan_definition_name(&quot;Databricks Notebook&quot;)<br>scan.set_data_source_name(&quot;customer_data&quot;)<br>scan.add_spark_session(spark, data_source_name=&quot;customer_data&quot;)</pre><pre>scan.add_sodacl_yaml_str(checks)</pre><pre>#execute the scan<br>scan.execute()</pre><pre># Inspect the scan logs<br>scan.get_logs_text()</pre><p>Output: In the customers_data, we have duplicate customer_id. Therefore, we can see from the output that check failed where expected.</p><pre>INFO   | Soda Core 3.2.1<br>INFO   | Using DefaultSampler<br>INFO   | Scan summary:<br>INFO   | 8/9 checks PASSED: <br>INFO   |     customers_df in customer_data<br>INFO   |       row_count &gt; 0 [PASSED]<br>INFO   |       customer_id greater than equal to 1 [PASSED]<br>INFO   |       missing_count(customer_id) = 0 [PASSED]<br>INFO   |       missing_percent(customer_id) &lt; 5% [PASSED]<br>INFO   |       max_length(name) = 11 [PASSED]<br>INFO   |     orders_df in customer_data<br>INFO   |       values in (customer_id) must exist in customers_df (customer_id) [PASSED]<br>INFO   |     order_items_df in customer_data<br>INFO   |       Confirm that the required columns are present [PASSED]<br>INFO   |       Confirm that column type is correct [PASSED]<br>INFO   | 1/9 checks FAILED: <br>INFO   |     customers_df in customer_data<br>INFO   |       duplicate_count(customer_id) fail when &gt; 0 [FAILED]<br>INFO   |         check_value: 1<br>INFO   | Oops! 1 failures. 0 warnings. 0 errors. 8 pass.</pre><h4>Conclusion</h4><p>Integrating Soda Core with Databricks offers a powerful solution for managing data quality in your data pipelines. By automating data validation, anomaly detection, and monitoring, you can ensure that your data is accurate, reliable, and ready for analysis.</p><p>To automate the execution of data quality checks, integrate Soda Core with Databricks workflows. This ensures that data quality checks are part of your regular data pipeline, providing continuous monitoring and alerting.</p><h4>Reference:</h4><p><a href="https://docs.soda.io/">Home</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f856bab1910d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dive Into Data Quality with Deequ]]></title>
            <link>https://medium.com/@pkharbanda09/dive-into-data-quality-with-deequ-bb1fd332d7c0?source=rss-a4529e4beae0------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb1fd332d7c0</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[deequ]]></category>
            <dc:creator><![CDATA[Palak Kharbanda]]></dc:creator>
            <pubDate>Sun, 06 Nov 2022 13:59:03 GMT</pubDate>
            <atom:updated>2022-11-06T14:07:18.117Z</atom:updated>
            <content:encoded><![CDATA[<h4>Have you ever imagined how flawed data can impact businesses?</h4><p>Nowadays, every organisation is overwhelmingly recognising the importance of data. Especially in Data Mesh, “<strong>Data as a Product</strong>” being one of its core principles means that the data must be thought of as a product by the team because they are in charge of its quality and representation.</p><p>Having inaccurate or flawed data can lead to failures in production systems, unintended outputs, and wrong inferences, leading to poor business decisions.</p><p>In this article, we will learn about data quality and how to use Deequ in order to test data quality.</p><h3><strong>Data Quality and its dimensions</strong></h3><blockquote><strong>Data Quality</strong> is defined as how well the dataset satisfies the user&#39;s needs. It is the foundation based on which we gather information.</blockquote><p><strong>Data Quality Dimensions</strong> are aspects used to determine the quality of data. These are most often used when defining <strong>Data quality rules</strong>*. These dimensions include the following aspects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*qmrBgXKVRO-xINbBuMZtyA.png" /><figcaption>Data Quality Dimensions</figcaption></figure><p><strong>Accuracy</strong> refers to the degree to which data accurately reflects an event or object described.</p><p><strong>Validity</strong> refers to data that conforms to a specific format or follows business rules.</p><p><strong>Timeliness</strong> refers to the degree to which data is available for use in the expected time frame.</p><p><strong>Completeness</strong> refers to the data that is sufficient to deliver meaningful inferences and decisions.</p><p><strong>Uniqueness</strong> refers to the data that is properly identified and recorded once in the dataset. It ensures that there are not any<strong> </strong>duplicate records.</p><p><strong>Consistency</strong> refers to the data represented consistently across the dataset.</p><p><strong>Note: </strong>Data Quality Rules<strong>* </strong>are business rules of data intended to assure the quality of data in terms of Accuracy, Validity, Completeness, Timeliness, Uniqueness, and Consistency.</p><h3><strong>Deequ</strong>:</h3><blockquote>Deequ is an open-source framework for testing data quality. It is built on top of Apache Spark and is designed to verify the quality of many large production datasets.</blockquote><p>It allows us to calculate data quality metrics on our data set and define data quality constraints. Furthermore, It also specifies what constraint checks can be made on data.</p><p>There are three significant components in Deequ.</p><ul><li>Constraint Suggestion</li><li>Constraint Verification</li><li>Metrics Computation</li></ul><p>Let’s implement these in Databricks notebooks using some sample data.</p><p><strong>Setup</strong></p><p>1. Install Deequ library to the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lIlX8CNW5rPfqRUN" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QiDMDYk9QD8UtyYS" /></figure><p>2. Create Dummy Data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PQ7QYCibgsiM71is" /></figure><p>Let’s analyse this data manually and try to link the data quality dimensions with the sample data.</p><p>Data violating the data quality dimensions are as follows</p><p>Accuracy: Incorrect spelling of country for ID 2,</p><p>Validity: Wrong email format for ID 2,</p><p>Completeness: DOB missing for ID 5 and Country missing for ID 4,</p><p>Uniqueness: ID 7 is a duplicate entry of ID 1</p><p>Now, let’s have a glance at the different components of Deequ</p><p><strong>1. Constraint Suggestion:</strong></p><p>It would be challenging to define constraints if we have numerous datasets, Deequ provides possible data quality constraint checks to be tested on the data. Deequ profiles the data and, based on that, suggests useful constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*afo-IgwK_Y8M73gB" /></figure><p>In the above snapshot, the output displays the suggested constraints and the Scala code for each constraint. As there were null records present for DOB column and Country column in the dataset, Deequ suggests completeness constraint for both.</p><p><strong>2. Constraint Verification:</strong></p><p>With Deequ, we can verify our data by defining a set of data quality constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ra2iQSFx3RwqaFx8" /></figure><p>Here, we are checking the following properties of the data</p><p>Validating if ID consists of any null records or a non-negative number.</p><ul><li>The number of rows is 7</li><li>Values of the id and name attribute are never NULL</li><li>Values of the id attribute are non-negative</li><li>Email Id values are unique</li><li>Values of the Country attribute can only contain values from the Array (“USA”, “Canada”, “UK”, “France”)</li></ul><p>Verification result include all the metrics computed during the run. checkResultsAsDataFrame is used to display the output as a data frame. It displays each constraint status and provides a message if any constraint fails.</p><p><strong>3. Metrics Computation:</strong></p><p>Deequ provides us with a way to compute the data quality metrics. Some of the metrics provided by Deequ are ApproxCountDistinct, Completeness, Maximum and Uniqueness. It computes metrics using some set of aggregation queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NzJAsoLxYrnDIkmF" /></figure><p>In the above snapshot, we have used the following metrics:</p><ol><li>Size: It returns the number of rows in a DataFrame i.e, 7.</li><li>Completeness: It returns a fraction of non-null values in a column. We have used it for the DOB and Name column.</li><li>CountDistinct: It returns the number of distinct values i.e, 6 for the Email column.</li><li>Uniqueness: It returns a fraction of unique values over the number of all values of a column. In sample data, the value for two records is the same for the Email column.</li></ol><h4>Conclusion:</h4><p>Data quality is critical and it cannot be ignored. Analysing the data manually can be a tedious task. Therefore, Deequ can be used to look over the quality of data. We can also monitor our data quality reports by using Databricks dashboards.</p><p><strong>Note:</strong></p><p>apache-spark version 3.2.1 is incompatible with Deequ 2.0. We need to use apache-spark 3.1.x</p><p>I have used the below configuration</p><p>Cluster Configuration: 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)</p><p>Deequ Library Used: com.amazon.deequ:deequ:2.0.0-spark-3.1</p><p><strong>Reference</strong>:</p><ul><li><a href="https://aws.amazon.com/blogs/big-data/test-data-quality-at-scale-with-deequ/">Test data quality at scale with Deequ | Amazon Web Services</a></li><li><a href="https://github.com/awslabs/deequ">GitHub - awslabs/deequ: Deequ is a library built on top of Apache Spark for defining &quot;unit tests for data&quot;, which measure data quality in large datasets.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb1fd332d7c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interesting things about PowerShell]]></title>
            <link>https://medium.com/@pkharbanda09/interesting-things-about-powershell-5e45f03b6295?source=rss-a4529e4beae0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e45f03b6295</guid>
            <category><![CDATA[cmdlet]]></category>
            <category><![CDATA[background-jobs]]></category>
            <category><![CDATA[powershell]]></category>
            <dc:creator><![CDATA[Palak Kharbanda]]></dc:creator>
            <pubDate>Tue, 21 Sep 2021 19:05:35 GMT</pubDate>
            <atom:updated>2021-10-07T16:11:56.092Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>PowerShell</strong> is a command line shell scripting language which combines the features of Command Prompt with a new scripting instruction set.</blockquote><p>It is robust solution that helps users automate a range of tedious or time-consuming tasks. Unlike the other shells which accept and return text, It can accept and return objects.</p><p><strong>CMDLETS</strong></p><p>Cmdlets are the driving force behind PowerShell’s functional capabilities. These are commands which implement various functions to perform specific tasks.</p><p><strong>1.Get-Help</strong></p><p>Get-Help command provides access to the information that you need to run and operate with all of the available commands.</p><p>Command : Get-Help Get-Process</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M7i31iuM3xQ09b08XfR73w.png" /></figure><p><strong>2.Get-Command</strong></p><p>Get-Command is a cmdlet that displays all of the commands that are accessible in your current session.</p><p>Command: Get-command *alias*</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gynd-ZXewwv_W_jh7Qjm9g.png" /></figure><p><strong>3.Get-Service</strong></p><p>Get-Service command provides the details about services that are installed on the system.</p><p>Command : Get-Service</p><p>Output :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*nQo0V4mlASzMX3CWRclNwQ.png" /></figure><p><strong>4.Get-Process</strong></p><p>Get-Process command lists down the running processes. It can quickly identify the problematic process. Stop-Process can be used to terminate it, after you have the name or process ID.</p><p>Command : Get-Process</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7q3aUHwLN1vSlOQUCghHw.png" /></figure><p><strong>5.Stop-Process</strong></p><p>Stop-Process command can be used to terminate processes that have been frozen or are no longer responding.</p><p>Command : Stop-Process -processname WinSCP</p><p>Output :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7gS_fAniiYbJadDllIn3Sg.png" /></figure><p><strong>6.Test-Path</strong></p><p>Test-Path can be used to verify whether items exist in a specified path. If the folder exists, it will return True otherwise it will return False.</p><p>Command: Test-Path C:\Users</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6pdMuNMPOco6VYIIhTyacA.png" /></figure><p><strong>7.Start-Transcript</strong></p><p>Start-Transcript and Stop-Transcript can be used to enable logging in the scripts<strong>. Start-Transcript</strong> creates a record of all or part of a PowerShell session to a text file.</p><p>Command : Start-Transcript</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*TGJ-hgJ0Z7cFkiVQAs1oXQ.png" /></figure><p><strong>8.Stop-Transcript</strong></p><p>The <strong>Stop-Transcript</strong> cmdlet stops the transcript that was started by Start-Transcript cmdlet.</p><p>Command : Stop-Transcript</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*Lf_B771CDDL3rCY1UNXBkg.png" /></figure><p><strong>9.Set- Alias</strong></p><p>Set-Alias command creates an alias to a cmdlet in the current PowerShell session.</p><p>Command : Set-Alias -Name list -Value Get-ChildItem</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWXgWZv0C4Zbyk2h3kVm6Q.png" /></figure><p><strong>10.Get-Alias</strong></p><p>Get-Alias cmdlet displays the cmdlet name which uses the <strong>Name</strong> parameter to display the list alias.</p><p>Command : Get-Alias -Name list</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*obKrn4PLBjiaillRqxVwQA.png" /></figure><blockquote><strong>Other Interesting Features</strong></blockquote><p><strong>Background Jobs</strong></p><p>If cmdlet takes time to execute, PowerShell provides the functionality to send a cmdlet to the background to complete. A series of commands can be sent to execute at once and let them complete on their own schedule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z0nSRVIc9UGzSKSgcczt3A.png" /></figure><p>Recieve-Job command returns the result of the background job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yhnsXzXxR7Np70ev_AnBDg.png" /></figure><p><strong>Insert timestamps into PowerShell outputs</strong></p><p>Timestamp can be entered in series to determine how long a single step occurs or to use as a logging mechanism for your scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oNUyA3NyLKt5w_c7qpQt_g.png" /></figure><blockquote><strong>Conclusion</strong></blockquote><p>PowerShell is more powerful than other shell types as it consists of a command-line shell along with scripting language built on the .NET framework. It is more extensible via cmdlets. Above cmdlets can be used in day-to-day activities. Therefore, Powershell is the way to go for sysadmins, developers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e45f03b6295" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>