<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Karthikeyan T on Medium]]></title>
        <description><![CDATA[Stories by Karthikeyan T on Medium]]></description>
        <link>https://medium.com/@kaartickprince?source=rss-5d6225a28d4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gR0MJZ5DzIeQAtKdipZbpw.jpeg</url>
            <title>Stories by Karthikeyan T on Medium</title>
            <link>https://medium.com/@kaartickprince?source=rss-5d6225a28d4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:12:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kaartickprince/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How dbx is helps in testing spark applications in the databrick environment ?]]></title>
            <link>https://medium.com/@kaartickprince/how-dbx-is-helps-in-testing-spark-applications-in-the-databrick-environment-4ecd9b2666d1?source=rss-5d6225a28d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ecd9b2666d1</guid>
            <category><![CDATA[dbx]]></category>
            <category><![CDATA[end-to-end-testing]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Karthikeyan T]]></dc:creator>
            <pubDate>Tue, 23 May 2023 03:41:58 GMT</pubDate>
            <atom:updated>2023-07-11T21:49:14.379Z</atom:updated>
            <content:encoded><![CDATA[<h3>How does dbx help in testing spark applications in the databricks environment ?</h3><p>As data engineers, one of the challenges we encounter is the difficulty of replicating the exact environment of Databricks locally. Despite configuring our local environment to match Databricks as closely as possible, there are often uncertainties until the code is actually deployed to the live environment.</p><p><strong>dbx</strong> provides a solution for testing Spark code directly on the actual cluster, allowing data engineers to validate their code in an environment that closely resembles the production cluster. This eliminates the uncertainties and ensures that the code will perform as expected when deployed to the actual cluster for testing purposes.</p><p><strong>dbx</strong>, developed by Databricks Labs, is a tool that enhances the functionality of the Databricks command-line interface (Databricks CLI). It is an open-source solution specifically created to support efficient development lifecycle and streamline continuous integration and continuous delivery/deployment (CI/CD) processes on the Databricks platform. With <strong>dbx</strong>, users can experience accelerated development, seamless integration, and automated deployment on the Databricks platform.</p><p><strong>dbx</strong> streamlines the launching and deployment of jobs across various environments, making the process more straightforward. Additionally, it assists in packaging your project and delivering it to your Databricks environment in a version-controlled manner. With a focus on command-line interface (CLI) usage, <strong>dbx</strong> is designed to be actively employed within CI/CD pipelines and as a fundamental component of local tools, facilitating efficient development and integration workflows.</p><p>With <strong>dbx</strong>, you can effortlessly execute your local code in batch mode on the target clusters of your choice. It offers a seamless way to run your code in bulk without any hassle, enabling you to efficiently process and analyze data on your desired clusters.</p><h4>Pre-requesites are:</h4><p>1 . A databricks cluser in remote (<a href="https://docs.databricks.com/clusters/configure.html">https://docs.databricks.com/clusters/configure.html</a>)</p><p>2. dbx installed in local (pip install dbx — upgrade)</p><p>3. python 3.8 or above</p><h4>Setup environmental variables :</h4><p>DATABRICKS_HOST is the URL where your cluster reside</p><p>DATABRICKS_TOKEN</p><p>To generate an access token in your Data-bricks workspace, follow these steps:</p><ol><li>Access your Data-bricks workspace and locate your username in the top bar.</li><li>Click on your username, and from the drop-down menu, select “User Settings.”</li><li>In the User Settings, navigate to the “Access tokens” tab.</li><li>Click on the “Generate new token” button.</li><li>Optionally, you can enter a comment to help you identify this token in the future. You can also adjust the token’s default lifetime if needed. Leaving the “Lifetime (days)” box empty will create a token with no expiration (although it is not recommended).</li><li>Click on “Generate” to create the token.</li><li>The generated token will be displayed. Copy the token and ensure you store it securely.</li><li>Finally, click “Done” to complete the process.</li></ol><h4><strong>Setting up dbx in your py-spark project :</strong></h4><p>Summary : dbx pushes the executable code to databick file system and execute the code from there, you could see some random named folders in the file system of data bricks once you run the dbx commands</p><p><strong><em>Step 1 :</em></strong></p><blockquote><em>dbx configure — profile DEFAULT — environment default</em></blockquote><p>To create the necessary configuration files for your dbx project, follow these steps from your terminal:</p><ol><li>Navigate to the root folder of your dbx project.</li><li>Run the dbx configure command with the specified option.</li><li>This command will create a hidden folder named .dbx within your project&#39;s root folder.</li><li>Inside the .dbx folder, you will find the lock.json and project.json files, which store the project-specific configuration details.</li></ol><p>By running the dbx configure command with the appropriate option, the necessary configuration files will be created, enabling proper project setup and management for your dbx project.</p><pre>{<br>  &quot;environments&quot;: {<br>    &quot;default&quot;: {<br>      &quot;profile&quot;: &quot;DEFAULT&quot;,<br>      &quot;storage_type&quot;: &quot;mlflow&quot;,<br>      &quot;properties&quot;: {<br>        # where the code is placed in the databricks file system<br>        &quot;workspace_directory&quot;: &quot;/Shared/dbx/projects/&lt;current-folder-name&gt;&quot;,<br>        # where the wheel file resides<br>        &quot;artifact_location&quot;: &quot;dbfs:/dbx/&lt;current-folder-name&gt;&quot;<br>      }<br>    }<br>}</pre><p><em>Step 2:</em></p><p>Create a folder named conf within your dbx project’s root folder</p><p><em>Step 3:</em></p><p>To add a deployment.yaml file to the conf directory, follow these steps:</p><ol><li>Locate the conf directory in your project.</li><li>Create a new file named deployment.yaml within the conf directory.</li><li>Open the deployment.yaml file in a text editor.</li><li>Paste the below contents ( update the values as required) into the deployment.yaml file, ensuring it adheres to the required format and structure.<br>Here sample main is the main file of the project! dbx copies the files in the directory to the files system on data bricks and deployed to the cluster for execution</li></ol><pre>build:<br>  no_build: true<br>environments:<br>  default:<br>    workflows:<br>      - name: &quot;dbx-demo-job&quot;<br>        spark_python_task:<br>          python_file: &quot;file://sample_main.py&quot;</pre><p><em>Step 4:</em></p><blockquote>dbx execute — cluster-id=&lt;existing-cluster-id&gt; dbx-demo-job — no-package</blockquote><p><em>Step: 5<br></em>To view the results of a run locally, you can check the output displayed in your terminal. The terminal will show any relevant output or log messages produced during the run.</p><p>If you want to view the run’s results on your cluster, you can follow these steps:</p><ol><li>Access your cluster in the Databricks workspace.</li><li>Open the “Driver logs” tab for the cluster.</li><li>Within the “Driver logs” tab, navigate to the “Standard output” pane.</li><li>The “Standard output” pane will display the logs and output generated by the run on your cluster. You can view any relevant information or results in this section.</li></ol><p>By checking the “Standard output” pane in the “Driver logs” tab of your cluster, you can examine the run’s results and any associated logs directly on your Databricks cluster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ecd9b2666d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rare scenarios while caching a dataframe in spark]]></title>
            <link>https://medium.com/@kaartickprince/rare-scenarios-while-caching-a-dataframe-in-spark-36b36f924342?source=rss-5d6225a28d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/36b36f924342</guid>
            <category><![CDATA[cache]]></category>
            <category><![CDATA[udf]]></category>
            <category><![CDATA[persist]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Karthikeyan T]]></dc:creator>
            <pubDate>Tue, 22 Nov 2022 09:43:34 GMT</pubDate>
            <atom:updated>2022-11-23T12:22:29.547Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is Caching/Persisting in spark :</h3><p>Caching or Persisting a dataframe is an optimization technique which could greatly help in the performance improvement of a spark job.</p><p>In Spark, every action runs all the transformation before it. Even if the same transformation is used in multiple actions, by default spark does not reuse the calculated transformation instead they are recalculated for every action. Caching is an optimization mechanism to store the intermediate computations so that they can be used in subsequent actions.</p><p>The advantages of persisting/caching a dataframe could greatly reduce the execution time and also uses less resources since computations are reused.</p><h3>Different storage levels for caching :</h3><p>We can use various storage levels for caching our dataframes based on the requirements</p><p>The cache() / persist(): The method cache/persist with no arguments persists in the dataset with the default storage level (MEMORY_AND_DISK).</p><p>We can use any storage level from org.apache.spark.storage.StorageLevel and pass it as an argument for cache/persist method.</p><p>The storage levels are :</p><p>DISK_ONLY, DISK_ONLY_2 and DISK_ONLY_3: The dataframe is stored in disk and the CPU computation time is greatly impacted on the number of I/O done. <em>DISK_ONLY stores only one copy, DISK_ONLY_2 replicates each partition to two cluster nodes, DISK_ONLY_3 replicates each partition to three cluster nodes</em>.</p><p>MEMORY_ONLY,MEMORY_ONLY_2: The dataframe is stored in memory only. If the dataframe is large and when there is not enough memory then not all partitions are persisted, instead some are stored and the others are re-computed when required. <em>MEMORY_ONLY stores only one copy of partitions, MEMORY_ONLY_2 replicates each partition to two cluster nodes.</em></p><p>MEMORY_ONLY_SER,MEMORY_ONLY_SER_2: This is the similar to MEMORY_ONLY being it serialises the DataFrame in memory if space is full. <em>MEMORY_ONLY_SER stores only one copy of partitions, MEMORY_ONLY_SER_2 replicates each partition to two cluster nodes</em></p><blockquote><em>MEMORY_AND_DISK, MEMORY_AND_DISK_2: The DataFrame will be stored in memory first, and when required storage is greater than available memory, then it stores some partitions to disk. The CPU computation time is greatly impacted by the number of I/O done on disk. </em>MEMORY_AND_DISK stores one copy of partitions, MEMORY_AND_DISK_2 replicates each partition to two cluster nodes</blockquote><p>MEMORY_AND_DISK_SER, MEMORY_AND_DISK_SER_2: This is similar to MEMORY_AND_DISK being it serialises the DataFrame in memory and on disk if space is full or not available. <em>MEMORY_AND_DISK_SER stores only one copy of partitions, MEMORY_AND_DISK_SER_2 replicates each partition to two cluster nodes</em></p><p>OFF_HEAP: Store serialised data-frames/rdds. This memory lies outside the JVM, but it is used by JVM in some cases.</p><h3>Rare scenarios while caching a spark dataframe</h3><h4>Scenario 1 :</h4><blockquote>With all those advantages of caching, there are situations where caching greatly helps in cases other than just the reuse of computations</blockquote><blockquote>Caching is a must in situations when API calls are made from a computation and that computation is also used in subsequent actions.</blockquote><p>If you are using an UDF to make API calls in a computation the first action will make the API calls, and if the same computation is recalculated in another action then the API calls are done again, which could be duplicate API requests. This will greatly impact the servers and also the computation speed of the spark job. So it’s a must to cache the dataframe calculated after the first computation so no duplicate calls will be made in the subsequent actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7v6Ek-AiGr321JfiNSUxfA.png" /></figure><h4>Scenario 2:</h4><blockquote>Caching doesn&#39;t work if the source is updated:</blockquote><blockquote>if the spark job has an action which changes/updates the source of the dataframes then spark invalidates the cache</blockquote><p>Consider a situation when spark reads from a source and does some computations before writing to the same source again. In this case, the underlying data is changed. So caches are invalidated. Also, any dataframe that comes subsequently will do the computations again from the updated source, and here the previously cached computations are not used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*e8Ue2ooLvTzW_9lLQAos9g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36b36f924342" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>