<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sushil Kumar on Medium]]></title>
        <description><![CDATA[Stories by Sushil Kumar on Medium]]></description>
        <link>https://medium.com/@iamsushil?source=rss-4f70ed3de659------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*apZdF35wwtgO0gsd9wDBUQ.jpeg</url>
            <title>Stories by Sushil Kumar on Medium</title>
            <link>https://medium.com/@iamsushil?source=rss-4f70ed3de659------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:33:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@iamsushil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automating .whl File Deployment to Azure Databricks with GitHub Actions]]></title>
            <link>https://python.plainenglish.io/automating-whl-file-deployment-to-azure-databricks-with-github-actions-b2172989f3de?source=rss-4f70ed3de659------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2172989f3de</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Sushil Kumar]]></dc:creator>
            <pubDate>Mon, 23 Dec 2024 08:01:54 GMT</pubDate>
            <atom:updated>2025-01-06T04:21:17.858Z</atom:updated>
            <content:encoded><![CDATA[<h3><em>Automating .whl File Deployment to Azure Databricks with GitHub Actions</em></h3><h3>Introduction<a href="https://shell.stackenterprise.co/articles/4148#heading-introduction">​</a></h3><p>In the world of data engineering and data science, the ability to streamline and automate workflows is invaluable. One common task is deploying Python packages, such as .whl files, to Azure Databricks. In this article, we’ll explore how to automate this process using GitHub Actions, a powerful and flexible workflow automation tool.</p><h3>What is a .whl File?<a href="https://shell.stackenterprise.co/articles/4148#heading-what-is-a.whl-file">​</a></h3><p>A .whl file, short for &quot;Wheel,&quot; is a built package format for Python that simplifies the process of distributing Python projects. Wheels are the standard binary package format used for packaging Python libraries.</p><h3>Prerequisites<a href="https://shell.stackenterprise.co/articles/4148#heading-prerequisites">​</a></h3><p>Before you begin, you should have the following:</p><ol><li>Azure Databricks Workspace: You need access to an Azure Databricks workspace.</li><li>Valid mount point on Databricks</li><li>Azure Databricks Cluster: You should have a running cluster in your workspace.</li><li>.whl File: You should have the .whl file that you want to install. You can create this file using pip by packaging your Python library, or you can obtain it from a trusted source.</li><li>Github repository.</li><li>Secrets setup in your Github repository.</li></ol><h3>Steps to Create and Push .whl File to Azure Databricks using Github actions<a href="https://shell.stackenterprise.co/articles/4148#heading-steps-to-create-and-push.whl-file-to-azure-databricks-using-github-actions">​</a></h3><h3>Step 1: Build and Package Your Python Code<a href="https://shell.stackenterprise.co/articles/4148#heading-step-1-build-and-package-your-python-code">​</a></h3><p>Currently I have uploaded all my required python code along with requirements.txt file onto my GitHub repository. In this section i am building and packaging python code. Later as a part of additional steps I am adding SonarQube and Python test steps as well which will give additional benefits for my python library and i can avoid any vulnerabilities.</p><ol><li>Lets start with the Github actions workflow, as I want to get package versions so adding a step to add package_version along with Environment</li></ol><pre>jobs:<br>  version_number:<br>    runs-on: ubuntu-latest<br>    outputs:<br>      output1: ${{steps.step1.outputs.test}}<br>    steps:<br>    - id: step1<br>      run: |<br>       date_part=$(date +%y%m.%-d%H)<br>       PKG_VERSION=&quot;1.&quot;$date_part<br>       echo $PKG_VERSION    <br>       echo &quot;PKG_VERSION=${PKG_VERSION}&quot; &gt;&gt; $GITHUB_ENV<br>       echo &quot;::set-output name=test::${PKG_VERSION}&quot;</pre><p>2. Next step we have a requirement to get Python package based on different versions so for that I am using matrix method and passing different versions of Python. Currently I am running only with Python version 3.8</p><pre>test_on_ubuntu:<br>    runs-on: ubuntu-latest # This is a special 8 core runner<br>    needs: version_number<br>    strategy:<br>      matrix:<br>        python-version: [ &#39;3.8&#39; ] <br><br><br>###if you want other version of pythons then please replace last line<br>### python-version: [ &#39;3.8&#39;, &#39;3.9&#39;, &#39;3.10&#39;, &#39;3.11&#39; ]</pre><p>3. Now lets go further and start adding additional steps, next I am installing Python dependencies as they may require for getting coverage, running python tests etc.</p><pre>- name: Install dependencies<br>        run: |<br>          pip install -r requirements.txt<br>          pip install pylint<br>          pip install pytest<br>          pip install coverage<br>          pip install --upgrade databricks-cli</pre><p>4. Next I am using some of the secrets which needs to be changed during runtime so running replace command.</p><pre>- name: Insert the credentials<br>        run: |<br>          sed -i &quot;s/env_PITSSecret/${{ secrets.env_PITSSecret }}/g&quot; tests/testconfig.ini<br>          sed -i &quot;s/env_Azure/${{ secrets.env_Azure }}/g&quot; tests/testconfig.ini<br>          sed -i &quot;s/env_dbtoken/${{ secrets.env_dbtoken }}/g&quot; tests/testconfig.ini</pre><p>5. In next step I am running SonarQube tests to find out any bugs/vulnerability/hotspots etc.</p><pre>- name: SonarQube Scan<br>        uses: sonarsource/sonarqube-scan-action@master<br>        with:<br>          args: &gt;<br>            -Dsonar.projectKey=com.projectname<br>            -Dsonar.python.coverage.reportPaths=./reports/sonar-report.xml<br>            -Dsonar.python.coverage.testExecutionReportPaths=./reports/sonar-report.xml<br>        env:<br>          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}<br>          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}</pre><p>6. Next step I am running command to build a wheel file and creating a release to store generated Build file onto my Github artifact location</p><pre>- name: Buildwheel<br>        run: |<br>          python -m pip install --user --upgrade build<br>          python -m build .<br>          echo &quot;VERSION = $(python setup.py --version)&quot; &gt;&gt; $GITHUB_ENV<br>      - name: Create Release<br>        uses: softprops/action-gh-release@v1<br>        with:<br>         files: ./dist/*.whl<br>         tag_name: v1.2.0-${{ matrix.python-version }}  # Replace with the name of the existing tag you want to use<br>        env:<br>          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}<br>          if: startsWith(github.ref, &#39;refs/tags/&#39;)  # Check if the workflow is running on a tag<br><br>### Later uploading wheel file incase if you want to debug on your local machine<br>      - name: Upload the wheels<br>              uses: actions/upload-artifact@v3<br>              with:<br>                name: built_wheel<br>                path: dist/*.whl</pre><p>7. Currently I have commented out code python tests however if anyone wants to use they can leverage this</p><pre># - name: Install dependencies<br>      #   run: |<br>      #     echo &quot;Python version is : ${{ matrix.python-version }}&quot;<br>      #     pip install dist/analytics-2.0.0.dev0-py3-none-any.whl[Full]<br>      #     python -m unittest discover<br>          <br>      # - name: List the tests<br>      #   run: pytest --collect-only tests <br>      <br>      # - name: Test with pytest<br>      #   run: |<br>      #     conda install pytest<br>      #     pytest<br>      # - name: Install tox<br>      #   run: pip install tox<br>        <br>      # - name: Run tests with tox<br>      #   run: tox</pre><h3>Step 2: Pushing the file to Azure Databricks dbfs mount.<a href="https://shell.stackenterprise.co/articles/4148#heading-step-2-pushing-the-file-to-azure-databricks-dbfs-mount">​</a></h3><p>Now that your Python package is packaged and released, it’s time to deploy it to Azure Databricks. This step involves setting up the Databricks CLI, configuring it with your Databricks host and access token, and copying the .whl file to your Databricks workspace.</p><pre>- name: Deploy .whl to Databricks DBFS<br>        run: |<br>          # Set the Databricks host and token from secrets<br>         <br>          DATABRICKS_TOKEN=${{ secrets.DATABRICKS_TOKEN }}<br>          DATABRICKS_HOST=${{ secrets.DATABRICKS_HOST }}<br> <br>          # Configure the Databricks CLI with the token<br>          echo &quot;[DEFAULT]&quot; &gt; ~/.databrickscfg<br>          echo &quot;host = $DATABRICKS_HOST&quot; &gt;&gt; ~/.databrickscfg<br>          echo &quot;token = $DATABRICKS_TOKEN&quot; &gt;&gt; ~/.databrickscfg<br>           <br>          # Copy wheel files<br>          for f in dist/*.whl; do<br>              databricks fs cp $f dbfs:/analytics-automation/ --overwrite #- you can replace dbfs path with your actual location<br>          don</pre><h3>Step 3: Verify file upload<a href="https://shell.stackenterprise.co/articles/4148#heading-step-3-verify-file-upload">​</a></h3><p>After installing the package, you should verify that it’s available for your Azure Databricks.</p><ol><li>Login to Azure databricks → Click on Catalog → Expand your metastore → Click on new schema → make sure you enable “Browse DBFS” option.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*yXrW3QSIzT1DVUbS.png" /></figure><p>Here is the complete Github actions file :</p><pre>name: Python package<br><br>on:<br>  push:<br>    branches:<br>      - &#39;feature/prashanth-ver1&#39;<br><br>jobs:<br>  version_number:<br>    runs-on: ubuntu-latest<br>    outputs:<br>      output1: ${{steps.step1.outputs.test}}<br>    steps:<br>    - id: step1<br>      run: |<br>       date_part=$(date +%y%m.%-d%H)<br>       PKG_VERSION=&quot;1.&quot;$date_part<br>       echo $PKG_VERSION    <br>       echo &quot;PKG_VERSION=${PKG_VERSION}&quot; &gt;&gt; $GITHUB_ENV<br>       echo &quot;::set-output name=test::${PKG_VERSION}&quot;<br>  test_on_ubuntu:<br>    runs-on: ubuntu-latest # This is a special 8 core runner<br>    needs: version_number<br>    strategy:<br>      matrix:<br>        python-version: [ &#39;3.8&#39; ]<br><br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v2<br><br>      - name: Set up Python<br>        uses: actions/setup-python@v2<br>        with:<br>          python-version: &#39;3.8&#39;<br><br>      - name: Install dependencies<br>        run: |<br>          pip install -r requirements.txt<br>          pip install pylint<br>          pip install pytest<br>          pip install coverage<br>          pip install --upgrade databricks-cli<br><br>      - name: Insert the credentials<br>        run: |<br>          sed -i &quot;s/env_PITSSecret/${{ secrets.env_PITSSecret }}/g&quot; tests/testconfig.ini<br>          sed -i &quot;s/env_Azure/${{ secrets.env_Azure }}/g&quot; tests/testconfig.ini<br>          sed -i &quot;s/env_dbtoken/${{ secrets.env_dbtoken }}/g&quot; tests/testconfig.ini<br><br>      - name: Run pylint<br>        run: pylint ${{ github.workspace }}/*.py<br><br>      - name: SonarQube Scan<br>        uses: sonarsource/sonarqube-scan-action@master<br>        with:<br>          args: &gt;<br>            -Dsonar.projectKey=com.analytics<br>            -Dsonar.python.coverage.reportPaths=./reports/sonar-report.xml<br>            -Dsonar.python.coverage.testExecutionReportPaths=./reports/sonar-report.xml<br>        env:<br>          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}<br>          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}<br><br>      - name: Buildwheel<br>        run: |<br>          python -m pip install --user --upgrade build<br>          python -m build .<br>          echo &quot;VERSION = $(python setup.py --version)&quot; &gt;&gt; $GITHUB_ENV<br><br>      - name: Create Release<br>        uses: softprops/action-gh-release@v1<br>        with:<br>         files: ./dist/*.whl<br>         tag_name: v1.2.0-${{ matrix.python-version }}  # Replace with the name of the existing tag you want to use<br>        env:<br>          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}<br>          if: startsWith(github.ref, &#39;refs/tags/&#39;)  # Check if the workflow is running on a tag<br><br>      - name: Upload the wheels<br>        uses: actions/upload-artifact@v3<br>        with:<br>          name: built_wheel<br>          path: dist/*.whl<br>  <br>      # - name: Install dependencies<br>      #   run: |<br>      #     echo &quot;Python version is : ${{ matrix.python-version }}&quot;<br>      #     pip install dist/qgcanalytics-2.0.0.dev0-py3-none-any.whl[Full]<br>      #     python -m unittest discover<br>          <br>      # - name: List the tests<br>      #   run: pytest --collect-only tests <br>      <br>      # - name: Test with pytest<br>      #   run: |<br>      #     conda install pytest<br>      #     pytest<br>      # - name: Install tox<br>      #   run: pip install tox<br>        <br>      # - name: Run tests with tox<br>      #   run: tox<br><br><br><br>      - name: Deploy .whl to Databricks DBFS<br>        run: |<br>          # Set the Databricks host and token from secrets<br>          <br>          DATABRICKS_TOKEN=${{ secrets.DATABRICKS_TOKEN }}<br>          DATABRICKS_HOST=${{ secrets.DATABRICKS_HOST }}<br> <br>          # Configure the Databricks CLI with the token<br>          echo &quot;[DEFAULT]&quot; &gt; ~/.databrickscfg<br>          echo &quot;host = $DATABRICKS_HOST&quot; &gt;&gt; ~/.databrickscfg<br>          echo &quot;token = $DATABRICKS_TOKEN&quot; &gt;&gt; ~/.databrickscfg<br>           <br>          # Copy wheel files<br>          for f in dist/*.whl; do<br>              databricks fs cp $f dbfs:/xxx-analytics-automation/ --overwrite<br>          done<br><br></pre><h3>Conclusion<a href="https://shell.stackenterprise.co/articles/4148#heading-conclusion">​</a></h3><p>Boost your data engineering efficiency with automated Python package deployments to Azure Databricks via GitHub Actions. This approach eliminates manual steps, reduces errors, and ensures your team always has access to the most recent code, facilitating seamless collaboration and data pipeline development.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a></li><li><a href="https://cofeed.app/"><strong>Check out CoFeed, the smart way to stay up-to-date with the latest in tech</strong></a> <strong>🧪</strong></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2172989f3de" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/automating-whl-file-deployment-to-azure-databricks-with-github-actions-b2172989f3de">Automating .whl File Deployment to Azure Databricks with GitHub Actions</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Impact of Row and Column Level Security with Databricks Unity Catalog on Query Performance]]></title>
            <link>https://python.plainenglish.io/the-impact-of-row-and-column-level-security-with-databricks-unity-catalog-on-query-performance-25ef84981616?source=rss-4f70ed3de659------2</link>
            <guid isPermaLink="false">https://medium.com/p/25ef84981616</guid>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[row-level-security]]></category>
            <dc:creator><![CDATA[Sushil Kumar]]></dc:creator>
            <pubDate>Wed, 25 Sep 2024 23:36:54 GMT</pubDate>
            <atom:updated>2024-09-26T12:09:37.199Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KaD1J1uJl6i7jqXgMw87qA.png" /></figure><p>Databricks Unity Catalog offers a centralized governance platform that provides robust security features, including row and column-level security (RLS and CLS). While these features enhance data privacy and access control, they can also impact query performance.</p><p>This article explores the potential performance implications of RLS and CLS in Databricks Unity Catalog but first, let’s cover the basics.</p><p><strong>Understanding RLS and CLS in Unity Catalog</strong></p><ul><li><strong>Row-Level Security (RLS):</strong> RLS restricts access to rows based on user attributes or conditions. It’s often implemented using user-defined functions (UDFs).</li><li><strong>Column-Level Security (CLS):</strong> CLS masks or redacts specific columns or parts of columns to protect sensitive data. It’s often implemented using user-defined functions (UDFs).</li></ul><p>More Details : <a href="https://www.databricks.com/blog/announcing-general-availability-row-and-column-level-security-databricks-unity-catalog">https://www.databricks.com/blog/announcing-general-availability-row-and-column-level-security-databricks-unity-catalog</a></p><blockquote><em>To evaluate the performance of this new column-level masking feature, I’ve implemented a straightforward User-Defined Function (UDF). This UDF is applied to a specific table and its corresponding columns to assess the impact on query execution times and overall system performance</em></blockquote><pre>select Count(*) from immuta_p_tst.rls_test.RLS_Performance_Test ;<br>-- 438276851<br><br>-- Let&#39;s create a UDF for Column level Masking<br>-- This Function will MASK the Email_ID column<br>CREATE  FUNCTION IF NOT EXISTS immuta_p_tst.rls_test.masked_email (email_id string) returns string RETURN (<br>  CASE<br>    WHEN upper(&#39;sushilkumar@gmail.com&#39;) = upper(current_user()) THEN email_id<br>    ELSE (CAST(NULL AS string))<br>  END<br>)<br><br>-- Table Alteration for Masking<br>Alter table immuta_p_tst.rls_test.RLS_Performance_Test alter column  email_id set mask immuta_p_tst.rls_test.masked_email;</pre><h4>To establish a baseline, I’ll first run a straightforward SELECT statement without applying the column masking function.</h4><blockquote><strong>Run 1: The first run completed in 10.28 mins</strong></blockquote><blockquote><strong>Run 2: The Second leveraged cached data and completed in a mere 1.32 seconds.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VH2muyukyeXvH7HY6DXuPA.png" /><figcaption>Without Column Level Masking Run1 &amp; Run 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nONAtqRQe_-3pOVNaOtWQw.png" /></figure><h3>Alright, let’s see what happens when we apply the Column Masking function to the table.</h3><blockquote><strong>Run1 : First run took around 11.97 mins</strong></blockquote><blockquote><strong>Run 2 : Seconds run took even more around 13.35 mins and “SURPRISE“ this time its not leveraged the CACHE data</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*quSO3PctzaITwzm-Ya1t-A.png" /><figcaption>With CLS Run1 &amp; Run 2</figcaption></figure><blockquote>The provided results indicate that applying the column masking function to the RLS_Performance_Test table has a <strong>significant impact on query performance.</strong></blockquote><p><strong>Key Observations:</strong></p><ol><li><strong>Increased Execution Time:</strong> Both runs with the column masking function applied took significantly longer than the previous runs without the masking function. This suggests that the masking process introduces additional computational overhead.</li><li><strong>Lack of Caching Benefits:</strong> Surprisingly, the second run with the masking function did not leverage cached data, even though it’s likely the same query as the first run. This could be due to several factors:</li></ol><ul><li><strong>Masking Function Complexity:</strong> The masking function itself might be computationally expensive, preventing the query optimizer from effectively caching results.</li><li><strong>Caching Configuration:</strong> The caching mechanism might be configured to avoid caching queries involving certain functions or operations, such as masking.</li><li>After implementing column masking, we observe a ‘<strong>Secure View’</strong> step in the query plan and this might be causing the execution plan to be recalculated due to potential changes in access permissions.</li></ul><blockquote>What is a Secure View ??</blockquote><blockquote>The “Secure View” step in the query plan indicates that the query is accessing data through a secure view. This means that the data being processed might be filtered, masked, or redacted based on specific security rules or policies.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wRZiymcwoWt4N5RMBMZHrQ.png" /><figcaption>Query Profile and Explain Plan</figcaption></figure><p><strong>Conclusion</strong></p><blockquote>While RLS and CLS are essential security measures, they can introduce performance trade-offs. Implement them thoughtfully and assess the impact on query execution. <strong><em>If you notice performance degradation, it’s likely due to the security features.</em></strong></blockquote><p><strong>PS : I’ve already reported this performance issue to the Databricks team. Hopefully, it will be addressed in a future update, resolving the problem.</strong></p><p><strong><em>Disclosure:</em></strong><em> Chat GPT helped me in the writing, preparation, or research of this article.</em></p><h3>Reference</h3><ul><li><a href="https://www.databricks.com/blog/announcing-general-availability-row-and-column-level-security-databricks-unity-catalog">Announcing the General Availability of Row and Column Level Security with Databricks Unity Catalog</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/tables/row-and-column-filters">Filter sensitive table data using row filters and column masks - Azure Databricks</a></li></ul><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25ef84981616" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/the-impact-of-row-and-column-level-security-with-databricks-unity-catalog-on-query-performance-25ef84981616">The Impact of Row and Column Level Security with Databricks Unity Catalog on Query Performance</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Big Debate : Databricks vs Snowflake]]></title>
            <link>https://medium.com/@iamsushil/the-big-debate-databricks-vs-snowflake-c0d44401c226?source=rss-4f70ed3de659------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0d44401c226</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Sushil Kumar]]></dc:creator>
            <pubDate>Sun, 31 Dec 2023 00:36:55 GMT</pubDate>
            <atom:updated>2023-12-31T00:36:55.862Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Big Debate : Databricks vs Snowflake</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/0*udz14tgXFKmmAmfP" /></figure><p>The debate between Snowflake and Databricks for the best modern cloud solution for analytics is a common topic of discussion. However, it’s important to recognize that both platforms are designed to address distinct needs, making a direct comparison apples to oranges.</p><p>To address this misconception, I’ll provide an unbiased breakdown of the key differences between the two platforms, highlighting their respective strengths and weaknesses. Before delving into the specifics of Snowflake and Databricks, it’s crucial to grasp the broader cloud analytics ecosystem.</p><p>Databricks: A Unified Analytics Platform</p><p>Databricks stands out as a unified analytics platform, encompassing a comprehensive range of features for data engineering, data science, and machine learning. Its foundation lies in Apache Spark, a distributed computing framework designed for large-scale data processing. Databricks seamlessly integrates Spark with Delta Lake, a data storage format that ensures data reliability and integrity.</p><p>Snowflake: A Cloud-Native Data Warehouse</p><p>Snowflake, on the other hand, excels as a cloud-native data warehouse, delivering a secure and scalable platform for data warehousing and business intelligence. Its architecture separates storage and compute, enabling independent scaling of each resource to meet fluctuating data volumes and processing demands. This architecture also eliminates the need for infrastructure management, simplifying operations and reducing costs.</p><h3><strong>Key Differentiators: A Comparative Analysis</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*r5eWJphMvvQrK4xR.png" /></figure><p>To effectively compare Databricks and Snowflake, let’s delve into their key differentiating factors:</p><p><strong>Data Processing:</strong></p><ul><li><strong>Databricks</strong>: Databricks excels in data processing, particularly for real-time data streaming and machine learning applications. Its Apache Spark foundation provides powerful data processing capabilities, enabling complex data transformations and analyses.</li><li><strong>Snowflake</strong>: Snowflake focuses on data warehousing, providing a robust platform for storing and analyzing structured data. Its SQL-based interface caters to business intelligence users, enabling them to easily query and visualize data.</li></ul><p><strong>Data Ingestion:</strong></p><ul><li><strong>Databricks</strong>: Databricks supports a wide range of data ingestion methods, including streaming data sources, cloud storage platforms, and on-premises databases. Its Delta Lake format facilitates efficient data ingestion and processing.</li><li><strong>Snowflake</strong>: Snowflake offers seamless data ingestion from various sources, including cloud storage, databases, and semi-structured data files. Its architecture ensures data integrity and reliability during ingestion.</li></ul><p><strong>Ease of Use:</strong></p><ul><li><strong>Databricks</strong>: Databricks’ interface is geared towards data engineers and data scientists, providing a collaborative environment for data exploration and experimentation. However, its complexity may pose challenges for less technical users.</li><li><strong>Snowflake</strong>: Snowflake’s user-friendly interface simplifies data warehousing tasks, making it accessible to business users and analysts. Its SQL-based approach is familiar to those with traditional data warehousing experience.</li></ul><p><strong>Pricing:</strong></p><ul><li>Databricks: Databricks employs a usage-based pricing model, where costs are determined by the amount of data stored, compute resources utilized, and features used.</li><li>Snowflake: Snowflake also utilizes a usage-based pricing structure, but its pricing model is more complex, considering factors such as storage, compute, data transfer, and queries.</li></ul><blockquote><strong>Choosing the Right Platform: A Decision Framework</strong></blockquote><p>The choice between Databricks and Snowflake ultimately depends on the specific needs and technical expertise of the organization. Here’s a decision framework to guide your choice:</p><ul><li><strong>Data Processing Requirements:</strong> If your organization prioritizes real-time data processing, machine learning, and data engineering, Databricks is a strong choice.</li><li><strong>Data Warehousing Needs:</strong> If data warehousing, business intelligence, and structured data analysis are your primary focus, Snowflake is well-suited.</li><li><strong>Technical Expertise:</strong> If your team comprises data engineers and data scientists with strong technical skills, Databricks’ flexibility can be advantageous.</li><li><strong>Ease of Use:</strong> For teams with less technical expertise or a preference for SQL-based interfaces, Snowflake’s user-friendliness is a valuable asset.</li></ul><p><strong>Conclusion</strong></p><p>Databricks and Snowflake represent two powerful and versatile cloud data warehousing platforms, each catering to specific use cases and technical requirements. By carefully evaluating your organization’s data needs, technical capabilities, and ease-of-use preferences, you can make an informed decision that aligns with your strategic data management objectives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0d44401c226" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Policy-Based Access Control (PBAC): What It Is and Why You Need It in Your Modern Data Lakehouse ?]]></title>
            <link>https://python.plainenglish.io/policy-based-access-control-pbac-what-it-is-and-why-you-need-it-in-your-modern-data-lakehouse-672e869c2082?source=rss-4f70ed3de659------2</link>
            <guid isPermaLink="false">https://medium.com/p/672e869c2082</guid>
            <category><![CDATA[policy]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-security]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[access-control]]></category>
            <dc:creator><![CDATA[Sushil Kumar]]></dc:creator>
            <pubDate>Tue, 17 Oct 2023 10:54:57 GMT</pubDate>
            <atom:updated>2023-10-20T09:14:43.193Z</atom:updated>
            <content:encoded><![CDATA[<h3>Policy-Based Access Control (PBAC): What It Is and Why You Need It in Your Modern Data Lakehouse?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*kQ8jGZqerI3SZPqs7tFgwQ.jpeg" /></figure><p>In a recent project, I worked on PBAC implementation in a Databricks environment. It was a new challenge, but I learned a lot and am excited to share my knowledge with others.</p><p>Before we go deep into PBAC, first let’s understand what are the traditional methods for access control in the data warehouse world.</p><ul><li><strong>Role-based access control (RBAC):</strong> RBAC is a method of access control that assigns users to roles, and then grants permissions to those roles. For example, a user in the “Sales” role might have permission to access the sales data platform, while a user in the “Marketing” role might have permission to access the marketing data platform.</li></ul><blockquote><strong>Roles represent a collection of permissions or privileges that define what actions a user can perform.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*-hkwMh_QIF4E6hQaYJfYjA.png" /><figcaption><strong>RBAC</strong></figcaption></figure><ul><li><strong>Attribute-based access control (ABAC):</strong> ABAC is a method of access control that grants access to resources based on the attributes of the user, the resource, and the environment. For example, an ABAC policy might grant users access to a resource if they are located in a certain country and have a specific job title.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*rt9T0HD0e_Ky1oBaPjo9kQ.png" /></figure><ul><li><strong>Discretionary access control (DAC):</strong> DAC is a method of access control that allows individual users to grant or deny access to their own resources. For example, a user might grant access to a file to a colleague who needs to work on the file.</li></ul><h3><strong>What is Policy-based access control (PBAC) ?</strong></h3><blockquote>Policy-Based Access Control takes the best of ABAC and RBAC, but makes it accessible to everyone. Like, ABAC, PBAC is capable of supporting both roles and attributes. So you can restrict access to who (role) what (resource or asset) and when (time of day).</blockquote><blockquote><strong>Policy-based access control (PBAC)</strong> is a data access governance strategy that grants access to data based on policies that define who can access what data and how they can use it. PBAC policies can be based on a variety of factors, such as the user’s role, department, location, and the data’s sensitivity.</blockquote><p>PBAC helps to ensure that data is only accessible to the users who need it and that users can only use the data in the ways that are authorized. This is important for protecting the security and privacy of data, as well as for ensuring compliance with regulations.</p><p>For example, you could define a policy that allows data scientists to access all non-sensitive data in the data lakehouse, but only allows data engineers to access sensitive data.</p><h3><strong>How does PBAC work?</strong></h3><p>PBAC systems use a <em>combination of four types of attributes</em> to determine if a user should be granted access to a resource and what permissions they are entitled to on that resource:</p><ul><li><strong>Subject</strong> attributes, such as the department or job title of a user</li><li><strong>Object</strong> attributes that describe the resource being accessed</li><li><strong>Action</strong> attributes, which describe the action someone is trying to attempt (such as reading or editing privileges)</li><li><strong>Contextual</strong> or <strong>environment</strong> attributes that assess the time, location, etc. of the attempt to gain access</li></ul><p>The combination of these attributes with policies is what makes PBAC unique. PBAC uses Boolean logic to evaluate whether an access request is legitimate and grants only the relevant access if all the conditions met properly by the user. Policies are programmed to process the attributes in an “if, then” manner.</p><p>Here are some specific examples of how PBAC can be used in a modern data lakehouse:</p><ul><li>Allow users in the “<em>Sales</em>” role to access all data in the “<em>Sales</em>” schema.</li><li>Allow users in the “<em>Marketing</em>” role to access all data in the “<em>Marketing</em>” schema, except for the “<em>customer_data</em>” table.</li><li>Allow users in the “<em>Marketing</em>” role to access all data in the “<em>Marketing</em>” Table, but Mask all the values of confidential(PII) columns like “<em>IBAN,CreditCard,Account_Number</em>”.</li><li>Allow users in the “<em>Management</em>” role to access all data in the data lakehouse, but only during <em>business hours</em>.</li><li>Allow users in the “<em>Data Science</em>” role to access all data in the data lakehouse, but only from <em>certain IP addresses</em>.</li><li>Allow users to access data in the data lakehouse only if they have completed a specific training course on data security.</li></ul><h3>Why do you need PBAC in your modern data lakehouse?</h3><ul><li><strong>To protect sensitive data.</strong> Modern data lakehouses are increasingly being used to store and process sensitive data from a variety of sources. This data could include personally identifiable information (PII), financial data, or intellectual property. PBAC allows you to define and enforce policies to protect this sensitive data from unauthorized access.</li><li><strong>To comply with regulations.</strong> Many industries are subject to regulations that require them to implement certain security measures to protect sensitive data. PBAC can help you comply with these regulations by providing you with a fine-grained approach to access control.</li><li><strong>To enable data sharing and collaboration.</strong> Modern data lakehouses are often used by teams of people, including data scientists, data engineers, and business analysts. PBAC allows you to share data with these teams in a secure and controlled manner.</li><li><strong>Simplified administration:</strong> PBAC can simplify the administration of your data lakehouse by providing you with a central place to define and manage access policies. This can save you time and effort, and help to ensure that your access policies are consistent and up-to-date.</li></ul><h3>Conclusion</h3><p>PBAC is an essential tool for modern data lakehouses. It helps organizations to improve security, compliance, data governance, agility, and efficiency. While PBAC can be complex to implement and manage, it is worth the investment for organizations that need to protect their data and ensure that it is used responsibly.</p><p>Here are some additional tips for implementing PBAC in your modern data lakehouse:</p><ul><li>Start by defining your data access requirements. What data needs to be protected? Who needs access to the data? What are the purposes for accessing the data?</li><li>Develop a set of PBAC policies that meet your data access requirements. Be sure to involve relevant stakeholders in the policy development process.</li><li>Implement PBAC using a technology solution that meets your needs. There are a variety of PBAC solutions available, both commercial and open source like <a href="https://www.immuta.com/">Immuta</a> , <a href="https://privacera.com/">privacera</a></li><li>Test your PBAC policies thoroughly before deploying them in production.</li><li>Monitor your PBAC policies on an ongoing basis and make changes as needed.</li></ul><p>If you are using a modern data lakehouse, I strongly encourage you to consider implementing PBAC. PBAC can help you to protect your data, comply with regulations, and govern your data effectively.</p><p><strong><em>Disclosure:</em></strong><em> Chat GPT helped me in the writing, preparation, or research of this article.</em></p><h3>Reference</h3><p><a href="https://www.immuta.com/">Homepage</a></p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><strong><em>(X</em></strong>), <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=672e869c2082" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/policy-based-access-control-pbac-what-it-is-and-why-you-need-it-in-your-modern-data-lakehouse-672e869c2082">Policy-Based Access Control (PBAC): What It Is and Why You Need It in Your Modern Data Lakehouse ?</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Say Goodbye to Pandas : Introducing Polars]]></title>
            <link>https://python.plainenglish.io/say-goodbye-to-pandas-introducing-polars-fabbd4a28b6a?source=rss-4f70ed3de659------2</link>
            <guid isPermaLink="false">https://medium.com/p/fabbd4a28b6a</guid>
            <category><![CDATA[polar]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Sushil Kumar]]></dc:creator>
            <pubDate>Mon, 07 Aug 2023 22:40:57 GMT</pubDate>
            <atom:updated>2023-08-09T01:26:49.248Z</atom:updated>
            <content:encoded><![CDATA[<h3>Say Goodbye to Pandas : Introducing Polars</h3><h3>A Lightning-Fast DataFrame Library for Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*13cw_mEkU9-oGdzB.jpeg" /><figcaption>Polars</figcaption></figure><p>Polars is a highly performant DataFrame library for manipulating structured data. The core is written in Rust, but the library is available in Python, Rust &amp; NodeJS. Its key features are:</p><ul><li><strong>Fast</strong>: Polars is written from the ground up, designed close to the machine and without external dependencies.</li><li><strong>I/O</strong>: First class support for all common data storage layers: local, cloud storage &amp; databases.</li><li><strong>Easy to use</strong>: Write your queries the way they were intended. Polars, internally, will determine the most efficient way to execute using its query optimizer.</li><li><strong>Out of Core</strong>: Polars supports out of core data transformation with its streaming API. Allowing you to process your results without requiring all your data to be in memory at the same time</li><li><strong>Parallel</strong>: Polars fully utilises the power of your machine by dividing the workload among the available CPU cores without any additional configuration.</li><li><strong>Vectorized Query Engine</strong>: Polars uses <a href="https://arrow.apache.org/">Apache Arrow</a>, a columnar data format, to process your queries in a vectorized manner. It uses <a href="https://en.wikipedia.org/wiki/Single_instruction,_multiple_data">SIMD</a> to optimize CPU usage.</li></ul><blockquote><strong>Why do we need Polars ??</strong></blockquote><blockquote>Pandas is an essential Python library for Data Science. But its biggest downside is that it can be <strong>slow for operations on large datasets</strong>. Polars is a Pandas alternative designed to <strong>process data faster</strong>.</blockquote><h4>Comparing Polars and Pandas:</h4><ol><li><strong>Performance:</strong> Polars is faster than pandas for many operations, especially when dealing with large datasets. Polars achieves this performance improvement by utilizing lazy evaluation, chunking, and parallelism. This means that Polars can handle larger datasets and perform operations faster than pandas in many cases.</li><li><strong>Memory usage:</strong> Polars uses less memory than pandas, especially when dealing with large datasets. This is because Polars uses lazy evaluation and chunking to reduce the memory footprint of operations.</li><li><strong>Ease of use:</strong> Polars has a similar API to pandas, so it is easy to use for those familiar with pandas. However, some operations may require a different approach in Polars, so there may be a learning curve for some users.</li><li><strong>Functionality: </strong>Polars has a similar set of functionality as pandas, with some differences due to the design choices made by the developers. However, Polars provides excellent support for time-series data, which can be challenging to handle in pandas.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*FH12ztXa1NAlTvHW.png" /><figcaption>The Polars logo</figcaption></figure><h4><strong>Installing Polars</strong></h4><p>First install Polars in your virtual environment.</p><pre>pip install polars</pre><h4>Importing</h4><p>To use the library import it into your project</p><pre>import polars as pl</pre><h4>Reading Data in Polars:</h4><pre>#To read data from a CSV file:<br><br>df = pl.read_csv(&quot;your_file.csv&quot;)<br><br>#To read data from a Parquet file:<br><br>df = pl.read_parquet(&quot;your_file.parquet&quot;)<br><br>#To read data from an Arrow record batch:<br><br>df = pl.DataFrame.from_arrow_record_batch(arrow_record_batch)<br><br>#create a Polars DataFrame from a Pandas DataFrame:<br><br>import pandas as pd<br>pandas_df = pd.DataFrame(...)<br>df = pl.DataFrame(pandas_df)</pre><h4>Display Data:</h4><pre># Display the first few rows of the DataFrame<br>print(df.head())<br><br># Display summary statistics of the DataFrame<br>print(df.describe())<br><br># Display the column names<br>print(df.columns)<br><br># Display the n number of random rows from the df<br>print(df.sample(2))</pre><h4>Data Structures</h4><p>The core base data structures provided by Polars are Series and DataFrames.</p><h4>→ Series</h4><p>Series are a 1-dimensional data structure. Within a series all elements have the same <a href="https://pola-rs.github.io/polars-book/user-guide/concepts/data-types/">Data Type</a></p><pre>import polars as pl<br><br>s = pl.Series(&quot;a&quot;, [1, 2, 3, 4, 5])<br>print(s)<br><br>Output : <br><br>shape: (5,)<br>Series: &#39;a&#39; [i64]<br>[<br>    1<br>    2<br>    3<br>    4<br>    5<br>]</pre><h4>→ DataFrame</h4><p>A DataFrame is a 2-dimensional data structure that is backed by a Series, and it can be seen as an abstraction of a collection (e.g. list) of Series. Operations that can be executed on a DataFrame are very similar to what is done in a SQL like query. You can GROUP BY, JOIN, PIVOT, but also define custom functions.</p><pre>from datetime import datetime<br><br>df = pl.DataFrame(<br>    {<br>        &quot;integer&quot;: [1, 2, 3, 4, 5],<br>        &quot;date&quot;: [<br>            datetime(2022, 1, 1),<br>            datetime(2022, 1, 2),<br>            datetime(2022, 1, 3),<br>            datetime(2022, 1, 4),<br>            datetime(2022, 1, 5),<br>        ],<br>        &quot;float&quot;: [4.0, 5.0, 6.0, 7.0, 8.0],<br>    }<br>)<br><br>print(df)<br><br>Output : <br><br>shape: (5, 3)<br>┌─────────┬─────────────────────┬───────┐<br>│ integer ┆ date                ┆ float │<br>│ ---     ┆ ---                 ┆ ---   │<br>│ i64     ┆ datetime[μs]        ┆ f64   │<br>╞═════════╪═════════════════════╪═══════╡<br>│ 1       ┆ 2022-01-01 00:00:00 ┆ 4.0   │<br>│ 2       ┆ 2022-01-02 00:00:00 ┆ 5.0   │<br>│ 3       ┆ 2022-01-03 00:00:00 ┆ 6.0   │<br>│ 4       ┆ 2022-01-04 00:00:00 ┆ 7.0   │<br>│ 5       ┆ 2022-01-05 00:00:00 ┆ 8.0   │<br>└─────────┴─────────────────────┴───────┘</pre><h4>Lazy / Eager API</h4><p>Polars supports two modes of operation: <strong><em>lazy</em></strong> and <strong><em>eager</em></strong>.</p><p>In the <strong>Eager API</strong> the query is executed immediately while in the <strong>Lazy API</strong> the query is only evaluated once it is &#39;needed&#39;. Deferring the execution to the last minute can have significant performance advantages that is why the Lazy API is preferred in most cases</p><h4>Contexts</h4><p>A <strong>Context</strong>, as implied by the name, refers to the context in which an expression needs to be evaluated. There are three main contexts <a href="https://pola-rs.github.io/polars-book/user-guide/concepts/contexts/#fn:1"><strong>1</strong></a>:</p><ol><li>Selection: df.select([..]), df.with_columns([..])</li><li>Filtering: df.filter()</li><li>Groupby / Aggregation: df.groupby(..).agg([..])</li></ol><h4>Expressions</h4><p>Polars has a powerful concept called expressions that is central to its very fast performance.</p><p>Expressions are at the core of many data science operations:</p><ul><li>taking a sample of rows from a column</li><li>multiplying values in a column</li><li>extracting a column of years from dates</li><li>convert a column of strings to lowercase</li></ul><p>However, expressions are also used within other operations:</p><ul><li>taking the mean of a group in a groupby operation</li><li>calculating the size of groups in a groupby operation</li><li>taking the sum horizontally across columns</li></ul><p>Polars performs these core data transformations very quickly by:</p><ul><li>automatic query optimization on each expression</li><li>automatic parallelization of expressions on many columns</li></ul><h4>Column-based API</h4><p>A Polars DataFrame is closer to a collection of columns. This means that you <strong>can combine columns in </strong><strong>Polars</strong> in ways that are not possible in Spark, because Spark preserves the relationship of the data in each row.</p><p>Consider this sample dataset:</p><pre>import polars as pl<br><br>df = pl.DataFrame({<br>    &quot;foo&quot;: [&quot;a&quot;, &quot;b&quot;, &quot;c&quot;, &quot;d&quot;, &quot;d&quot;],<br>    &quot;bar&quot;: [1, 2, 3, 4, 5],<br>})<br><br>dfs = spark.createDataFrame(<br>    [<br>        (&quot;a&quot;, 1),<br>        (&quot;b&quot;, 2),<br>        (&quot;c&quot;, 3),<br>        (&quot;d&quot;, 4),<br>        (&quot;d&quot;, 5),<br>    ],<br>    schema=[&quot;foo&quot;, &quot;bar&quot;],<br>)</pre><p><strong>Example : Combining </strong><strong>head and </strong><strong>sum</strong></p><p>In Polars you can write something like this:</p><pre>df.select(<br>    pl.col(&quot;foo&quot;).sort().head(2),<br>    pl.col(&quot;bar&quot;).filter(pl.col(&quot;foo&quot;) == &quot;d&quot;).sum()<br>)</pre><p>Output:</p><pre>shape: (2, 2)<br>┌─────┬─────┐<br>│ foo ┆ bar │<br>│ --- ┆ --- │<br>│ str ┆ i64 │<br>╞═════╪═════╡<br>│ a   ┆ 9   │<br>├╌╌╌╌╌┼╌╌╌╌╌┤<br>│ b   ┆ 9   │<br>└─────┴─────┘</pre><p>The expressions on columns foo and bar are completely independent. Since the expression on bar returns a single value, that value is repeated for each value output by the expression on foo. But a and b have no relation to the data that produced the sum of 9.</p><p>To do something similar in Spark, you&#39;d need to compute the sum separately and provide it as a literal.</p><h4><strong>Conclusion :</strong></h4><p><strong>“Polars: A New Star on the Rise”</strong></p><p>Polars is getting a lot of attention as a new tool. It offers almost all the functions that we need to manipulate our dataframe. Next to that, it offers a lazy API that helps us optimising our queries before we execute them. It’s really good at handling big amounts of data quickly. Especially when dealing with big sets of information, Polars’ clever way of working can be really helpful. Overall someone could with very little trouble, and a day of work, replace Pandas with Polars, and the code would run faster, look better, and probably not drive so many people crazy like Pandas and it’s strange syntax. I hope the examples provided an easy way for you to get jumpstarted with this new and exciting library.</p><p>Let the battle of the bears begin!</p><h4><strong>Reference</strong></h4><ul><li><a href="https://github.com/pola-rs/polars">GitHub - pola-rs/polars: Extremely fast Query Engine for DataFrames, written in Rust</a></li><li><a href="https://pola-rs.github.io/polars-book/user-guide/">Introduction</a></li></ul><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fabbd4a28b6a" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/say-goodbye-to-pandas-introducing-polars-fabbd4a28b6a">Say Goodbye to Pandas : Introducing Polars</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Maximizing Spark Performance: Minimizing Shuffle Overhead]]></title>
            <link>https://python.plainenglish.io/maximizing-spark-performance-minimizing-shuffle-overhead-e21b7a4e5e71?source=rss-4f70ed3de659------2</link>
            <guid isPermaLink="false">https://medium.com/p/e21b7a4e5e71</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[shuffle]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Sushil Kumar]]></dc:creator>
            <pubDate>Sun, 23 Jul 2023 15:11:40 GMT</pubDate>
            <atom:updated>2023-08-09T01:27:25.896Z</atom:updated>
            <content:encoded><![CDATA[<p>“Shuffle the cards, not the problems.” — Anonymous</p><figure><img alt="Shuffle" src="https://cdn-images-1.medium.com/max/1024/0*FWoHBRg6Ltvj8jG6.jpg" /><figcaption>Shuffle</figcaption></figure><p>As per Wikipedia.</p><p><strong>Shuffling</strong> is a procedure used to <a href="https://en.wikipedia.org/wiki/Randomization">randomize</a> a deck of <a href="https://en.wikipedia.org/wiki/Playing_card">playing cards</a> to provide an element of chance in <a href="https://en.wikipedia.org/wiki/Card_game">card games</a></p><h4>But what is Shuffling in the Spark world ??</h4><p><strong>Apache Spark processes queries by distributing data over multiple nodes and calculating the values separately on every node.</strong> However, occasionally, <strong>the nodes need to exchange the data</strong>. After all, that’s the purpose of Spark — processing data that doesn’t fit on a single machine.</p><p><strong>Shuffling is the process of exchanging data between partitions</strong>. As a result, data rows can move between worker nodes when their source partition and the target partition reside on a different machine.</p><p>Spark doesn’t move data between nodes randomly. Shuffling is a time-consuming operation, so it happens only when there is no other option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X5w3oJofekiPCZbJ3RD1Yg.png" /><figcaption>Spark Shuffle</figcaption></figure><h4>Performance Impact</h4><p>The <strong>Shuffle</strong> is an expensive operation since it involves disk I/O, data serialization, and network I/O. To organize data for the shuffle, Spark generates sets of tasks — <em>map</em> tasks to organize the data, and a set of <em>reduce</em> tasks to aggregate it. This nomenclature comes from MapReduce and does not directly relate to Spark’s map and reduce operations.</p><p>Internally, results from individual map tasks are kept in memory until they can’t fit. Then, these are sorted based on the target partition and written to a single file. On the reduce side, tasks read the relevant sorted blocks.</p><p>Certain shuffle operations can consume significant amounts of heap memory since they employ in-memory data structures to organize records before or after transferring them. Shuffle also generates a large number of intermediate files on disk.</p><blockquote><strong><em>Most Important Part → How to Avoid Spark Shuffle ??</em></strong></blockquote><ul><li><strong>Use Appropriate Partitioning:</strong> Ensure that your data is appropriately partitioned from the beginning. If your data is already partitioned based on the operation you are performing, Spark can avoid shuffling altogether. Use functions like repartition() or coalesce() to control the partitioning of your data.</li></ul><pre># Sample data<br>data = [(1, &quot;A&quot;), (2, &quot;B&quot;), (3, &quot;C&quot;), (4, &quot;D&quot;), (5, &quot;E&quot;)]<br><br># Create a DataFrame<br>df = spark.createDataFrame(data, [&quot;id&quot;, &quot;name&quot;])<br><br># Bad - Shuffling involved due to default partitioning (200 partitions)<br>result_bad = df.groupBy(&quot;id&quot;).count()<br><br># Good - Avoids shuffling by explicitly repartitioning (2 partitions)<br>df_repartitioned = df.repartition(2, &quot;id&quot;)<br>result_good = df_repartitioned.groupBy(&quot;id&quot;).count()</pre><ul><li><strong>Filter Early:</strong> Apply filters or conditions to your data as early as possible in your transformations. This way, you can reduce the amount of data that needs to be shuffled through the subsequent stages.</li></ul><pre># Sample data<br>sales_data = [(101, &quot;Product A&quot;, 100), (102, &quot;Product B&quot;, 150), (103, &quot;Product C&quot;, 200)]<br>categories_data = [(101, &quot;Category X&quot;), (102, &quot;Category Y&quot;), (103, &quot;Category Z&quot;)]<br><br># Create DataFrames<br>sales_df = spark.createDataFrame(sales_data, [&quot;product_id&quot;, &quot;product_name&quot;, &quot;price&quot;])<br>categories_df = spark.createDataFrame(categories_data, [&quot;product_id&quot;, &quot;category&quot;])<br><br># Bad - Shuffling involved due to regular join<br>result_bad = sales_df.join(categories_df, on=&quot;product_id&quot;)<br><br># Good - Avoids shuffling using broadcast variable<br># Filter the small DataFrame early and broadcast it for efficient join<br>filtered_categories_df = categories_df.filter(&quot;category = &#39;Category X&#39;&quot;)<br>result_good = sales_df.join(broadcast(filtered_categories_df), on=&quot;product_id&quot;)</pre><ul><li><strong>Use Broadcast Variables:</strong> If you have small lookup data that you want to join with a larger dataset, consider using broadcast variables. Broadcasting the small dataset to all nodes can be more efficient than shuffling the larger dataset.</li></ul><pre># Sample data<br>products_data = [(101, &quot;Product A&quot;, 100), (102, &quot;Product B&quot;, 150), (103, &quot;Product C&quot;, 200)]<br>categories_data = [(101, &quot;Category X&quot;), (102, &quot;Category Y&quot;), (103, &quot;Category Z&quot;)]<br><br># Create DataFrames<br>products_df = spark.createDataFrame(products_data, [&quot;product_id&quot;, &quot;product_name&quot;, &quot;price&quot;])<br>categories_df = spark.createDataFrame(categories_data, [&quot;category_id&quot;, &quot;category_name&quot;])<br><br># Bad - Shuffling involved due to regular join<br>result_bad = products_df.join(categories_df, products_df.product_id == categories_df.category_id)<br><br># Good - Avoids shuffling using broadcast variable<br># Create a broadcast variable from the categories DataFrame<br>broadcast_categories = broadcast(categories_df)<br><br># Join the DataFrames using the broadcast variable<br>result_good = products_df.join(broadcast_categories, products_df.product_id == broadcast_categories.category_id)</pre><ul><li><strong>Avoid Using </strong><strong>groupByKey():</strong> Prefer reduceByKey() or aggregateByKey() instead of groupByKey() as the former performs partial aggregation locally before shuffling the data, leading to better performance.</li></ul><pre># Sample data<br>data = [(1, &quot;click&quot;), (2, &quot;like&quot;), (1, &quot;share&quot;), (3, &quot;click&quot;), (2, &quot;share&quot;)]<br><br># Create an RDD<br>rdd = sc.parallelize(data)<br><br># Bad - Shuffling involved due to groupByKey<br>result_bad = rdd.groupByKey().mapValues(len)<br><br># Good - Avoids shuffling by using reduceByKey<br>result_good = rdd.map(lambda x: (x[0], 1)).reduceByKey(lambda a, b: a + b)</pre><ul><li><strong>Use Data Locality:</strong> Whenever possible, try to process data that is already stored on the same node where the computation is happening. This reduces network communication and shuffling.</li></ul><pre># Sample data<br>data = [(1, 10), (2, 20), (1, 5), (3, 15), (2, 25)]<br><br># Create a DataFrame<br>df = spark.createDataFrame(data, [&quot;key&quot;, &quot;value&quot;])<br><br># Bad - Shuffling involved due to default data locality<br>result_bad = df.groupBy(&quot;key&quot;).max(&quot;value&quot;)<br><br># Good - Avoids shuffling by repartitioning and using data locality<br>df_repartitioned = df.repartition(&quot;key&quot;)  # Repartition to align data by key<br>result_good = df_repartitioned.groupBy(&quot;key&quot;).max(&quot;value&quot;)</pre><ul><li><strong>Use Memory and Disk Caching:</strong> Caching intermediate data that will be reused in multiple stages can help avoid recomputation and reduce the need for shuffling.</li></ul><pre># Sample data<br>data = [(1, 10), (2, 20), (1, 5), (3, 15), (2, 25)]<br><br># Create a DataFrame<br>df = spark.createDataFrame(data, [&quot;key&quot;, &quot;value&quot;])<br><br># Bad - Shuffling involved due to recomputation of the filter condition<br>result_bad = df.filter(&quot;value &gt; 10&quot;).groupBy(&quot;key&quot;).sum(&quot;value&quot;)<br><br># Good - Avoids shuffling by caching the filtered data<br>df_filtered = df.filter(&quot;value &gt; 10&quot;).cache()<br>result_good = df_filtered.groupBy(&quot;key&quot;).sum(&quot;value&quot;)</pre><ul><li><strong>Optimize Data Serialization:</strong> Choose efficient serialization formats like Avro or Kryo to reduce the data size during shuffling.</li></ul><pre># Create a Spark session with KryoSerializer<br>spark = SparkSession.builder \<br>    .appName(&quot;AvoidShuffleExample&quot;) \<br>    .config(&quot;spark.serializer&quot;, &quot;org.apache.spark.serializer.KryoSerializer&quot;) \<br>    .getOrCreate()</pre><ul><li><strong>Tune Spark Configurations:</strong> Adjust Spark configuration parameters like spark.shuffle.partitions, spark.reducer.maxSizeInFlight, and spark.shuffle.file.buffer to fine-tune the shuffling behavior.</li><li><strong>Monitor and Analyze:</strong> Use Spark’s monitoring tools like Spark UI and Spark History Server to analyze the performance of your jobs and identify areas where shuffling can be optimized.</li></ul><p>By following these best practices and optimizing your Spark jobs, you can significantly reduce the need for shuffling, leading to improved performance and resource utilization. However, in some cases, shuffling may still be unavoidable, especially for complex operations or when working with large datasets. In such cases, <strong><em>focus on optimizing shuffling rather than completely avoiding it</em></strong>.</p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e21b7a4e5e71" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/maximizing-spark-performance-minimizing-shuffle-overhead-e21b7a4e5e71">Maximizing Spark Performance: Minimizing Shuffle Overhead</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Deep Dive into Apache Spark Join Strategies]]></title>
            <link>https://python.plainenglish.io/a-deep-dive-into-the-inner-workings-of-apache-spark-join-strategies-5d47dd29a4cb?source=rss-4f70ed3de659------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d47dd29a4cb</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[spark-join]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Sushil Kumar]]></dc:creator>
            <pubDate>Thu, 20 Jul 2023 14:45:58 GMT</pubDate>
            <atom:updated>2024-09-23T14:46:37.390Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Join</em></strong> operations are frequently used in big data analytics to merge two data sets, represented as tables or DataFrames, based on a common matching key. Spark’s DataFrame and Dataset APIs, along with Spark SQL, provide a variety of join transformations such as inner joins, outer joins, left joins, right joins, and more. However, these operations often require a significant amount of data movement across Spark executors, similar to how it works in relational databases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sGcNY9Vu49tIa9y3.jpg" /></figure><p>Although many Data Engineers are familiar with ANSI SQL Join types , they may not have a clear understanding of how these joins function within the Spark engine. To gain a better understanding of Spark’s join strategies, it is important to first recognize the various factors that can impact join operations.</p><p><a href="https://differ.blog/differ-ai">Introducing DifferAI</a></p><p><strong>What Factors affect Join Operations?</strong></p><blockquote><strong>Data Size:</strong> The size of the data being joined can affect the performance of join operations.</blockquote><blockquote><strong>Data Distribution:</strong> If the data being joined is not distributed evenly across the cluster, it can result in data <strong>skew</strong>, which can cause performance issues.</blockquote><blockquote><strong>Join Type: </strong>Different join types, such as inner join, outer join, and left join, can have varying performance characteristics depending on the data being joined. Eg : <strong><em>Equi Join</em></strong><em> — “=” / </em><strong><em>Non Equi Join</em></strong><em> — “&lt;,&gt;,≥, ≤”</em></blockquote><blockquote><strong>Data Format:</strong> The format of the data being joined can also affect performance. For example, <strong>Parquet</strong> or <strong>ORC</strong> format can be more efficient than <strong>CSV</strong> or <strong>JSON</strong> format due to their columnar storage and compression capabilities.</blockquote><blockquote><strong>Join Hints :</strong> If end-user want more control over the join strategy selection then they supply Join Hints like /*+ <strong>BROADCAST</strong>(table name)*/*</blockquote><p>Based on these factors Spark has <strong><em>5 distinct join strategies</em></strong> by which it exchanges<em>,</em> moves, sorts, groups, and merges data across executors:</p><ul><li>Broadcast Hash Join (BHJ)</li><li>Shuffle Hash Join (SHJ)</li><li>Sort Merge Join (SMJ)</li><li>Broadcast Nested Loop Join (BNLP)</li><li>Cartesian Product Join (CPJ)</li></ul><h4><strong>1. Broadcast Hash Join (BHJ)</strong></h4><p>In this technique, one of the data sets is small enough to fit in memory and is broadcasted to all the worker nodes in the cluster.</p><p>10 mb is the default value of <strong>spark.sql.autoBroadcastJoinThreshold</strong></p><pre>#Join Hints for broadcast join in Pyspark<br>df.hint(&quot;broadcast&quot;)</pre><pre><br>-- Join Hints for broadcast join<br>SELECT /*+ BROADCAST(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;<br>SELECT /*+ BROADCASTJOIN (t1) */ * FROM t1 left JOIN t2 ON t1.key = t2.key;<br>SELECT /*+ MAPJOIN(t2) */ * FROM t1 right JOIN t2 ON t1.key = t2.key;</pre><p><strong>When to use a Broadcast Hash Join (BHJ) ?</strong></p><ul><li>When each key within the smaller and larger data sets is hashed to the same partition by Spark</li><li>When one data set is much smaller than the other (and within the default config of 10 MB, or more if you have sufficient memory)</li><li>When you only want to perform an equi-join, to combine two data sets based on matching unsorted keys</li><li>When you are not worried by excessive network bandwidth usage or OOM errors, because the smaller data set will be broadcast to all Spark executors</li></ul><h4><strong>2. Shuffle Hash Join (SHJ)</strong></h4><p>In Spark, Shuffle Hash Join is one of the default join strategies used when <strong>joining two large data sets</strong>.</p><p>Shuffle Hash Join involves two primary steps:</p><ol><li><strong>Shuffling</strong>: This step partitions the data from the Join tables based on the Join key and distributes them across partitions to ensure that records with the same Join keys are assigned to the corresponding partitions.</li><li><strong>Hash Join:</strong> After shuffling the data, a classic single node Hash Join algorithm is performed on the data in each partition.</li></ol><pre>#Join Hints for shuffle hash join in Pyspark<br>df.hint(&quot;shuffle_hash&quot;)</pre><pre>-- Join Hints for shuffle hash join<br>SELECT /*+ SHUFFLE_HASH(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;</pre><p><strong>When to use a Shuffle Hash Join (SHJ)?</strong></p><ul><li>The datasets being joined are too large to fit into the memory of a single node.</li><li>The datasets are partitioned across multiple nodes in a cluster.</li><li>The join keys are not skewed, meaning they are evenly distributed across the partitions.</li><li>The join operation requires a full outer join or a left outer join, which cannot be performed using a broadcast join.</li></ul><p>In general, Shuffle Hash Join is a good choice when dealing with <strong>large-scale data processing</strong> in distributed environments and can significantly improve performance compared to other join methods.</p><h4>3. Sort Merge Join (SMJ)</h4><p>Sort Merge Join is a method used in distributed computing to join large datasets that are spread across multiple nodes in a cluster.</p><p>In Spark, Sort Merge Join involves the following steps:</p><ol><li><strong>Shuffle</strong>: The data from both tables is partitioned based on the join key. The partitioning is done in such a way that records with the same join key are sent to the same partition.</li><li><strong>Sort</strong>: The data in each partition is sorted based on the join key.</li><li><strong>Merge</strong>: The sorted data is then merged across partitions to perform the join operation.</li></ol><p><strong><em>spark.sql.join.preferSortMergeJoin=false (Default value is true)</em></strong></p><pre>#Join Hints for sort merge join in Pyspark<br>df.hint(&quot;merge&quot;)</pre><pre>-- Join Hints for shuffle sort merge join<br>SELECT /*+ SHUFFLE_MERGE(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;<br>SELECT /*+ MERGEJOIN(t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;<br>SELECT /*+ MERGE(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;</pre><p><strong>When to use a Sort Merge Join (SMJ)?</strong></p><ul><li>The data size is too large to fit into the memory of a single node, but not so large that it requires a Shuffle Hash Join.</li><li>The join keys are not evenly distributed across the partitions.</li><li>The data in each partition is already sorted by the join key, or the cost of sorting the data is lower than the cost of shuffling the data.</li><li>The join operation requires a full outer join or a left outer join, which cannot be performed using a broadcast join.</li></ul><h4><strong>4. Broadcast Nested Loop Join (BNLP)</strong></h4><p>Broadcast Nested Loop Join is a method used in Spark to join two datasets where one of the datasets is small enough to fit into the memory of a single node. In this method, the smaller dataset is broadcast to all the nodes in the cluster, and then a nested loop join is performed with the larger dataset.</p><p>The steps involved in Broadcast Nested Loop Join are as follows:</p><ol><li>The smaller dataset is broadcast to all the nodes in the cluster.</li><li>The larger dataset is partitioned based on the join key.</li><li>For each partition of the larger dataset, a nested loop join is performed with the broadcasted smaller dataset.</li><li>The results of each partition are then combined to form the final output.</li></ol><pre>SELECT /*+ BROADCAST(t1), MERGE(t1, t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;</pre><p><strong>When to use a Broadcast Nested Loop Join (BNLP) ?</strong></p><ul><li>One of the datasets is small enough to fit into the memory of a single node.</li><li>The join key is selective, meaning it results in a small number of matching records.</li><li>The larger dataset is not skewed, meaning the data is evenly distributed across the partitions.</li><li>The join operation requires an inner join or an equi-join.</li></ul><h4><strong>5. Cartesian Product Join (CPJ) </strong>aka Shuffle-and-Replication Nested Loop Join</h4><p>Cartesian Product Join (CPJ) is a method used in Spark to join two datasets by generating <strong>all possible combinations of the records in both datasets</strong>. In CPJ, each record in one dataset is paired with every record in the other dataset, resulting in a Cartesian product of the two datasets.</p><pre>#Join Hints for shuffle-and-replicate nested loop join in Pyspark<br>df.hint(&quot;shuffle_replicate_nl&quot;)</pre><pre>-- Join Hints for shuffle-and-replicate nested loop join<br>SELECT /*+ SHUFFLE_REPLICATE_NL(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;</pre><p><strong>Note</strong> : Cartesian Product Join can be a <strong>very expensive operation</strong>, especially when dealing with large datasets, as it generates all possible combinations of the records. It is generally not recommended to use this method unless it is absolutely necessary. Instead, other join methods such as Hash Join, Sort Merge Join, and Broadcast Nested Loop Join should be considered first.</p><h3>Priority of join hints</h3><p>For the scenario that multiple different join hints are added for the same table, Spark follows the priority list below:</p><ol><li>BROADCAST</li><li>MERGE</li><li>SHUFFLE_HASH</li><li>SHUFFLE_REPLACE_NL</li></ol><h3>Conclusion</h3><p>While Apache Spark automatically chooses the best join algorithm, developers can override this decision using <strong>hints</strong>. However, providing hints without a thorough understanding of the data can result in Out Of Memory (OOM) errors. Conversely, if the developer is knowledgeable about the underlying data and chooses not to provide hints, they may miss an opportunity to optimize the join operation. Therefore, it is important for developers to carefully consider the use of hints when performing join operations in Spark.</p><p><strong><em>References -</em></strong></p><p><a href="https://learning.oreilly.com/library/view/learning-spark-2nd/9781492050032/">Learning Spark, 2nd Edition</a></p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d47dd29a4cb" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/a-deep-dive-into-the-inner-workings-of-apache-spark-join-strategies-5d47dd29a4cb">A Deep Dive into Apache Spark Join Strategies</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Performance: Exploring Delta Engine Optimizations in Databricks - (Part 1/3)]]></title>
            <link>https://python.plainenglish.io/unlocking-performance-exploring-delta-engine-optimizations-in-databricks-8e0ba7d6e1e1?source=rss-4f70ed3de659------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e0ba7d6e1e1</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Sushil Kumar]]></dc:creator>
            <pubDate>Sun, 04 Jun 2023 00:14:28 GMT</pubDate>
            <atom:updated>2023-08-09T01:29:09.286Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*AFfRekd2jgBRc5TMxCyTjg.jpeg" /></figure><p>Databricks, with its <strong>Delta</strong> Engine, offers a suite of powerful optimizations that can significantly enhance the performance of your data processing workflows. In this article, we will explore key optimizations provided by Databricks for the Delta Engine, enabling you to maximize the efficiency and speed of your data operations.</p><ol><li><strong>Data Skipping using Stats: </strong>One of the fundamental optimizations in the Delta Engine is data skipping. By storing metadata regarding the minimum and maximum values for each column in a file, the engine can skip reading irrelevant files during query execution. This metadata-based file pruning drastically reduces the amount of data processed, leading to substantial performance gains.</li></ol><blockquote><strong>PySpark:</strong></blockquote><pre>from pyspark.sql import SparkSession<br><br># Create a SparkSession<br>spark = SparkSession.builder.getOrCreate()<br><br># Read the Delta table<br>deltaTable = spark.read.format(&quot;delta&quot;).load(&quot;/path/to/delta_table&quot;)<br><br># Generate statistics for data skipping<br>deltaTable.generate(&quot;STATISTICS&quot;)<br><br># Query the Delta table with data skipping<br>result = deltaTable.filter(&quot;column_name&quot; = &quot;value&quot;).select(&quot;column_name&quot;).show()</pre><blockquote><strong>Spark SQL:</strong></blockquote><pre>ALTER TABLE delta_table GENERATE STATISTICS</pre><p><strong>2. Delta Cache:</strong> Databricks introduces the Delta Cache, a unique caching mechanism that operates at the file level. By caching data on local worker nodes, subsequent queries can leverage cache hits, resulting in significantly faster query execution times. It’s important to note that Delta Cache differs from the traditional Spark cache, offering a more optimized and efficient caching strategy.</p><blockquote><strong>PySpark:</strong></blockquote><pre># Enable Delta Cache<br>spark.conf.set(&quot;spark.databricks.io.cache.enabled&quot;, &quot;true&quot;)<br><br># Read the Delta table <br>deltaTable = spark.read.format(&quot;delta&quot;).load(&quot;/path/to/delta_table&quot;)<br><br># Cache the Delta table<br>deltaTable.cache()</pre><blockquote><strong>Spark SQL:</strong></blockquote><pre>--Cache Delta table for faster subsequent queries<br>CACHE TABLE delta_table;</pre><p><strong>3. Z Ordering:</strong> Z ordering is a powerful technique employed by the Delta Engine to colocate related data within the same files. By internally sorting data based on a designated Z ordering column and dividing it into files, this optimization enhances data skipping capabilities. Z ordering can be likened to the concept of a clustered index, improving query performance by minimizing the amount of data that needs to be read.</p><blockquote><strong>PySpark:</strong></blockquote><pre># Read the Delta table <br>deltaTable = spark.read.format(&quot;delta&quot;).load(&quot;/path/to/delta_table&quot;)<br><br># Perform Z ordering <br>zOrderedTable = deltaTable.orderBy(col(&quot;z_ordering_column&quot;))<br><br># Write the Z ordered table back to Delta format <br>zOrderedTable.write.format(&quot;delta&quot;).mode(&quot;overwrite&quot;).save(&quot;/path/to/z_ordered_table&quot;)</pre><blockquote><strong>Spark SQL:</strong></blockquote><pre><br>OPTIMIZE delta_table ZORDER BY (OrderColumn)</pre><p><strong>4. Partitioning/Bucketing:</strong> While not specific to Databricks, partitioning and bucketing techniques are highly effective in optimizing data processing. By organizing data based on specific partitioning or bucketing columns, you can achieve enhanced data skipping capabilities. These techniques enable faster query execution by directly targeting relevant partitions or buckets, minimizing the amount of data accessed.</p><blockquote><strong>PySpark:</strong></blockquote><pre>#partitionBy()<br>df.write.option(&quot;header&quot;,True) \<br>        .partitionBy(&quot;state&quot;) \<br>        .mode(&quot;overwrite&quot;) \<br>        .csv(&quot;/tmp/zipcodes-state&quot;)<br><br>#Use repartition() and partitionBy() together<br>dfRepart.repartition(2)<br>        .write.option(&quot;header&quot;,True) \<br>        .partitionBy(&quot;state&quot;) \<br>        .mode(&quot;overwrite&quot;) \<br>        .csv(&quot;c:/tmp/zipcodes-state-more&quot;)<br><br>#bucketBy()<br>df.write\<br>    .bucketBy(16, &quot;key&quot;)\<br>    .sortBy(&quot;value&quot;)\<br>    .saveAsTable(&quot;table_name&quot;)</pre><blockquote><strong>Spark SQL:</strong></blockquote><pre>CREATE TABLE events (<br>  date DATE,<br>  eventId STRING,<br>  eventType STRING,<br>  data STRING)<br>USING delta<br>PARTITIONED BY (date)</pre><p><strong>5. Data skipping with Z-order indexes for Delta Lake:</strong> Data Skipping Indexes provide a highly efficient way to skip irrelevant data blocks during query execution. These indexes are automatically created based on the statistics gathered during the data skipping optimization process. By skipping unnecessary data blocks, queries can achieve significant performance improvements.</p><blockquote><strong>PySpark:</strong></blockquote><pre># Query the data with the data skipping index  , region is Z-order column<br>resultDF = salesDF.filter(col(&quot;region&quot;) == &quot;East&quot;).groupBy(&quot;product&quot;).agg(sum(&quot;revenue&quot;).alias(&quot;total_revenue&quot;))<br></pre><blockquote><strong>Spark SQL:</strong></blockquote><pre>--Run a query that uses Data Skipping Indexes  <br>SELECT sum(revenue) as &#39;total_revenue&#39; FROM sales WHERE region &gt; &quot;East&quot; <br>group by product;</pre><p><strong>6. Bin Packing:</strong> Over time, as the number of transactions increases, small files can accumulate and cause performance overhead. Databricks addresses this issue with the OPTIMIZE command, allowing you to compact numerous small files into a smaller number of larger files. This bin packing optimization reduces the file open/close overhead and improves query performance.</p><blockquote><strong>PySpark:</strong></blockquote><pre>from delta.tables import *<br>deltaTable = DeltaTable.forPath(spark, &quot;/data/events&quot;)<br>deltaTable.optimize().executeCompaction()</pre><blockquote><strong>Spark SQL:</strong></blockquote><pre>OPTIMIZE delta.&#39;/data/events&#39;<br><br>OR <br><br>OPTIMIZE events WHERE date &gt;= &#39;2022-11-18&#39;</pre><p>Stay tuned for Part 2…………..</p><h3>References:</h3><ul><li><a href="https://learn.microsoft.com/en-us/azure/databricks/delta/tutorial">Tutorial: Delta Lake - Azure Databricks</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/">Azure Databricks documentation</a></li></ul><p><strong>Thank you for reading!</strong></p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e0ba7d6e1e1" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/unlocking-performance-exploring-delta-engine-optimizations-in-databricks-8e0ba7d6e1e1">Unlocking Performance: Exploring Delta Engine Optimizations in Databricks - (Part 1/3)</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Ignited: Unleashing the Performance Beast within Apache Spark]]></title>
            <link>https://medium.com/@iamsushil/spark-ignited-unleashing-the-performance-beast-within-apache-spark-4778f6f05cde?source=rss-4f70ed3de659------2</link>
            <guid isPermaLink="false">https://medium.com/p/4778f6f05cde</guid>
            <category><![CDATA[spark-performance]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark-optimization]]></category>
            <dc:creator><![CDATA[Sushil Kumar]]></dc:creator>
            <pubDate>Sat, 03 Jun 2023 23:29:55 GMT</pubDate>
            <atom:updated>2023-06-03T23:44:28.770Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Introduction: </strong>Apache Spark has revolutionized big data processing, empowering businesses to extract valuable insights from vast datasets. However, harnessing the full potential of Spark requires careful attention to performance optimization. In this article, I will share my personal experiences and insights on how to supercharge your Spark applications and unlock blazing-fast processing speeds. Let’s embark on a journey to maximize the performance of your Spark workloads.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hh5s2spqwhbcn3OMA-y1yA.jpeg" /><figcaption>#CatalystOptimizer #SparkSQL #spark</figcaption></figure><ol><li><strong>Understand Your Data:</strong> To optimize Spark performance, start by gaining a deep understanding of your data. Analyze its characteristics, including <strong><em>size, structure, and patterns</em>.</strong> This understanding allows you to make informed decisions about partitioning, caching, and applying appropriate transformations to minimize data movement and unnecessary computations.</li><li><strong>Proper Data Partitioning:</strong> Partitioning is a critical aspect of Spark performance. Ensure that your data is efficiently partitioned based on how it will be accessed and processed. Utilize techniques like <strong><em>bucketing</em></strong> and <strong><em>sorting</em></strong> to improve data locality and minimize shuffling overhead. Careful partitioning can significantly reduce network traffic and enhance parallelism, leading to substantial performance gains.</li><li><strong>Leveraging Data Persistence:</strong> Spark offers various data persistence options, such as caching and checkpointing. Identify the datasets that are reused frequently and strategically cache them in memory or on disk. <strong><em>Caching</em></strong> eliminates the need for repetitive computations, leading to substantial time savings. Additionally, periodic checkpointing can prevent unnecessary recomputation in case of failures, ensuring fault tolerance while improving performance.</li><li><strong>Optimal Resource Allocation: </strong>Efficient resource allocation plays a vital role in Spark performance optimization. Analyze the characteristics of your Spark cluster and adjust the executor memory, CPU cores, and parallelism settings accordingly. Balancing resource allocation prevents underutilization or oversubscription, enabling Spark to make optimal use of available resources and deliver faster processing speeds.</li><li><strong>Transformations and Actions:</strong> Choosing the right set of Spark transformations and actions is crucial for performance. Prefer narrow transformations, such as map and filter, over wide transformations like groupByKey, which require <strong><em>data shuffling</em></strong>. Utilize aggregateByKey, reduceByKey, or combineByKey for efficient data aggregation. Additionally, employ lazy evaluation and pipeline operations to minimize unnecessary data materialization and optimize execution plans.</li><li><strong>Monitoring and Tuning:</strong> Continuously monitor your Spark applications using Spark UI, logs, and performance monitoring tools. Identify potential bottlenecks, such as long-running stages, skewed data, or inefficient operations. Utilize tuning techniques like broadcast variables, data repartitioning, and Spark configuration optimizations to alleviate performance issues. Experiment with different settings and measure their impact to fine-tune your Spark jobs.</li><li><strong>Spark SQL Optimization:</strong> If you leverage Spark SQL for data processing, optimizing SQL queries is paramount. Utilize techniques like query optimization, indexing, and appropriate join strategies. Leverage DataFrame and Dataset APIs to take advantage of Catalyst’s query optimization capabilities. Understanding Spark SQL’s underlying <strong><em>execution plans</em></strong> helps identify optimization opportunities and enhance overall performance.</li></ol><blockquote><strong>Conclusion:</strong> By implementing these performance optimization techniques, I have witnessed remarkable improvements in the speed and efficiency of my Spark applications. However, remember that every Spark workload is unique, and performance optimization is an iterative process. Continuously explore new optimizations, stay updated with the latest Spark enhancements, and adapt your strategies as needed. With perseverance and a deep understanding of Spark’s internals, you can unlock the true power of Apache Spark and propel your data-driven initiatives to new heights. Happy optimizing!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4778f6f05cde" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>