<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hamza Khribi on Medium]]></title>
        <description><![CDATA[Stories by Hamza Khribi on Medium]]></description>
        <link>https://medium.com/@khribi.hamza?source=rss-b134365e93df------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UGhljNfZz9v_mygn5k1zUA.jpeg</url>
            <title>Stories by Hamza Khribi on Medium</title>
            <link>https://medium.com/@khribi.hamza?source=rss-b134365e93df------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:04:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@khribi.hamza/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bridging the Gap: How Databricks Solved the Unity Catalog Compute Compatibility Challenge]]></title>
            <link>https://medium.com/@khribi.hamza/bridging-the-gap-how-databricks-solved-the-unity-catalog-compute-compatibility-challenge-609248a05fe4?source=rss-b134365e93df------2</link>
            <guid isPermaLink="false">https://medium.com/p/609248a05fe4</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <dc:creator><![CDATA[Hamza Khribi]]></dc:creator>
            <pubDate>Fri, 13 Jun 2025 13:24:32 GMT</pubDate>
            <atom:updated>2025-06-13T13:24:32.742Z</atom:updated>
            <content:encoded><![CDATA[<p>Unity Catalog has become the cornerstone of data governance in the Databricks ecosystem, providing centralized access control, auditing, and lineage tracking across your data lakehouse. However, organizations have long faced a frustrating trade-off when it comes to compute resources, particularly in machine learning workflows.</p><p>The challenge? Traditional shared compute clusters used extensively in ML use cases — especially for cost optimization with expensive GPU resources — weren’t compatible with Unity Catalog’s governance capabilities. Data science teams found themselves choosing between cost efficiency and proper data governance, a compromise no organization should have to make.</p><h3>Understanding Databricks Compute Types</h3><p>To appreciate the significance of this solution, let’s first understand the three main compute access modes:</p><p><strong>Dedicated Clusters (formerly Single User)</strong> These clusters are fully Unity Catalog compatible and support any Databricks runtime without restrictions. However, they’re assigned to individual users, making them expensive for team collaboration, especially when GPU resources are involved.</p><p><strong>Standard Clusters (formerly Shared)</strong> While these clusters support multiple users, they can’t run Machine Learning runtimes due to Databricks’ User Isolation security requirements. This limitation makes them unsuitable for most data science workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s-6EwpKNQOMQcuWhw4kR-Q.png" /></figure><p><strong>No Isolation Shared Clusters</strong> These clusters work perfectly with ML runtimes and allow multiple users, making them ideal for cost optimization. The catch? They’re not Unity Catalog compatible, leaving teams without proper governance controls.</p><h3>The Game-Changing Solution</h3><p>Databricks has introduced an elegant workaround that finally bridges this gap: <strong>group-assigned dedicated clusters</strong>. This feature allows organizations to create what’s essentially a shared cluster experience while maintaining full Unity Catalog compatibility.</p><p>Here’s how it works:</p><ol><li><strong>Create a dedicated cluster</strong> instead of a traditional shared cluster</li><li><strong>Assign the cluster to a group</strong> rather than an individual user</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wheIZfi5nvmXE3a3SGKocw.png" /></figure><p>3.<strong> Configure Unity Catalog permissions</strong> for the assigned group</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DLAumQ1mPmK7T3LZN04OMA.png" /><figcaption>Group sandbox-users doesn’t have sufficient permissions to read from catalog ‘prob’</figcaption></figure><p>4.<strong> Grant appropriate catalog privileges</strong> to the group</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lhKWQlsP6CbMa7qISNgJIg.png" /><figcaption>After granting the right privilege we are able to read from the catalog using the dedicated group custer</figcaption></figure><p>This approach maintains the security and governance benefits of dedicated clusters while providing the collaborative, cost-effective experience teams need.</p><h3>Real-World Impact: Data Science Teams</h3><p>Consider a typical data science team working with GPU-intensive machine learning models. Previously, they faced two suboptimal choices:</p><ul><li>Use shared no isolation clusters for cost efficiency but sacrifice Unity Catalog’s governance capabilities</li><li>Provision individual GPU clusters for each data scientist to access Unity Catalog — an expensive proposition that quickly becomes cost-prohibitive</li></ul><p>With group-assigned dedicated clusters, the same team can:</p><ul><li>Share expensive GPU resources across team members</li><li>Maintain full Unity Catalog integration for proper data governance</li><li>Reduce infrastructure costs significantly compared to individual clusters</li><li>Ensure consistent access controls and audit trails</li></ul><h3>Implementation Benefits</h3><p>This solution delivers several key advantages:</p><p><strong>Cost Optimization</strong>: Teams can share expensive compute resources, particularly GPU clusters, without sacrificing governance capabilities.</p><p><strong>Simplified Management</strong>: IT administrators can manage permissions at the group level rather than individual cluster assignments.</p><p><strong>Enhanced Governance</strong>: Full Unity Catalog integration ensures proper access controls, auditing, and lineage tracking across all ML workflows.</p><p><strong>Team Collaboration</strong>: Data scientists can collaborate effectively on shared resources while maintaining security boundaries.</p><h3>Conclusion</h3><p>Databricks’ group-assigned dedicated clusters represent more than just a technical workaround — they’re a fundamental shift toward making Unity Catalog governance accessible without compromising operational efficiency. For organizations running machine learning workloads at scale, this feature eliminates a long-standing barrier to implementing comprehensive data governance.</p><p>The days of choosing between cost optimization and proper governance are over. With this solution, data science teams can have both, enabling them to focus on what matters most: extracting value from data while maintaining the highest standards of security and compliance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=609248a05fe4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Four Underrated Features of Databricks Unity Catalog]]></title>
            <link>https://medium.com/@khribi.hamza/four-underrated-features-of-databricks-unity-catalog-8e235a75b2aa?source=rss-b134365e93df------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e235a75b2aa</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[databricks-unity-catalog]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Hamza Khribi]]></dc:creator>
            <pubDate>Wed, 30 Oct 2024 17:35:06 GMT</pubDate>
            <atom:updated>2024-10-30T17:35:06.885Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s fast-paced, data-driven world, ensuring robust governance and unified management of data assets is crucial for organizations. This is where Databricks <strong>Unity Catalog</strong> steps in as an indispensable solution. Designed to create a unified data platform, Unity Catalog offers seamless governance across diverse environments, ensuring data integrity, security, and compliance, all while simplifying data operations.</p><p>But beyond its core promise of data governance, Unity Catalog boasts several underrated features that make it a standout solution for data management, especially for organizations looking to scale their data operations and AI models securely. Below are four features of Unity Catalog that you may not have realized could transform the way you approach your data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*TCjU8TWGRFs5sMYkJycHeg@2x.jpeg" /></figure><p><strong>1. Data Lineage: Track Your Data Journey</strong></p><p>Data lineage is a crucial feature of Unity Catalog that allows organizations to track the origin, movement, and transformation of data across their workflows. It answers fundamental questions like, “Where did this data come from?” , “How has it been modified? ” and “Who modified it?”.</p><p>Data lineage is vital because it fosters transparency, ensuring data quality and trust. It helps organizations understand dependencies and troubleshoot data quality issues faster.</p><p>This level of traceability is crucial for businesses that rely on accurate data to make informed decisions. « le cerise sur le gateau » is that Unity catalog offers lineage out of the box and you don’t have to be a data practitioner to be able to access this feature on the databricks UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wI1zllkLKAfmaolaN41pOQ.png" /><figcaption>An example of data lineage on Databricks UI</figcaption></figure><p><strong>2. Governance of Models and Functions: A Unique Approach</strong></p><p>Unlike many data governance solutions, Unity Catalog extends its governance capabilities to <strong>models and functions</strong>, treating them as primary data assets. This distinction is important for several reasons, especially in terms of data privacy and security.</p><p>For example, functions like <strong>data masking</strong> are essential for protecting sensitive information while ensuring that users can still access necessary data for analysis. Unity Catalog provides an environment where such functions can be managed.</p><p>In the AI landscape, Unity Catalog offers a secure platform for building and deploying models, including <strong>generative AI models</strong> on Databricks. Consider this scenario: if you are using a generative AI model like LLaMA, surely you have wondered what if this model receives a prompt that might compromise my table? Unity Catalog ensures by using a single line of code that the latter reads from a source table without being allowed to write or modify data in that table. This simplified governance gives developers the freedom to innovate without compromising data integrity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*cOrbNR5aaf0sK2C__pjP1A.png" /><figcaption>An example of a function registered in raw schema.</figcaption></figure><p><strong>3. Lakehouse Federation: Expanding Your Horizons</strong></p><p>Unity Catalog’s <strong>lakehouse federation</strong> is a game-changing feature, especially with its recent support for <strong>Hive and AWS Glue federation</strong>. This allows organizations to read metadata of assets that exist outside of Databricks, making data observability and discovery more comprehensive and powerful.</p><p>For developers, this means they can easily discover which tables they need from on-premise or external systems before establishing a connection using JDBC or ODBC. It simplifies cross-environment data access and governance by allowing Databricks users to manage and govern external data assets within the Unity Catalog ecosystem. This feature is indispensable for businesses dealing with hybrid or multi-cloud architectures, as it brings all data assets into a single governance framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BRAHF-0gOSBmh0yKtNHozQ.png" /><figcaption>Multiple options of external connections including HMS.</figcaption></figure><p><strong>4. State-of-the-Art Data Model Segregation and Governance</strong></p><p>Unity Catalog enforces <strong>best practices</strong> for data model standardization and segregation, particularly through <strong>external locations and storage credentials</strong>. If you come from a world where mounting external tables was common, you’ll appreciate how Unity Catalog simplifies the complexity of managing all of this for you.</p><p>By enforcing a structured approach to data storage, access control, and security, Unity Catalog ensures that organizations adhere to industry standards and minimize risks.</p><p>These are just a few of the underrated capabilities that Unity Catalog offers, and they could be the key to convincing your business leaders to upgrade your existing data workspaces. Whether it’s <strong>leveraging data lineage for transparency, securing AI models, or simplifying access to external data</strong>, these features provide practical solutions that can greatly enhance your current data strategy. Feel free to comment and let me know if you’re interested in a series of articles that guide you through implementing these features<strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e235a75b2aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>