<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Serge Smertin on Medium]]></title>
        <description><![CDATA[Stories by Serge Smertin on Medium]]></description>
        <link>https://medium.com/@nf-x?source=rss-ecd56c5c8f37------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*xiLZeJXPUEmUC7C-.jpg</url>
            <title>Stories by Serge Smertin on Medium</title>
            <link>https://medium.com/@nf-x?source=rss-ecd56c5c8f37------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:54:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nf-x/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[UCX v0.35.0]]></title>
            <link>https://medium.com/databricks-labs/ucx-v0-35-0-a42f627e9c16?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/a42f627e9c16</guid>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Thu, 05 Sep 2024 15:57:01 GMT</pubDate>
            <atom:updated>2024-09-05T15:57:01.799Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re thrilled to announce the release of UCX v0.35.0, which brings significant enhancements to Databricks Unity Catalog (UC) Automated Migration especially for AWS, focusing on UC role management, documentation, and linter improvements. This release also introduces support for SQL notebooks in the functional testing framework and updates the sqlglot dependency to include new features and bug fixes.</p><p>🔒 UC Role Management:<br>- Introduced the `databricks labs ucx delete-credential` command to delete roles created by UCX, supported only for AWS.<br>- Added the `delete_uc_role(role_name: str)` function in the `access.py` file and the `delete_uc_roles` method to the `Credentials` class in the `credentials.py` file to delete UC roles.<br>- Included new methods to the `IamRoleCreation` class and updated the `delete_role` method in the `AWSResources` class for AWS resource management and external location creation in Databricks.</p><p>📚 Documentation and Linters:<br>- Added basic documentation for linter message codes and a new command to display message codes for UC migration assistance.<br>- Implemented linters for detecting Direct File System Access (DFSA) in Python and SQL code to promote the use of Unity Catalog tables or volumes instead of direct file system access.</p><p>🛠️ Bug Fixes and Improvements:<br>- Fixed various `mypy`, `pylint`, and `pytest` build warnings to enhance codebase maintainability and developer experience.<br>- Improved the log warning about skipped grants to provide more specific information about skipped Hive metastore grant actions.<br>- Added new methods and classes to improve logging and error handling for Hive metastore grants during ACL migration.</p><p>📝 Testing:<br>- Introduced support for SQL notebooks in the functional testing framework, expanding the testing capabilities beyond Python notebooks.<br>- Updated the SQLglot dependency to include new features and bug fixes.<br>- Removed the `inventorize_mounts()` method from the `Mounts` class, replacing it with the `.snapshot()` API.</p><p>With these new improvements and fixes, ucx v0.35.0 aims to provide a more stable, efficient, and user-friendly library for software engineers working with AWS, Unity Catalog, and Databricks. Happy coding! 💻🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a42f627e9c16" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-labs/ucx-v0-35-0-a42f627e9c16">UCX v0.35.0</a> was originally published in <a href="https://medium.com/databricks-labs">Databricks Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Unleashing UCX v0.20.0: Exciting New Features and Improvements for Seamless Upgrades]]></title>
            <link>https://medium.com/databricks-labs/unleashing-ucx-v0-20-0-exciting-new-features-and-improvements-for-seamless-upgrades-4f5a0c7e4182?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f5a0c7e4182</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Thu, 28 Mar 2024 14:20:22 GMT</pubDate>
            <atom:updated>2024-03-28T14:20:22.592Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re thrilled to announce the release of UCX v0.20.0, your go-to companion for upgrading to Unity Catalog. This version brings significant improvements, new features, and enhancements that will make your upgrading experience smoother than ever. Here’s a summary of the most exciting updates you’ll find in this release:</p><p><strong>💡 New features and functionalities:</strong></p><ul><li><em>A new </em><em>is_partitioned column</em>: A new boolean column, is_partitioned, has been added to the Table class in the hive_metastore module and the tables table in the inventory database. This indicates if a table is partitioned or not.</li><li><em>Deploying static views for the interactive dashboard</em>: Two static views, misc_patterns_vw and code_patterns, have been added to the interactive dashboard, with the latter renamed from code_patterns_vw.</li><li><em>Improved Table ACL migration logic</em>: We’ve fixed and improved Table ACL migration logic for enhanced functionality.</li></ul><p><strong>🔧 Workflow improvements:</strong></p><ul><li><em>ACL migration strategy configuration</em>: We’ve introduced an ACL migration strategy configuration for the migrate-tables workflow, allowing users to pass a list of ACL migration strategies instead of a single strategy. This provides enhanced flexibility in handling various ACL migration types.</li><li><em>Added support for AVRO in SYNC</em>: AVRO is now a supported table format in the SYNC process, making it more versatile and compatible with various data formats.</li><li><em>Assessment of interactive cluster usage</em>: A new assessment of interactive cluster usage compared to UC compute limitations has been added, allowing users to avoid potential performance bottlenecks.</li><li><em>External location validation</em>: We’ve added external location validation when creating catalogs with the create-catalogs-schemas command, ensuring data integrity.</li><li><em>Flag for identifying JAR submitted jobs</em>: A flag has been added to the Job table to identify jobs submitted as JAR file, improving assessment functionality for Java/Scala jobs.</li></ul><p><strong>🐛 Bug fixes and stability improvements:</strong></p><ul><li><em>Skipping installation if remote and local versions are the same</em>: A prompt has been added to override the installation if remote and local versions are the same, avoiding unnecessary installations.</li></ul><p><strong>📚 Updates to documentation and testing:</strong></p><ul><li><em>Testing fixtures guidelines in CONTRIBUTING.md</em>: We’ve added guidelines for writing testing fixtures and debugging integration tests. This helps to ensure a solid and reliable testing environment.</li><li><em>Increasing code coverage</em>: We’ve added unit tests and increased code coverage by 1%, making the codebase more robust and fault-tolerant.</li></ul><p>To learn more about UCX v0.20.0, check out the <a href="https://github.com/databrickslabs/ucx/releases/tag/v0.20.0">official release notes</a> and start benefiting from these new improvements and features. Happy upgrading! 🎉</p><p><em>Please share and subscribe to </em><a href="https://medium.com/databricks-labs/newsletters/updates-from-databricks-labs"><em>Updates from Databricks Labs</em></a><em> newsletter to stay up-to-date with the latest releases from </em><a href="https://github.com/databrickslabs"><em>GitHub</em></a><em> namespace. Subscribing ensures you’re the first to know about the latest enhancements, bug fixes, and exciting features that will take your Data Intelligence Platform experience to the next level. You‘re encouraged to </em><a href="https://github.com/databrickslabs"><em>follow</em></a><em> the GitHub org as well ;)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f5a0c7e4182" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-labs/unleashing-ucx-v0-20-0-exciting-new-features-and-improvements-for-seamless-upgrades-4f5a0c7e4182">🚀 Unleashing UCX v0.20.0: Exciting New Features and Improvements for Seamless Upgrades</a> was originally published in <a href="https://medium.com/databricks-labs">Databricks Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Unleashing UCX v0.18.0: A Powerful Release for Seamless Migrations and Improved Compatibility]]></title>
            <link>https://medium.com/databricks-labs/unleashing-ucx-v0-18-0-a-powerful-release-for-seamless-migrations-and-improved-compatibility-ebacd8da5d77?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebacd8da5d77</guid>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Thu, 21 Mar 2024 20:08:36 GMT</pubDate>
            <atom:updated>2024-03-21T20:08:36.235Z</atom:updated>
            <content:encoded><![CDATA[<p>Greetings, Databricks enthusiasts! 🤝 We’re thrilled to announce the release of UCX v0.18.0, bringing you an array of new features, enhancements, and bug fixes targeted at streamlining your workflows, increasing reliability, and expanding compatibility. Let’s dive into the highlights of this much-anticipated release.</p><h3>🔒 Legacy Table ACL Grants Migration</h3><p>The migrate-tables workflow now supports a legacy table ACL grants migration feature, addressing issue <a href="https://github.com/databrickslabs/ucx/issues/340">#340</a> and paving the way for future improvements. This enhancement introduces a new method, get_tables_to_migrate, and an optional acl_strategy parameter in the migrate_tables method. The GrantsCrawler class and a new AclMigrationWhat enum class have been implemented as well. The hive_metastore module also received updates, enhancing the migration process for external tables, DBFS root delta tables, and table-level ACLs. (<a href="https://github.com/databrickslabs/ucx/issues/1054">#1054</a>)</p><h3>⚙️ databricks labs ucx cluster-remap</h3><p>The new databricks labs ucx cluster-remap command simplifies remapping legacy cluster configurations to UC-compatible ones, enabling a smooth transition of tables to UC. A revert functionality, databricks labs ucx revert-cluster-remap, has also been introduced. (<a href="https://github.com/databrickslabs/ucx/issues/994">#994</a>)</p><h3>💻 databricks labs ucx migrate-local-code</h3><p>The release debuts the databricks labs ucx migrate-local-code command for migrating local code to a Databricks environment, supporting Python and SQL files. This experimental feature aims to assist in managing code migration, maintaining consistency across workspaces, and improving compatibility with Unity Catalog. (<a href="https://github.com/databrickslabs/ucx/issues/1067">#1067</a>)</p><h3>🗃️ Instance Pool in Cluster Policy</h3><p>UCX v0.18.0 extends flexibility in configuring workflow clusters by adding the ability to specify an instance pool ID for cluster policies. The update includes a new method, _get_instance_pool_id(), and integration and unit tests to verify table migration jobs using instance pools. (<a href="https://github.com/databrickslabs/ucx/issues/1078">#1078</a>)</p><h3>🔄 Fixed SYNC Command Failures</h3><p>Improvements to handling SYNC command failures in the _migrate_external_table method ensure better reliability and error handling for external table migrations in the Hive Metastore. (<a href="https://github.com/databrickslabs/ucx/issues/1073">#1073</a>)</p><h3>🚀 Additional Improvements</h3><ul><li>The ucx move command now supports moving Unity Catalog tables from one schema to another after the table upgrade process. (<a href="https://github.com/databrickslabs/ucx/issues/1062">#1062</a>)</li><li>Fixed integration test failures with new pytest fixtures, policy crawler integration tests, and migration of external tables in Hive Metastore. (<a href="https://github.com/databrickslabs/ucx/issues/1068">#1068</a>)</li><li>Refactoring the installer improves modularity and reusability in the codebase. (<a href="https://github.com/databrickslabs/ucx/issues/1055">#1055</a>)</li></ul><h3>📝 Additional Notes</h3><ul><li>Skipped the test_table_migration_job integration test (<a href="https://github.com/databrickslabs/ucx/issues/1075">#1075</a>)</li><li>Skipped unsupported locations while migrating to external location in Azure (<a href="https://github.com/databrickslabs/ucx/issues/1066">#1066</a>)</li><li>Triggering Assessment Workflow from Installer based on User Prompt (<a href="https://github.com/databrickslabs/ucx/issues/1007">#1007</a>)</li><li>Updated <a href="http://readme.md/">README.md</a> for Service Principal Installation Limit (<a href="https://github.com/databrickslabs/ucx/issues/1076">#1076</a>)</li><li>Updated missing documentation for cluster remap command (<a href="https://github.com/databrickslabs/ucx/issues/1079">#1079</a>)</li></ul><p>For the complete details on this release, please refer to <a href="https://github.com/databrickslabs/ucx/releases/tag/v0.18.0">Release v0.18.0</a>.</p><p>🚀 Let’s embrace UCX v0.18.0 and drive your Databricks migrations to new heights! 🔥</p><p><em>Please share and subscribe to </em><a href="https://medium.com/databricks-labs/newsletters/updates-from-databricks-labs"><em>Updates from Databricks Labs</em></a><em> newsletter to stay up-to-date with the latest releases from </em><a href="https://github.com/databrickslabs"><em>GitHub</em></a><em> namespace. Subscribing ensures you’re the first to know about the latest enhancements, bug fixes, and exciting features that will take your Data Intelligence Platform experience to the next level. You‘re encouraged to </em><a href="https://github.com/databrickslabs"><em>follow</em></a><em> the GitHub org as well ;)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebacd8da5d77" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-labs/unleashing-ucx-v0-18-0-a-powerful-release-for-seamless-migrations-and-improved-compatibility-ebacd8da5d77">🚀 Unleashing UCX v0.18.0: A Powerful Release for Seamless Migrations and Improved Compatibility</a> was originally published in <a href="https://medium.com/databricks-labs">Databricks Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New minor release of UCX v0.11.0 by Databricks Labs]]></title>
            <link>https://medium.com/databricks-labs/new-minor-release-of-ucx-v0-11-0-by-databricks-labs-145470efa7e0?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/145470efa7e0</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Fri, 26 Jan 2024 17:41:37 GMT</pubDate>
            <atom:updated>2024-01-26T17:46:15.757Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dm2iHRYsiE3HGgACTPvsYw.png" /></figure><p>We are thrilled to announce the release of UCX v0.11.0, the ultimate companion for upgrading to the Unity Catalog. To learn what happened in v0.10.0, see <a href="https://medium.com/databricks-labs/introducing-ucx-v0-10-0-enhanced-group-management-azure-resource-role-assignments-and-more-2cc826b58ff4">the previous post</a>. Our latest updates are designed to provide you with an even smoother transition to Unity Catalog while enhancing functionality and improving user experience. Here’s what’s new:</p><p>🚀 Introducing the databricks labs ucx alias command allows you to create a view of tables from one schema or catalog in another schema or catalog. This new feature adds flexibility and simplifies the upgrade process to Unity Catalog (<a href="https://github.com/databrickslabs/ucx/pull/837">#837</a>).</p><p>🔒 Prepare for migration of AWS S3 IAM storage credentials with databricks labs ucx save-aws-iam-profiles command. This feature scans your instance profiles, identifies AWS S3 access, and saves a CSV with permission details, enabling you to stay on top of your security settings and later create the relevant Unity Catalog storage credentials (<a href="https://github.com/databrickslabs/ucx/pull/817">#817</a>).</p><p>📊 Assessment Dashboard upgrade! With the addition of total view counts, you can now better understand your data usage and make informed decisions (<a href="https://github.com/databrickslabs/ucx/pull/834">#834</a>).</p><p>♻️ Internal enhancements: The assess_jobs and assess_clusters tasks in the assessment workflow has been cleaned up, improving testing and reducing redundancy (<a href="https://github.com/databrickslabs/ucx/pull/825">#825</a>).</p><p>📄 Assessment report documentation is now available, making it easier to generate, interpret, and act upon the assessment report (<a href="https://github.com/databrickslabs/ucx/pull/806">#806</a>).</p><p>🐛 Fixed an escaping issue for SQL object names, ensuring seamless interaction with your SQL objects (<a href="https://github.com/databrickslabs/ucx/pull/836">#836</a>).</p><p>📌 Dependency updates: The databricks-sdk requirement has been updated from ~=0.17.0 to ~=0.18.0, ensuring that you have the latest version for optimal functionality (<a href="https://github.com/databrickslabs/ucx/pull/832">#832</a>).</p><p>Upgrade to UCX v0.11.0 today and discover the improvements that will make your journey to Unity Catalog even more rewarding!</p><p><a href="https://github.com/databrickslabs/ucx/releases/tag/v0.11.0">Read the release notes on GitHub for more information</a>. Happy upgrading! 💼🚀</p><p><em>Please share and subscribe to </em><a href="https://medium.com/databricks-labs/newsletters/updates-from-databricks-labs"><em>Updates from Databricks Labs</em></a><em> newsletter to stay up-to-date with the latest releases from </em><a href="https://github.com/databrickslabs"><em>GitHub</em></a><em> namespace. Subscribing ensures you’re the first to know about the latest enhancements, bug fixes, and exciting features that will take your Data Intelligence Platform experience to the next level. You‘re encouraged to </em><a href="https://github.com/databrickslabs"><em>follow</em></a><em> the GitHub org as well ;)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=145470efa7e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-labs/new-minor-release-of-ucx-v0-11-0-by-databricks-labs-145470efa7e0">💡New minor release of UCX v0.11.0 by Databricks Labs</a> was originally published in <a href="https://medium.com/databricks-labs">Databricks Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing UCX v0.10.0: Enhanced Group Management, Azure Resource Role Assignments, and More!]]></title>
            <link>https://medium.com/databricks-labs/introducing-ucx-v0-10-0-enhanced-group-management-azure-resource-role-assignments-and-more-2cc826b58ff4?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cc826b58ff4</guid>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Fri, 19 Jan 2024 18:50:25 GMT</pubDate>
            <atom:updated>2024-01-19T18:52:04.060Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re thrilled to announce the release of UCX v0.10.0, which is designed to improve your Databricks Unity Catalog upgrade experience. This version includes several new features, bug fixes, and improvements, so let’s dive in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uO9vSMYS9OFOQ6yTtwkTxg.png" /></figure><p>This release adds the databricks labs ucx validate-groups-membership command, which validates the membership of groups across account and workspace levels. This group manager component ensures that group management works smoothly and reliably, providing valuable insights into potential discrepancies. <a href="https://github.com/databrickslabs/ucx/pull/772">#772</a></p><p>Introducing the save-azure-storage-accounts command, which identifies all storage accounts used by tables in an HMS and saves their corresponding service principals and permissions in a CSV file on the workspace. This feature uses the new AzureResourcePermissions.save_spn_permissions method, AzureResources class, AzureRoleAssignment, and Principal classes. <a href="https://github.com/databrickslabs/ucx/pull/764">#764</a></p><p>A new optional debug_truncate_bytes parameter has been added to the config, allowing for an increased truncation limit when logging API requests and responses. This improvement facilitates easier debugging for UCX users. <a href="https://github.com/databrickslabs/ucx/pull/782">#782</a></p><p>This release adds support for crawling grants and applying Hive Metastore UDF ACLs. Error handling for delta issues in crawlers and hive metastore has been extended, making the tool more robust and reliable. <a href="https://github.com/databrickslabs/ucx/pull/812">#812</a>, <a href="https://github.com/databrickslabs/ucx/pull/795">#795</a></p><p>The Python requirement has been changed from 3.10.6 to 3.10, allowing for compatibility with any Python version greater than or equal to 3.10. <a href="https://github.com/databrickslabs/ucx/pull/805">#805</a></p><p>This release also includes various bug fixes, refactoring, and improved code readability. The databricks labs ucx repair-run command now executes correctly, the handling of DELTASHARING table format has been fixed, and the listing of workflows via CLI has been improved, among other enhancements. <a href="https://github.com/databrickslabs/ucx/pull/801">#801</a>, <a href="https://github.com/databrickslabs/ucx/pull/802">#802</a>, <a href="https://github.com/databrickslabs/ucx/pull/811">#811</a>, <a href="https://github.com/databrickslabs/ucx/pull/777">#777</a>, <a href="https://github.com/databrickslabs/ucx/pull/814">#814</a>, <a href="https://github.com/databrickslabs/ucx/pull/794">#794</a></p><p>Read the full version changelog <a href="https://github.com/databrickslabs/ucx/blob/main/CHANGELOG.md">here</a>.</p><p>Join us in adopting UCX v0.10.0 for enhanced Databricks Unity Catalog upgrades, and stay tuned for future posts here on this channel! ✨</p><p><em>Please share and subscribe to </em><a href="https://medium.com/databricks-labs/newsletters/updates-from-databricks-labs"><em>Updates from Databricks Labs</em></a><em> newsletter to stay up-to-date with the latest releases from </em><a href="https://github.com/databrickslabs"><em>GitHub</em></a><em> namespace. Subscribing ensures you’re the first to know about the latest enhancements, bug fixes, and exciting features that will take your Data Intelligence Platform experience to the next level. You‘re encouraged to </em><a href="https://github.com/databrickslabs"><em>follow</em></a><em> the GitHub org as well ;)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cc826b58ff4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-labs/introducing-ucx-v0-10-0-enhanced-group-management-azure-resource-role-assignments-and-more-2cc826b58ff4">Introducing UCX v0.10.0: Enhanced Group Management, Azure Resource Role Assignments, and More!</a> was originally published in <a href="https://medium.com/databricks-labs">Databricks Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trivial Terminal User Interfaces (TUI) with Databricks Labs Blueprint]]></title>
            <link>https://medium.com/databricks-labs/trivial-terminal-user-interfaces-tui-with-databricks-labs-blueprint-c19de020964c?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/c19de020964c</guid>
            <category><![CDATA[databricks-labs]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Mon, 15 Jan 2024 18:16:35 GMT</pubDate>
            <atom:updated>2024-01-15T18:16:35.861Z</atom:updated>
            <content:encoded><![CDATA[<p>Ensuring testable interactivity is a crucial aspect that cannot be overlooked. Thankfully, the solution lies in leveraging the powerful capabilities of the databricks.labs.blueprint.tui. This essential import opens up a world of possibilities for enhancing the user experience and facilitating thorough testing of your command-line apps. Let’s delve into a few illustrative examples of how Prompts can elevate the interactivity of your applications. We’ve decoupled this library from the installer of the <a href="https://github.com/databrickslabs/ucx">UCX</a> project so that more people can use it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*GksQ4jZat7iI-GLe.gif" /></figure><h4>Simple Text Questions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/0*8f9B28oCGd4lclNm.gif" /></figure><p>When it comes to soliciting user input in command-line applications, the question() method proves to be a more sophisticated alternative to the standard input() built-in function. By incorporating the Prompts module from databricks.labs.blueprint.tui, you can introduce a higher level of user engagement and control. Let’s explore the versatility of prompts.question() through a practical example:</p><pre>from databricks.labs.blueprint.tui import Prompts<br><br>prompts = Prompts()<br>answer = prompts.question(&#39;Enter a year&#39;, default=&#39;2024&#39;, valid_number=True)<br>print(answer)</pre><p>This concise snippet initiates a user prompt for entering a year. However, the features don’t stop there; optional arguments further amplify its functionality:</p><ul><li>default: This argument allows you to provide a default value that will be used if the user doesn’t input anything.</li><li>max_attempts <em>(default 10)</em>: With this parameter, you can specify the maximum number of attempts before an exception is raised, offering robust error handling for invalid or empty inputs.</li><li>valid_number: Setting this to True enforces that the user’s input must be a valid number.</li><li>valid_regex: This parameter ensures that the input adheres to a specified regular expression.</li><li>validate: This option enables you to define a custom validation function, such as a lambda expression (lambda x: &#39;awesome&#39; in x), allowing for intricate validation beyond basic checks.</li></ul><h4>Confirming Actions</h4><p>Ensuring the safety and intentionality of optional or potentially destructive actions within your command-line application is paramount. The confirm() method emerges as a reliable guardian for such scenarios, allowing users to confirm their choices explicitly. Here’s a practical implementation in action:</p><pre>if prompts.confirm(&#39;Destroy database?&#39;):<br>    print(&#39;DESTROYING DATABASE&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/0*ews8ni8EdqYZE45Z.gif" /></figure><p>In this succinct snippet, the script prompts the user with a confirmation, asking whether they truly want to destroy the database. The following condition ensures that the subsequent action, in this case, the message ‘DESTROYING DATABASE,’ is only executed if the user affirms the confirmation prompt. This straightforward yet robust approach adds a layer of security to your application and empowers users to make informed decisions when faced with optional or potentially hazardous actions. It acts as a guardian, preventing accidental or uninformed execution of critical commands, thereby enhancing your command-line app&#39;s reliability and user-friendly nature.</p><h4>Single Choice from a List</h4><p>Empowering users to select from a predefined list is a common requirement in command-line applications, and the choice() method is tailor-made for such scenarios. Here’s an illustrative example:</p><pre>answer = prompts.choice(&#39;Select a language&#39;, [&#39;Python&#39;, &#39;Rust&#39;, &#39;Go&#39;, &#39;Java&#39;])<br>print(answer)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/0*GimU0CUfoUT6WW0t.gif" /></figure><p>In this concise code snippet, the choice()method initiates an interactive prompt for the user to select a language from the provided list: Python, Rust, Go, or Java. The selected choice is then stored in the ‘answer’ variable and is subsequently printed to the console. This feature enhances the user experience by providing a structured and intuitive way to make selections and ensures that the chosen value aligns with the predefined options.</p><h4>Single Choice from Dictionary</h4><p>Facilitating user selection from a dictionary while presenting them with sorted keys is made effortless by the choice_from_dict() method. Here’s a practical example to showcase its functionality:</p><pre>answer = prompts.choice_from_dict(&#39;Select a locale&#39;, {<br>    &#39;Українська&#39;: &#39;ua&#39;,<br>    &#39;English&#39;: &#39;en&#39;<br>})<br>print(f&#39;Locale is: {answer}&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/0*lyq8ksqZ3Q91kwcl.gif" /></figure><p>In this succinct code snippet, the prompts.choice_from_dict() triggers an interactive prompt, displaying sorted keys from the provided dictionary: Українська key mapped to ua value and English mapped to en. The user is prompted to choose a locale, and the selected value is stored in the answer variable. Subsequently, the chosen locale is printed to the console. It also enhances clarity by presenting the options in a sorted order. This feature is particularly useful when dealing with any scenario where presenting options in an organized manner is essential for a more user-friendly experience in your command-line applications.</p><h4>Multiple Choices from Dictionary</h4><p>Enabling users to select multiple items from a dictionary in a clear and organized manner is effortlessly achieved through the multiple_choice_from_dict() method. Here’s an example illustrating its usage:</p><pre>answer = prompts.multiple_choice_from_dict(<br>    &#39;What projects are written in Python? Select [DONE] when ready.&#39;, {<br>    &#39;Databricks Labs UCX&#39;: &#39;ucx&#39;,<br>    &#39;Databricks SDK for Python&#39;: &#39;sdk-py&#39;,<br>    &#39;Databricks SDK for Go&#39;: &#39;sdk-go&#39;,<br>    &#39;Databricks CLI&#39;: &#39;cli&#39;,<br>})<br>print(f&#39;Answer is: {answer}&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/0*DUEEWqCg3izf25gW" /></figure><p>In this concise code snippet, prompts.multiple_choice_from_dict() initiates an interactive prompt, presenting the user with a selection of projects written in Python. The user can choose multiple items, and the selected values are stored in the answer variable. Once the user is done making selections, the chosen answers are printed to the console. This functionality is particularly useful when users must make multiple selections from a predefined set of options, providing a seamless and user-friendly way to gather input in your command-line applications. This ensures clarity and ease of use, enhancing the overall interactive experience for the user.</p><h4>Unit Testing Prompts</h4><p>Testing user interactions in a controlled environment is essential for ensuring the robustness of your code. The MockPrompts class is designed to facilitate this by allowing you to simulate user input for testing purposes. Here’s an example test for the question method using MockPrompts with regular expressions as keys and values as answers:</p><pre>from databricks.labs.blueprint.tui import MockPrompts<br><br>def test_ask_for_int():<br>    # Define a MockPrompts instance with a regular expression as the key<br>    # and an empty string as the default answer for any input.<br>    prompts = MockPrompts({r&quot;.*&quot;: &quot;&quot;})<br>    <br>    # Simulate a user interaction by calling the question method.<br>    # The default answer will be returned as the response.<br>    res = prompts.question(&quot;Number of threads&quot;, default=&quot;8&quot;, valid_number=True)<br>    <br>    # Assert that the result matches the expected default answer.<br>    assert &quot;8&quot; == res</pre><p>In this test, the MockPrompts instance is configured to match any input (using the regular expression .*) and respond with an empty string. This allows the test to focus on the behavior of the question method with a valid_number constraint and a default value. The assertion checks whether the result matches the expected default answer. This approach provides a controlled environment for testing your interactive prompts, allowing you to simulate various user inputs and ensure that your code handles them correctly.</p><h3>Installation</h3><p>If you build apps on top of Databricks Lakehouse, go ahead and try this <em>experimental</em> library out by installing it via pip install databricks-labs-blueprint. Please give it a star on <a href="https://github.com/databrickslabs/blueprint">GitHub</a> and <a href="https://pypi.org/project/databricks-labs-blueprint/">PyPI</a> if you&#39;d like.</p><p><em>Please share and subscribe to </em><a href="https://medium.com/databricks-labs/newsletters/updates-from-databricks-labs"><em>Updates from Databricks Labs</em></a><em> newsletter to stay up-to-date with the latest releases from </em><a href="https://github.com/databrickslabs"><em>GitHub</em></a><em> namespace. Subscribing ensures you’re the first to know about the latest enhancements, bug fixes, and exciting features that will take your Data Intelligence Platform experience to the next level. You‘re encouraged to </em><a href="https://github.com/databrickslabs"><em>follow</em></a><em> the GitHub org as well ;)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c19de020964c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-labs/trivial-terminal-user-interfaces-tui-with-databricks-labs-blueprint-c19de020964c">Trivial Terminal User Interfaces (TUI) with Databricks Labs Blueprint</a> was originally published in <a href="https://medium.com/databricks-labs">Databricks Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Introducing UCX v0.9.0: Enhanced Assessment, Migration, and Error Handling]]></title>
            <link>https://medium.com/databricks-labs/introducing-ucx-v0-9-0-enhanced-assessment-migration-and-error-handling-3ccc006e0e26?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ccc006e0e26</guid>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 21:55:51 GMT</pubDate>
            <atom:updated>2024-01-12T21:55:51.223Z</atom:updated>
            <content:encoded><![CDATA[<h3>🚀 Introducing UCX v0.9.0: Enhanced Assessment, Migration, and Error Handling for Unity Catalog Upgrades</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wV3g650v13Yw5omdsz9Aog.png" /></figure><p>We are thrilled to announce the release of UCX v0.9.0, a minor release that includes enhanced assessment, migration, and error handling. UCX is your trusted companion, guiding you through the account, groups, workspaces, and jobs upgrade process. Here’s a summary of the most important improvements in this release:</p><ul><li>A new assessment step has been added to estimate the size of DBFS root tables (<a href="https://github.com/databrickslabs/ucx/pull/741">#741</a>). This enhancement helps you decide whether a table is a good candidate for cloning by providing you with the necessary information. This feature calculates the table size by executing spark._jsparkSession.table(&quot;db.table&quot;).queryExecution().analyzed().stats().sizeInBytes().</li><li>Introducing the databricks labs ucx move command, which allows you to move tables and schemas between catalogs (<a href="https://github.com/databrickslabs/ucx/pull/756">#756</a>). The databricks labs ucx move command supports moving tables from one schema to another within the same catalog, moving tables from one schema to another across catalogs, moving schemas from one catalog to another, and moving relevant permissions.</li><li>The migration method is now determined based on whether the table is on the DBFS root (<a href="https://github.com/databrickslabs/ucx/pull/759">#759</a>). This ensures optimal migration based on the table’s location, making the process more efficient. The migration functionality uses different approaches based on the table location, such as copying the table content, deep cloning, syncing it to an external UC table, and skipping the table if it points to a Databricks dataset.</li><li>Error handling has been enhanced, including retry and rate limit adjustments for renamed workspace group operation and corrected rate limits for reflecting account groups to workspaces (<a href="https://github.com/databrickslabs/ucx/pull/751">#751</a>). Improved error handling ensures better stability and user experience during the migration process.</li><li>The UCX project now incorporates the <a href="https://pypi.org/project/databricks-labs-blueprint/">databricks-labs-blueprint</a> library for common utilities, ensuring compatibility and reusability across various projects (<a href="https://github.com/databrickslabs/ucx/pull/758">#758</a>).</li><li>Query execution exceptions from within Databricks Workflows have been converted to SDK exceptions, improving exception handling and stability (<a href="https://github.com/databrickslabs/ucx/pull/769">#769</a>). Error message inference from failed workflow runs has been improved, addressing integration test flakiness and enhancing the user experience (<a href="https://github.com/databrickslabs/ucx/pull/753">#753</a>). These changes ensure better error handling and consistent exception types for improved user experience.</li><li>Addressing an issue where missing users and temp groups appeared after workspace-local groups migration and ensuring the crawling task skips a table if it no longer exists (<a href="https://github.com/databrickslabs/ucx/pull/770">#770</a>). This improvement increases the stability of the assessment job and integration tests.</li><li>The assessment dashboard has been reorganized for better readability and clarity (<a href="https://github.com/databrickslabs/ucx/pull/738">#738</a>). The reorganized assessment report includes the quantity columns in all rows, making it more user-friendly and informative.</li><li>A new feature ensures that the metastore exists in the current workspace, further optimizing the upgrade process (<a href="https://github.com/databrickslabs/ucx/pull/735">#735</a>).</li></ul><p>Experience the full potential of Unity Catalog with UCX v0.9.0 and unlock the benefits of these fantastic features! 🔓✨</p><p><em>Please share and subscribe to </em><a href="https://medium.com/databricks-labs/newsletters/updates-from-databricks-labs"><em>Updates from Databricks Labs</em></a><em> newsletter to stay up-to-date with the latest releases from </em><a href="https://github.com/databrickslabs"><em>GitHub</em></a><em> namespace. Subscribing ensures you’re the first to know about the latest enhancements, bug fixes, and exciting features that will take your Data Intelligence Platform experience to the next level. You‘re encouraged to </em><a href="https://github.com/databrickslabs"><em>follow</em></a><em> the GitHub org as well ;)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ccc006e0e26" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-labs/introducing-ucx-v0-9-0-enhanced-assessment-migration-and-error-handling-3ccc006e0e26">🚀 Introducing UCX v0.9.0: Enhanced Assessment, Migration, and Error Handling</a> was originally published in <a href="https://medium.com/databricks-labs">Databricks Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reflecting on the Year 2023]]></title>
            <link>https://medium.com/@nf-x/reflecting-on-the-year-2023-7bfef22d572c?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/7bfef22d572c</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[year-in-review]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Sun, 31 Dec 2023 21:00:22 GMT</pubDate>
            <atom:updated>2024-01-14T15:41:07.338Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NzJQvKK9tKmZmtoy.png" /></figure><p>As we bid adieu to 2023, I’m thrilled to share the incredible journey of open-source work and innovation that unfolded throughout the year. It’s been a year marked by challenges, triumphs, and the relentless pursuit of excellence in tech and development.</p><h3>Databricks SDK for Python</h3><p>In April, I achieved a significant milestone with the launch of the <a href="https://github.com/databricks/databricks-sdk-py">Databricks SDK for Python</a> . Running with roughly <a href="https://pypistats.org/packages/databricks-sdk">3–4 million monthly downloads</a> — <em>isn’t that great?</em> What makes it even more noteworthy is the ingenious infrastructure reuse from <a href="https://ssmertin.com/articles/oss-year-2022-in-review-projects-launched/">the previous year’s</a> <a href="https://github.com/databricks/databricks-sdk-go">Go SDK for Databricks</a> . Overcoming hurdles in a large organisation with over 5000 people, I navigated the release under the Apache 2.0 license. I’ve approached driving adoption by implementing an <a href="https://github.com/databricks/databricks-sdk-go/commit/e798a9c4c1d9ccfa0451eb979aaaa7914ce79a92">integration</a> <a href="https://github.com/databricks/databricks-sdk-go/commit/04377a7ef21ae0abc9926ed299377494caa0c794">test</a> <a href="https://github.com/databricks/databricks-sdk-go/commit/7c15458a2365499ebb3eafb269375ba293e5eea3">cross-compiler</a> into runnable <a href="https://github.com/databricks/databricks-sdk-py/commit/5f566dffbfd0b65ec7b74603d02524ce98647853">Python</a> (and <a href="https://github.com/databricks/databricks-sdk-go/commit/4ae857a7acba6256b30567809d8c39a499944e2d">Go</a>) code examples over a hackathon in roughly 1000 lines of code, making it easier to maintain thousands of code examples over 7 different languages. And, according to this email from PyPI, <em>it paid off</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*tWnVg5fq_Rf8_JUZ.jpeg" /><figcaption>Got this email quickly after launching the <a href="https://pypi.org/project/databricks-sdk/">databricks-sdk</a> package</figcaption></figure><p>Another challenge was integrating the <a href="https://docs.databricks.com/en/dev-tools/auth/index.html">Unified Authentication layer</a> of Python SDK with <a href="https://grpc.io/docs/guides/auth/">gRPC transport interceptors</a> so that all the recent versions of <a href="https://pypi.org/project/databricks-connect">Databricks Connect</a> can “just work” without any configuration with the SDK-enabled ecosystem, like using the same environment variables or magically picking up auth tokens from Azure CLI or Databricks CLI. It also required a <a href="https://github.com/apache/spark/pull/40993">change to Apache Spark</a> .</p><h3>Reboot of Databricks CLI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*VhW07wx3zLo_1wuE.jpg" /></figure><p>May saw the big reboot of <a href="https://github.com/databricks/cli">Databricks CLI</a> , which was rewritten from Python to Go. The main driver behind a complete rewrite was the simplicity of dependencies and packaging for Go apps — cross-compiling a static binary across all the required operating systems and platforms gave our team way more freedom of choice for power features. Python imposed a significant maintenance challenge with dependencies across all the necessary platforms and environments. Same <a href="https://github.com/databricks/cli/commit/4c4a29301501caa9e7489446e3fd4cf8a84ac7b0">code generation infrastructure</a> as for Go SDK, Python SDK, and Java SDK, reused <a href="https://docs.databricks.com/en/dev-tools/auth/index.html">Unified Authentication layer</a> . And great experimental interactivity features, like progress spinners for long-running operations or drop-downs for unfilled IDs.</p><h3>Expanding the SDK ecosystem with Java and R</h3><p>May saw the launch of the <a href="https://github.com/databricks/databricks-sdk-java">Java SDK for Databricks</a> , adopting the same robust code generation infrastructure and authentication experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*GqmkNFo7jJzdoFoG.jpg" /><figcaption><a href="https://central.sonatype.com/artifact/com.databricks/databricks-sdk-java/overview"><em>com.databricks:databricks-sdk-java</em></a><em> DevX with god-object pattern for easier service discovery</em></figcaption></figure><p>June brought forth the birth of the <a href="https://github.com/databrickslabs/databricks-sdk-r">Databricks SDK for R</a> , an experimental yet impactful addition to the SDK family. With a shared code generation infrastructure, this project broke ground despite my R coding hiatus since 2017. ChatGPT helped me to write the working core of services just within two days — I was mostly asking things like <em>“How to translate this code in Go to R?”</em> or <em>“What does this code mean?”</em> because R has quite a distinctive syntax and quite different idioms from all the languages I’m working with. However, the <a href="https://adv-r.hadley.nz/">Advanced R</a> book by <a href="https://hadley.nz/">Hadley Wickham</a> helped me better understand the language and polish all the code generation templates right before the initial release. I initially thought <a href="https://r4ds.had.co.nz/">R for Data Science</a> book could help build better libraries, but that has a different focus. The journey to CRAN is ongoing, with a promising future for this experimental endeavor.</p><h3>Nurturing Innovation in Databricks Labs</h3><p>Throughout the year, I helped launch various projects within Databricks Labs, such as <a href="https://github.com/databrickslabs/discoverx">DiscoverX</a> , <a href="https://github.com/databrickslabs/doc-qa">Doc QA LLM chatbot</a> , and <a href="https://github.com/databrickslabs/dlt-meta">DLT Meta</a> . Ensuring the right design for long-term success became a priority, and the commitment to excellence in these projects mirrors the broader ethos of open-source development. November marked the rollout of the <a href="https://en.wikipedia.org/wiki/Contributor_License_Agreement">Contributor License Agreement</a> (CLA) <a href="https://github.com/cla-assistant/cla-assistant">Assistant</a> for Databricks Labs, enhancing our ability to track and accept external contributions. This move strengthens our commitment to fostering a vibrant and collaborative community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*ucgprTCC7p0_Kcl5.jpg" /><figcaption><a href="https://medium.com/databricks-labs/upgrading-jobs-code-between-databricks-runtime-versions-made-seamless-29c22e701c3a"><em>Comparing libraries</em></a><em> across different runtime versions.</em></figcaption></figure><p>Wrapping up the year in December, the launch of <a href="https://github.com/databrickslabs/sandbox">Databricks Labs Sandbox</a> promises an even more experimental platform. With a consistent CLI front end and existing release infrastructure, this sandbox sets the stage for prototyping growth in the coming year.</p><h3>Release Processes for the Shippable Software</h3><p>Managing numerous tools and libraries with frequent releases is no small feat. This year, I’ve launched plenty of tools. Those require regular releases, each release taking approximately 20–40 minutes per library — regenerating code where necessary, documenting Public API differences and breaking changes, bumping the <a href="https://semver.org/">semantic version</a> accordingly, collecting merged pull requests since the last release, updating the changelog file with all that (but in a way the release notes are editable), making that changelog part of the release pull request, waiting for the pull request to merge, tagging a commit on the main branch, pushing that tag and creating a GitHub release with the same release notes, as in the changelog. All these tools and libraries are released almost every other week. So I’ve decided to save many man-weeks-per-year by automating the release process to something between 2 and 15 minutes, depending on how fast CI runs and how well people make their pull request titles. This <a href="https://www.linkedin.com/posts/smertin_release-v0130-databricksdatabricks-sdk-go-activity-7087082430460776448-Pgn2">monumental step</a> has streamlined our workflow, allowing for faster <a href="https://github.com/databrickslabs/ucx/releases">iterations</a> <a href="https://github.com/databrickslabs/databricks-sdk-r/releases">and</a> <a href="https://github.com/databricks/databricks-sdk-java/releases">more</a> <a href="https://github.com/databricks/databricks-sdk-py/releases">frequent</a> <a href="https://github.com/databricks/databricks-sdk-go/releases">releases</a>.</p><h3>Technical Book Reviews</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/0*43N5I8IHSKf2aBVz.jpeg" /><figcaption><a href="https://manning.com/books/learn-concurrent-programming-with-go">Learn Concurrent Programming with Go</a> by <a href="https://cutajarjames.com/">James Cutajar</a></figcaption></figure><p>This year, I’ve tried myself as a technical book reviewer, volunteering time to read books before they see the print. Manning Publishing suggested reviewing two titles, drawing on my extensive experience in Go and Terraform development. Apparently, I have written most of the code in <a href="https://ssmertin.com/tags/go">Go</a> for the past couple of years, so I enjoyed reading <a href="https://manning.com/books/learn-concurrent-programming-with-go">Learn Concurrent Programming with Go</a> by <a href="https://cutajarjames.com/">James Cutajar</a> . It was an interesting angle compared to the one presented in <a href="https://ssmertin.com/articles/tech-book-reviews-go/#concurrency-in-go">Concurrency in Go</a> by <a href="https://katherine.cox-buday.com/">Katherine Cox-Buday</a> . Since I created a <a href="https://ssmertin.com/tags/terraform/">Terraform</a> provider, that has more <a href="https://registry.terraform.io/providers/databricks/databricks/latest">than 20 million downloads</a> , it was a pleasure for me to review some chapters of <a href="https://www.manning.com/books/terraform-in-depth">Terraform in Depth (MEAP)</a> by <a href="https://blog.tedivm.com/">Robert Hafner</a> . There are not many books on the subject, so every fresh perspective benefits the practitioner community.</p><p>Getting a book out is a commitment that can easily take more than a year or so. It takes a lot of people to release a book, and hats off to the authors who made it.</p><h3>More automation</h3><p>September witnessed the launch of <a href="https://github.com/databrickslabs/ucx">UCX</a> , a project automating the upgrade of Databricks Workspaces from Hive Metastore to Unity Catalog. Leveraging the Python SDK from April, this venture is pivotal in advancing the SDK to a more production-ready state. Much effort was invested into productizing the collection of existing disparate scripts, failure recovery, automated integration testing infrastructure, and making the installation and upgrade journey easy for users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*KSoo0x-0TuD9IYz8.jpg" /></figure><p>This was one of the reasons I created <a href="https://github.com/databricks/cli/pull/914">a plugin infrastructure</a> within Databricks CLI via the databricks labs command group so that all projects in Databricks Labs can have a consistent CLI and installation user experience. Luckily, the automated release infrastructure was already there and required only a bit of a change to apply it to more projects.</p><h3>Writing more blogs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*28u8CoykVogcvBXx.jpg" /></figure><p>This year I experimented more with Search Engine Optimisation (SEO) for this blog, and, according to Google Analytics, it started to work somewhere at the end of April, probably because of <a href="https://ssmertin.com/articles/into-to-data-quality-with-apache-spark/">this article</a> . I have plenty of <a href="https://ssmertin.com/talks/">public talks</a> , that I still want to turn into nice and expanded articles — not the other way around:</p><ul><li><a href="https://medium.com/mlopshowto/introduction-to-data-quality-with-apache-spark-5f2525ec3882?source=user_profile---------9----------------------------">Introduction to Data Quality with Apache Spark</a></li><li><a href="https://medium.com/mlopshowto/strategies-for-data-quality-with-apache-spark-4688632f149a?source=user_profile---------8----------------------------">Strategies for Data Quality With Apache Spark</a></li></ul><p>I mainly write for myself and appreciate when other people find it useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*_bu5IfJK24gzJJhG.jpg" /></figure><p>I’ve also started using Medium as a way to deliver release notes for <a href="https://medium.com/databricks-labs">Databricks Labs</a> , but cannot tell if people like that or not — it’s only been there for less than two months.</p><p>In retrospect, 2023 has been a year of innovation, challenges met head-on and a relentless pursuit of excellence. As we eagerly anticipate the dawn of a new year, the lessons learned and achievements unlocked in the realm of open-source development will undoubtedly propel us toward even greater heights. Here’s to another year of groundbreaking contributions and collaborative success! 🚀</p><h3>See Also</h3><ul><li><a href="https://medium.com/better-programming/open-source-dependencies-is-it-the-holy-grail-or-a-can-of-worms-2954e1e1cfc6?source=user_profile---------16----------------------------">Open Source: Is It the Holy Grail or a Can of Worms?</a></li><li><a href="https://ssmertin.com/talks/databricks-software-development-kits/">Unlocking the Power of Databricks SDKs</a> (talk)</li><li><a href="https://medium.com/@nf-x/oss-year-2022-in-review-projects-launched-e7ca0b1ce3d6?source=user_profile---------14----------------------------">OSS Year 2022 in Review: Projects Launched</a></li><li><a href="https://medium.com/@nf-x/what-happened-if-unit-tests-unlock-self-healing-in-go-197145a8a903?source=user_profile---------12----------------------------">What Happened if Unit-Tests Unlock Self-Healing in Go?</a></li><li><a href="https://medium.com/better-programming/fingerprinting-process-trees-on-linux-with-rust-ab21f73d03be?source=user_profile---------10----------------------------">Fingerprinting Process Trees on Linux With Rust</a></li></ul><p><em>If you’ve enjoyed reading this article, there’s a good chance others will too! By sharing this article on social media using the buttons at the top of the page, you’re helping to spread valuable information to your network and potentially even beyond. Please subscribe to the RSS feed to stay up-to-date with the upcoming content.</em></p><p><em>Originally published at </em><a href="https://ssmertin.com/articles/reflecting-on-year-2023/"><em>https://ssmertin.com</em></a><em> on December 31, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bfef22d572c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Introducing UCX v0.8.0: Powerful New Features to Streamline Your Unity Catalog Migration ]]></title>
            <link>https://medium.com/databricks-labs/introducing-ucx-v0-8-0-powerful-new-features-to-streamline-your-unity-catalog-migration-828d8c9b7aef?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/828d8c9b7aef</guid>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Thu, 28 Dec 2023 20:03:38 GMT</pubDate>
            <atom:updated>2023-12-28T20:03:38.707Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re excited to announce the release of <a href="https://github.com/databrickslabs/ucx/releases/tag/v0.8.0">UCX v0.8.0</a> by Databricks Labs, packed with new features and improvements to streamline your data migration process! With this update, we aim to provide you with an even more efficient and user-friendly experience. Here’s a roundup of the most exciting new features in UCX v0.8.0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uO9vSMYS9OFOQ6yTtwkTxg.png" /></figure><p><strong>Effortlessly Repair Failed Workflows with the </strong><strong>repair-run Command.</strong> No more headaches when workflows fail! With the new databricks labs ucx repair-run --step ... CLI command, you can perform a repair run of any failed workflows, like assessment, migrate-groups, and more. Effortlessly pick up where you left off! (<a href="https://github.com/databrickslabs/ucx/pull/724">#724</a>)</p><p><strong>Revert Migrated Tables with Ease.</strong> Made a mistake during migration? Don’t worry! The new databricks labs ucx revert-migrated-table command allows you to revert changes made to a table during the migration process. (<a href="https://github.com/databrickslabs/ucx/pull/729">#729</a>)</p><p><strong>Specify Group Lists for Enhanced Matching.</strong> We’ve now made it possible to specify a group list when using group match options, providing even more control over the adoption of Identity Federation. (<a href="https://github.com/databrickslabs/ucx/pull/725">#725</a>)</p><p><strong>Enhanced Installation Experience and Improved Logging.</strong> Upgrading UCX? We’ve resolved issues related to upgrading from older versions, making the transition as smooth as possible. Plus, we’ve improved logs to keep you informed throughout the process. (<a href="https://github.com/databrickslabs/ucx/pull/740">#740</a>)</p><p><strong>Rename Failure Summary to Assessment Summary.</strong> To better reflect the purpose of this panel, we’ve renamed the “Failure Summary” to “Assessment Summary.” Get a clearer overview of the assessment results in an instant. (<a href="https://github.com/databrickslabs/ucx/pull/733">#733</a>)</p><p><strong>Improved Error Handling and Documentation.</strong> We’ve implemented a retry mechanism to handle internal errors when getting permissions and have updated legacy table ACL documentation for clarity. (<a href="https://github.com/databrickslabs/ucx/pull/728">#728</a>)</p><p><strong>Faster Installer Execution.</strong> We’ve boosted installer execution speed, saving you valuable time during setup. (<a href="https://github.com/databrickslabs/ucx/pull/727">#727</a>)</p><p>Happy Unity Catalog migration with UCX! 💥</p><p><em>Please share and subscribe to </em><a href="https://medium.com/databricks-labs/newsletters/updates-from-databricks-labs"><em>Updates from Databricks Labs</em></a><em> newsletter to stay up-to-date with the latest releases from </em><a href="https://github.com/databrickslabs"><em>GitHub</em></a><em> namespace. Subscribing ensures you’re the first to know about the latest enhancements, bug fixes, and exciting features that will take your Data Intelligence Platform experience to the next level. You‘re encouraged to </em><a href="https://github.com/databrickslabs"><em>follow</em></a><em> the GitHub org as well ;)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=828d8c9b7aef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-labs/introducing-ucx-v0-8-0-powerful-new-features-to-streamline-your-unity-catalog-migration-828d8c9b7aef">🚀 Introducing UCX v0.8.0: Powerful New Features to Streamline Your Unity Catalog Migration 💼</a> was originally published in <a href="https://medium.com/databricks-labs">Databricks Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[UCX v0.7.0 by Databricks Labs — new release with CLI commands]]></title>
            <link>https://medium.com/databricks-labs/ucx-v0-7-0-by-databricks-labs-new-release-with-cli-commands-b4ddd0786598?source=rss-ecd56c5c8f37------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4ddd0786598</guid>
            <category><![CDATA[databricks-labs]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Serge Smertin]]></dc:creator>
            <pubDate>Thu, 21 Dec 2023 23:34:55 GMT</pubDate>
            <atom:updated>2023-12-21T23:34:55.731Z</atom:updated>
            <content:encoded><![CDATA[<h3>UCX v0.7.0 by Databricks Labs — new release with CLI commands</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*XskdjDxVdljiwki2.gif" /><figcaption>Installations and upgrades through <a href="https://docs.databricks.com/en/dev-tools/cli/index.html">Databricks CLI</a></figcaption></figure><p>Another week and another release of UCX sees the light of day!</p><ul><li>Added databricks labs ucx create-table-mapping and databricks labs ucx manual-workspace-info commands for CLI (<a href="https://github.com/databrickslabs/ucx/pull/682">#682</a>).</li><li>Added databricks labs ucx ensure-assessment-run to CLI commands (<a href="https://github.com/databrickslabs/ucx/pull/708">#708</a>).</li><li>Added databricks labs ucx installations command (<a href="https://github.com/databrickslabs/ucx/pull/679">#679</a>).</li><li>Added databricks labs ucx skip --schema ... --table ... command to mark table/schema for skipping in the table migration process (<a href="https://github.com/databrickslabs/ucx/pull/680">#680</a>).</li><li>Added databricks labs ucx validate-external-locations command for cli (<a href="https://github.com/databrickslabs/ucx/pull/715">#715</a>).</li><li>Added capturing ANY FILE and ANONYMOUS FUNCTION grants (<a href="https://github.com/databrickslabs/ucx/pull/653">#653</a>).</li><li>Added cluster override and handle case of write protected DBFS (<a href="https://github.com/databrickslabs/ucx/pull/610">#610</a>).</li><li>Added cluster policy selector in the installer (<a href="https://github.com/databrickslabs/ucx/pull/655">#655</a>).</li><li>Added detailed UCX pre-requisite information to README.md (<a href="https://github.com/databrickslabs/ucx/pull/689">#689</a>).</li><li>Added interactive wizard for databricks labs uninstall ucx command (<a href="https://github.com/databrickslabs/ucx/pull/657">#657</a>).</li><li>Added more granular error retry logic (<a href="https://github.com/databrickslabs/ucx/pull/704">#704</a>).</li><li>Parallel fetching of registered model identifiers was added to speed up the assessment workflow (<a href="https://github.com/databrickslabs/ucx/pull/691">#691</a>).</li><li>Added retry on workspace listing (<a href="https://github.com/databrickslabs/ucx/pull/659">#659</a>).</li><li>Added support for mapping workspace group to account group by prefix/suffix/regex/external id (<a href="https://github.com/databrickslabs/ucx/pull/650">#650</a>).</li><li>Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as crawl_tables was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file (<a href="https://github.com/databrickslabs/ucx/pull/661">#661</a>).</li><li>Changed the fields of the table “Tables” to lowercase (<a href="https://github.com/databrickslabs/ucx/pull/684">#684</a>).</li><li>Enabled integration tests for EXTERNAL table migrations (<a href="https://github.com/databrickslabs/ucx/pull/677">#677</a>).</li><li>Enforced mypy validation (<a href="https://github.com/databrickslabs/ucx/pull/713">#713</a>).</li><li>Filtering out inventory database from loading into tables and filtering out the same from grant detail view (<a href="https://github.com/databrickslabs/ucx/pull/705">#705</a>).</li><li>Fixed documentation for reflect_account_groups_on_workspace task and updated CONTRIBUTING.md guide (<a href="https://github.com/databrickslabs/ucx/pull/654">#654</a>).</li><li>Fixed secret scope apply task to raise ValueError (<a href="https://github.com/databrickslabs/ucx/pull/683">#683</a>).</li><li>Fixed legacy table ACL ownership migration and other integration testing issues (<a href="https://github.com/databrickslabs/ucx/pull/722">#722</a>).</li><li>Fixed some flaky integration tests (<a href="https://github.com/databrickslabs/ucx/pull/700">#700</a>).</li><li>New CLI command for workspace mapping (<a href="https://github.com/databrickslabs/ucx/pull/678">#678</a>).</li><li>Reduce server load for getting workspace groups and their members (<a href="https://github.com/databrickslabs/ucx/pull/666">#666</a>).</li><li>Throwing ManyError on migrate-groups tasks (<a href="https://github.com/databrickslabs/ucx/pull/710">#710</a>).</li><li>Updated installation documentation to use Databricks CLI (<a href="https://github.com/databrickslabs/ucx/pull/686">#686</a>).</li></ul><p><em>Please share and subscribe to </em><a href="https://medium.com/databricks-labs/newsletters/updates-from-databricks-labs"><em>Updates from Databricks Labs</em></a><em> newsletter to stay up-to-date with the latest releases from </em><a href="https://github.com/databrickslabs"><em>GitHub</em></a><em> namespace. Subscribing ensures you’re the first to know about the latest enhancements, bug fixes, and exciting features that will take your Data Intelligence Platform experience to the next level. You‘re encouraged to </em><a href="https://github.com/databrickslabs"><em>follow</em></a><em> the GitHub org as well ;)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4ddd0786598" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-labs/ucx-v0-7-0-by-databricks-labs-new-release-with-cli-commands-b4ddd0786598">UCX v0.7.0 by Databricks Labs — new release with CLI commands</a> was originally published in <a href="https://medium.com/databricks-labs">Databricks Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>