v0.1.0 — SDP-META (formerly DLT-META)

Install: pip install databricks-labs-sdp-meta==0.1.0
Existing users: pip install dlt-meta==0.1.0 still works via the compatibility redirect.

⚠️ Breaking Changes

• Project rename dlt-meta → sdp-meta, aligned with the Lakeflow Spark Declarative Pipelines product naming. A backward-compatibility wrapper is published so existing installations keep working with a deprecation warning. Issue #254, PR #380

  What changed	Before (v0.0.10)	After (v0.1.0)
  PyPI package	dlt-meta	databricks-labs-sdp-meta
  CLI command	databricks labs dlt-meta	databricks labs sdp-meta
  Python import	from dlt_meta import ...	from databricks.labs.sdp_meta import ...
  Main class	DLTMeta	SDPMeta
  Source layout	flat src/	src/databricks/labs/sdp_meta/

• Lakeflow Spark Declarative Pipelines API migration: DLT decorators/APIs migrated to pyspark.pipelines. Update any references that import from dlt to use the new module. Issue #274

• Renamed quarantine_table_name → quarantine_table in dataflow specs. Issue #243

Migration Guide

The legacy dlt-meta PyPI package is preserved as a thin compatibility shim that pulls in databricks-labs-sdp-meta and re-exports every public symbol with a DeprecationWarning. Legacy src.* imports from v0.0.10 also work via a sys.modules shim. Both shims will be removed in v0.2.0.

See docs/operations/migration for the step-by-step walkthrough.

Added

• Declarative Automation Bundle (DAB) template: end-to-end DAB-based deployment path for CI/CD, multi-target environments, and agent-driven flows. New CLI commands: bundle-init (with --quickstart zero-prompt fast path), bundle-add-flow, bundle-prepare-wheel, bundle-validate. PR #299, PR #350
• MCP Server support: optional MCP stdio server (databricks labs sdp-meta mcp) so MCP-capable clients (Claude Code, Cursor, Claude Desktop) can drive sdp-meta scaffolding and inspection. Install with pip install databricks-labs-sdp-meta[mcp]. PR #299
• Automatic liquid clustering (cluster_by_auto): Databricks automatically determines optimal clustering columns for bronze and silver tables. Supported via bronze_cluster_by_auto, bronze_quarantine_table_cluster_by_auto, and silver_cluster_by_auto. Issue #238
• End-to-end YAML support: YAML format now supported for onboarding files, DQE rules, silver transformations, and packaged demos — in addition to JSON. PR #299
• Row filter support: new where_clause field in silver transformations for pipeline-time row filtering, plus UC ROW FILTER support on bronze/silver tables, including coverage for multi-source CDC flows. Issue #303, Issue #306
• Multi-source AUTO CDC: multiple CDC sources can now feed into a single target via create_auto_cdc_flow. Issue #294
• build-and-upload-whl flag for onboard / deploy: builds the local sdp-meta wheel, uploads it to a UC volume, and uses it for the onboarding job — no PyPI access required on the pipeline cluster. Issue #295
• Windows PowerShell deploy script (scripts/deploy_app.ps1): native PowerShell port of scripts/deploy_app.sh using robocopy and the databricks CLI only — no Git Bash / WSL / Python required. See WINDOWS_DEPLOY.md.
• Apps UI + Git folder deploy guide (databricks_app/UI_GIT_DEPLOY.md): click-only deploy path — create a Databricks Git folder pointing at this repo, aim the App at databricks_app/, no local CLI required.
• Repo-wide line-ending policy (.gitattributes): prevents the bad interpreter: /bin/bash\r App-container crash across all tooling paths.
• Databricks App refactor: monolithic app.py split into routes/ (8 blueprints) + services/onboarding/ helpers; input validation hardened; new UC preflight probe surfaces required GRANT SQL before demos. Issue #248
• Docs migrated from Hugo to Docusaurus 3: 34 pages across 6 sections (Getting Started, Concepts, Reference, Guides, Operations, Contributing) with Databricks-branded CSS. Issue #315
• Added interactive notebook-based demo and LFC (Lakeflow Connect) Python demo. Issue #172

Changed

• Renamed "Lakeflow Declarative Pipelines" → "Lakeflow Spark Declarative Pipelines" throughout. Issue #285, PR #350
• Updated Databricks Asset Bundle → Declarative Automation Bundle terminology in demos and docs.
• Switched demo paths from DBFS to UC Volumes.

Fixed

• Git portability: renamed DAB template filenames containing literal " characters (Go text/template comparisons) to backtick literals. Fixes error: invalid path on all systems with core.protectNTFS=true (default since Git 2.22).
• Security: replaced unsafe eval() on uc_enabled widget with a strict parser; UC identifier validation added across CLI and demos. PR #260
• Security and correctness hardening from the post-release review: MCP filesystem sandbox, App XSS escaping, row-filter SQL validation, typed dataflowIds predicates, SCD2 __START_AT/__END_AT schema handling. PR #350
• Performance: O(N+M) schema modification for wide tables in CDC flows (was O(N×M)). Issue #284
• Fixed cross-platform file URI handling in CLI. Issue #251
• Fixed snapshot-based SCD processing (apply_changes_from_snapshot with snapshot_format: "delta"); renamed demo tables to sdp_meta. Issue #266
• Fixed integration test UC volume path construction.
• Removed orphaned enhanced-CLI subsystem; fixed flake8 lint errors.

Backward Compatibility

• pip install dlt-meta continues to work — it now pulls in databricks-labs-sdp-meta as a dependency. Issue #289
• from dlt_meta import ... and import src.* continue to work with DeprecationWarning. Both shims are scheduled for removal in v0.2.0.
• Legacy config key dlt_meta_schema is still read (with a logged warning); prefer sdp_meta_schema.

Full Changelog: v0.0.10...v0.1.0

What's Changed

Enhancements

• Added apply_changes_from_snapshot support in silver layer PR
• Added UI using databricks lakehouse app for onboarding/deploy commands PR
• Added support for non-Delta as sinks(delta, kafka) PR
• Added quarantine support in silver layer for data quality rules PR
• Added support for table comments, column comments, and cluster_by PR
• Added catalog support for sourceDetails and targetDetails PR
• Added DBDemos for dlt-meta PR
• Added YAML support for onboarding PR

Bug fixes

• Fixed issue cluster by not working with bronze append only table PR
• Fixed issue view name containing period when using DPM PR
• Fixed issue CLI onboarding overwrite option always set to True PR
• Fixed issue Silver DLT not creating based on passed database PR
• Fixed issue PyPI download stats display PR
• Fixed issue Silver Data Quality not working PR
• Fixed issue Removed DPM flag check inside dataflowpipeline PR
• Fixed issue Updated dlt-meta demos into Delta Live Tables Notebook github PR
• Fixed issue Adding multiple col support for auto_cdc api PR
• Fixed issue Added support for custom transformations for Kafka/Delta PR

New Contributors

• @JunHongPP made their first contribution in #162
• @junhonglau made their first contribution in #174
• @gchandra10 made their first contribution in #203

Full Changelog: v0.0.9...v0.0.10

What's Changed

Enhancements

• Added apply_changes_from_snapshot api support in bronze layer: PR
• Added dlt append_flow api support for silver layer: PR
• Added support for file metadata columns for autoloader: PR
• Added support for Bring your own custom transformation: Issue
• Added support to Unify PyPI releases with GitHub OIDC: PR
• Added demo for append_flow and file_metadata options: PR
• Added Demo for silver fanout architecture: PR
• Added hugo-theme-relearn themee: PR
• Added unit tests to showcase silver layer fanout examples: PR
• Added liquid cluster support: PR

Bug fixes

• Fixed issue for No such file or directory: '/demo' :PR
• Fixed issue DLT-META CLI onboard command issue for Azure: databricks.sdk.errors.platform.ResourceAlreadyExists :PR
• Fixed issue Changed dbfs.create to mkdirs for CLI: PR
• Fixed issue DLT-META CLI should use pypi lib instead of whl : PR
• Fixed issue Onboarding with multiple partition columns errors out: PR

New Contributors

• @mithilmehta121287 made their first contribution in #89
• @dvanderwood made their first contribution in #101
• @sravanivadali made their first contribution in #116
• @harsha-pasala made their first contribution in #129
• @dattawalake made their first contribution in #127

Full Changelog: v0.0.8...v0.0.9

Enhancements

• Added dlt append_flow api support: #58
• Added dlt append_flow api support for silver layer: #63
• Added support for file metadata columns for autoloader: #56
• Added support for Bring your own custom transformation: #68
• Added support to Unify PyPI releases with GitHub OIDC: #62
• Added demo for append_flow and file_metadata options: #74
• Added Demo for silver fanout architecture: #83
• Added documentation in docs site for new features: #64
• Added unit tests to showcase silver layer fanout examples: #67

Bug fixes

• Fixed issue for No such file or directory: '/demo' :#59
• Fixed issue DLT-META CLI onboard command issue for Azure: databricks.sdk.errors.platform.ResourceAlreadyExists :#51
• Fixed issue Changed dbfs.create to mkdirs for CLI: #53
• Fixed issue DLT-META CLI should use pypi lib instead of whl : #79

Full Changelog: v0.0.7...v0.0.8

Enhancement

1. Mismatched Keys: Update read_dlt_delta() with key "source_database" instead of "database" #33

2. Create dlt-meta cli documentation #45

• Readme and docs to include above features

Enhancement

1. Migrate to create_streaming_table api from create_streaming_live_table #37

Updates

• Readme and docs to include above features
• Added Data Quality support for silver layer
• Updated existing demos to incorporate unity catalog and integration test framework
• integration tests framework which can be used to launch demos

New Features

1. Unity Catalog Support (#28)

2. Databricks Labs CLI Support (#28)

• Added two commands for DLT-META
• onboard: Captures all onboarding details from command line and launch onboarding job to your databricks workspace
• deploy: Captures all DLT pipeline details from command line and launch DLT pipeline to your databricks workspace

Updates

• Readme and docs to include above features
• Updated existing demos to incorporate unity catalog and integration test framework
• integration tests framework which can be used to launch demos

Bug Fixes

• Introduced new source detail option for eventhub: eventhub.accessKeySecretName

Bug Fixes

• Infer datatypes from sequence_by to __START_AT, __END_AT for apply changes API

• Removed Git release tag from github actions

v0.0.2

New Features

• Table properties support for bronze, quarantine and silver tables using create_streaming_live_table api call
• Support for track history column using apply_changes api
• Support for delta as source
• Validation for bronze/silver onboarding

Bug Fixes

• Input schema parsing issue in onboarding

Updates

• Readme and docs to include above features