[SPARK-54821][SQL] Deprecate support for Derby JDBC datasource #53581
Conversation
|
BTW, Spark ships Derby jars to support embedded HMS, should we switch to an alternative? Seems the only option is H2. PS: there is no action in the Hive community as of now. |
docs/sql-data-sources-jdbc.md
Outdated
| * The included JDBC driver version supports kerberos authentication with keytab. | ||
| * There is a built-in connection provider which supports the used database. | ||
|
|
||
| There is a built-in connection providers for the following databases: |
There was a problem hiding this comment.
Please spin-off this as a new document PR, @pan3793 .
There was a problem hiding this comment.
@dongjoon-hyun, I opened #53598 to update the list, will rebase after merging that.
| import org.apache.spark.sql.types._ | ||
|
|
||
|
|
||
| @deprecated(since = "4.2.0") |
There was a problem hiding this comment.
Although I understand the intention, it looks a little weird because this is a private case class. Maybe, we need other way to warn this.
There was a problem hiding this comment.
+1. I think users can only use JDBC dialects via the JDBC data source, so we should document this deprecation in the JDBC data source doc page.
There was a problem hiding this comment.
in addition to mentioning deprecation in docs, how about adding a warning log on loading this class?
There was a problem hiding this comment.
If we don't plan to remove it why emit the warning log?
There was a problem hiding this comment.
agree with the opinions of @dongjoon-hyun and @cloud-fan.
There was a problem hiding this comment.
so I will just remove this change and keep code unchanged.
docs/sql-migration-guide.md
Outdated
|
|
||
| ## Upgrading from Spark SQL 4.1 to 4.2 | ||
|
|
||
| - Since Spark 4.2, support for Derby JDBC datasource is deprecated and will be removed in a future version. |
There was a problem hiding this comment.
Shall we remove this, and will be removed in a future version, because we cannot remove this until Apache Spark 5.0. According to the new rapid release plan, we will have 4.2 in 2026 with the existing release cadence. So, Apache Spark 5 could be 2027.
4 participants
What changes were proposed in this pull request?
Mark
DerbyDialectas deprecated, also update docs.Why are the changes needed?
https://db.apache.org/derby/
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Review.
Was this patch authored or co-authored using generative AI tooling?
No.