[HUDI-9258] Disable partial update when global index is used #13086
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
linliu-code
force-pushed
the
disable_partial_update
branch
from
8372a11 to
eced87d
Compare
github-actions
bot
added
size:M
linliu-code
force-pushed
the
disable_partial_update
branch
from
eced87d to
449340f
Compare
codope
reviewed
Apr 4, 2025
There was a problem hiding this comment.
Left one comment. Let's also file a followup doc ticket to update the limitation here. - https://hudi.apache.org/docs/sql_dml/#merge-into-partial-update. Could you please fix checkstyle and CI failures?
| // Check if goal index is enabled for specific indexes. | ||
| def isGlobalIndexEnabled(indexType: String, parameters: Map[String, String]): Boolean = { | ||
| Seq( | ||
| HoodieIndex.IndexType.GLOBAL_SIMPLE -> HoodieIndexConfig.SIMPLE_INDEX_UPDATE_PARTITION_PATH_ENABLE, |
There was a problem hiding this comment.
I get the intention that when update partition path is false, only then disable merge into partial update. But, would it be safer to take more conservative approach right now i.e. disable only when global index is enabled irrespective of partition path is true or not? Anyway, update partition path is true by default for global index right.
There was a problem hiding this comment.
yes, for Global_simple, the update partition path is enabled by default.
linliu-code
force-pushed
the
disable_partial_update
branch
from
76b7513 to
3328bc9
Compare
codope
force-pushed
the
disable_partial_update
branch
from
3328bc9 to
d8dd679
Compare
codope
approved these changes
Apr 4, 2025
4 tasks
voonhous
pushed a commit
to voonhous/hudi
that referenced
this pull request
Apr 8, 2025
…13086) * Disable global index for merge into query * Fix import styles * Add flag and test * Fix schema usage * Fix compilation * remove custom merge mode validation --------- Co-authored-by: Lokesh Jain <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> (cherry picked from commit 2e5ccb4)
voonhous
pushed a commit
to voonhous/hudi
that referenced
this pull request
Apr 9, 2025
…13086) * Disable global index for merge into query * Fix import styles * Add flag and test * Fix schema usage * Fix compilation * remove custom merge mode validation --------- Co-authored-by: Lokesh Jain <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> (cherry picked from commit 2e5ccb4)
voonhous
pushed a commit
to voonhous/hudi
that referenced
this pull request
Apr 11, 2025
…13086) * Disable global index for merge into query * Fix import styles * Add flag and test * Fix schema usage * Fix compilation * remove custom merge mode validation --------- Co-authored-by: Lokesh Jain <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> (cherry picked from commit 2e5ccb4)
voonhous
pushed a commit
to voonhous/hudi
that referenced
this pull request
Apr 15, 2025
…13086) * Disable global index for merge into query * Fix import styles * Add flag and test * Fix schema usage * Fix compilation * remove custom merge mode validation --------- Co-authored-by: Lokesh Jain <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> (cherry picked from commit 2e5ccb4)
voonhous
pushed a commit
to voonhous/hudi
that referenced
this pull request
Apr 16, 2025
…13086) * Disable global index for merge into query * Fix import styles * Add flag and test * Fix schema usage * Fix compilation * remove custom merge mode validation --------- Co-authored-by: Lokesh Jain <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> (cherry picked from commit 2e5ccb4)
vinishjail97
pushed a commit
to vinishjail97/hudi
that referenced
this pull request
Apr 28, 2025
…13086) * Disable global index for merge into query * Fix import styles * Add flag and test * Fix schema usage * Fix compilation * remove custom merge mode validation --------- Co-authored-by: Lokesh Jain <[email protected]> Co-authored-by: Sagar Sumit <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
Currently partial update does not function correctly when global index is used.
Since we are going to use FG reader to replace these adhoc merge logic soon, we disable partial update when global index is enabled.
Impact
No partial update when global index is used.
Risk level (write none, low medium or high below)
None, since this is the current behavior; we just enforce it.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist