Skip to content

[DRAFT][NMV] Keep merge mode test branch #13105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 71 commits into
base: master
Choose a base branch
from
Draft

[DRAFT][NMV] Keep merge mode test branch #13105

wants to merge 71 commits into from

Conversation

linliu-code
Copy link
Contributor

Change Logs

This branch is used for testing merge mode.

Impact

Describe any public API or user-facing feature change or any performance impact.

Risk level (write none, low medium or high below)

If medium or high, explain what verification was done to mitigate the risks.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

  • The config description must be updated if new configs are added or the default value of the configs are changed
  • Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
    ticket number here and follow the instruction to make
    changes to the website.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

yihua and others added 30 commits December 1, 2024 16:27
@yihua
WIP
904e41b
@yihua
WIP
3740d4e
@yihua
Fix reader context passdown
f3e8a8b
@yihua
Fix storage config passing
80bb2f6
@yihua
Fix record key fetching in HoodieSparkMergeHandleV2
fbf0201
@yihua
Fix tests
ce1ed83
@yihua
Fix checkstyle
b9b6deb
@yihua
Remove unused code in SparkTaskContextSupplier
0fd89f5
@yihua
Fix record reading from the base file in the file group record buffer
5d400c5
@yihua
Support log file only compaction
e8ebf71
@yihua
Refactor code in HoodieCompactor
e83378d
@yihua
Disable file group reader based compaction for the plan with bootstra…
cc7b458 
…p files
@yihua
Remove unnecessary recordKeyColumn from Spark reader context
894ed1f
@yihua
Rename CompactorBroadcastManager to EngineBroadcastManager
0d2b672
@yihua
Fix NPE on prevCommit in write status
6d599d8
@yihua
Disable file group reader based compaction for schema on read
3440d4e
@yihua
Fix testMergeOnReadStorageDefaultCompaction
0fee134
@yihua
Refactor HoodieSparkMergeHandleV2 to reuse code in HoodieMergeHandle
d194211
@yihua
Disable test for investigation
735dd2f
@yihua
Remove unused code in TaskContextSupplier
33a60c7
@yihua
Disable failed tests for investigation
c97d6e6
@yihua
Fix baseFileToMerge check in HoodieMergeHandle
e58f34c
@yihua
Rename API
cfe1118
@danny0405 @yihua
Some cosmetic changes
f648162
@danny0405
Rename to HoodieReadStats, more cosmetic changes
9b50d61
@danny0405
Release resource for merge handle v2
2f83b01
@danny0405
tweak the sequence for fg reader enabling check
61fd3cd
@yihua
Move test classes from hudi-spark-client to hudi-spark and disable te…
3f0102f 
…sts for investigation
@yihua
Avoid using fg reader based compaction if there is enum type
33b77a6
@yihua
Including decimal logical type in unsupported types in file group rea…
f872802 
…der-based compaction
yihua and others added 28 commits December 4, 2024 02:37
@yihua
Remove unnecessary changes
d9fad85
@yihua
Remove more unnecessary changes
c588e8d
@yihua
Address review comments
e152e0b
@yihua
Remove unused code
cf33546
@yihua
Add docs and disable fg reader compaction for virtual keys
9eaaccc
@yihua
Add more docs
4878c6c
@yihua
Add more docs for clarity
cd6f267
@yihua
Add more docs based on reviews
087f7fa
@yihua
Fix getValue in Hive reader context to account for non-existent field
85853ca
@yihua
Fix testMetadataColumnStatsIndexDeletionWithSQL by using comt time or…
bf91a62 
…dering
@yihua
Use writeSchemaWithMetaFields in the file group reader used by the co…
121f763 
…mpaction
@yihua
Fix HoodieTestReaderContext
2685868
@nsivabalan
Adding new FG reader support for reading inputs for clustering
2f72368
@danny0405 @nsivabalan
Cosmetic changes for MultipleSparkJobExecutionStrategy
572683f
@nsivabalan
Fixing spark parquet reader SQLConf parsing issue
1f9b973
@nsivabalan
fixing spillable map to contain schema Id instead of schema
441c521
@danny0405 @nsivabalan
Add schema encode/decode for record metadata schema
63d9059
@nsivabalan
renaming INTERNAL_META_SCHEMA to INTERNAL_META_SCHEMA_ID
2ae1ebe
@nsivabalan
Fixing compilation issues after rebase
43bd12f
@nsivabalan
Fixing schema handling with delete records
d497c4f
@yihua @linliu-code
Add integration test framework
9952a2c
@nsivabalan @linliu-code
Adding Data quality validation and test data generation to assist wit…
6aec849 
…h testing merge mode
@nsivabalan @linliu-code
More changes and fixes
cf0944a
@linliu-code
Fix code styling
22f14e8
@linliu-code
check in data generation
c4be407
@linliu-code
rebased
c5a48b4
@linliu-code
Use original data generation number
12ed797
@linliu-code
Merge branch 'master' into KEEP_merge_mode_test_branch
aee245f
@github-actions github-actions bot added the size:XL PR with lines of changes > 1000 label Apr 8, 2025
@hudi-bot
Copy link

hudi-bot commented Apr 8, 2025

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
size:XL PR with lines of changes > 1000
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@linliu-code @hudi-bot @yihua @danny0405 @nsivabalan