[HUDI-8902] Fix the incorrect data read after changing the column type from float to double #13188
Conversation
…float to double
Fix the incorrect data read after changing the column type from float to double
There are two reasons for the above problems：
1. `InternalSchemaCache` cannot obtain the path of the commit file correctly.
Because after version 1.x, we placed the commit file separately in the `timeline` directory.
However, `InternalSchemaCache` still retrievals files from `basePath/.hoodie`, thus failing to obtain the correct schema.
2. When we read parquet file without `enableVectorizedReader`, we use spark 's `CAST` for type conversion when we need to do scheme evolution,
but spark has a loss of precision when dealing with `CAST` that converts `float` to `double`.
Change logs:
1. pass the correct timeline path for `InternalSchemaCache`
2. for schema evolution without `enableVectorizedReader`, for the case `CAST(FLOAT as DOUBLE)` we turn it to `CAST(CAST(FLOAT as STRING) as DOUBLE)`.
Signed-off-by: TheR1sing3un <[email protected]>
|
@hudi-bot run azure |
Are you saying the schema evolution is broken, why the schema evolution related tests can pass? |
| Cast(attr, dstType, if (needTimeZone) timeZoneId else None) | ||
|
|
||
| // work around for the case when cast float to double | ||
| if (srcType == FloatType && dstType == DoubleType) { |
There was a problem hiding this comment.
ideally double cast should be avoided but given that spark's type conversion loses precision and it is not as if we are going to do this for every record every time, I am ok with this change.
I execute the ut : And then set a breakpoint at 
stacktrace: 
|
github-project-automation
bot
moved this from 🆕 New
to 🛬 Near landing
in Hudi PR Support
Because when the schema cannot be obtained from the commit file, it will be obtained from the schema history. Our tests can all go through this process, so there are no errors in the tests. |
github-project-automation
bot
moved this from 🛬 Near landing
to ✅ Done
in Hudi PR Support
…e from float to double (apache#13188) Fix the incorrect data read after changing the column type from float to double There are two reasons for the above problems： 1. `InternalSchemaCache` cannot obtain the path of the commit file correctly. Because after version 1.x, we placed the commit file separately in the `timeline` directory. However, `InternalSchemaCache` still retrievals files from `basePath/.hoodie`, thus failing to obtain the correct schema. 2. When we read parquet file without `enableVectorizedReader`, we use spark 's `CAST` for type conversion when we need to do scheme evolution, but spark has a loss of precision when dealing with `CAST` that converts `float` to `double`. Change logs: 1. pass the correct timeline path for `InternalSchemaCache` 2. for schema evolution without `enableVectorizedReader`, for the case `CAST(FLOAT as DOUBLE)` we turn it to `CAST(CAST(FLOAT as STRING) as DOUBLE)`. Signed-off-by: TheR1sing3un <[email protected]> (cherry picked from commit e43841f)
Fix the incorrect data read after changing the column type from float to double
There are two reasons for the above problems：
InternalSchemaCachecannot obtain the path of the commit file correctly. Because after version 1.x, we placed the commit file separately in thetimelinedirectory. However,InternalSchemaCachestill retrievals files frombasePath/.hoodie, thus failing to obtain the correct schema.enableVectorizedReader, we use spark 'sCASTfor type conversion when we need to do scheme evolution, but spark has a loss of precision when dealing withCASTthat convertsfloattodouble.Change logs:
InternalSchemaCacheenableVectorizedReader, for the caseCAST(FLOAT as DOUBLE)we turn it toCAST(CAST(FLOAT as STRING) as DOUBLE).Change Logs
InternalSchemaCacheenableVectorizedReader, for the caseCAST(FLOAT as DOUBLE)we turn it toCAST(CAST(FLOAT as STRING) as DOUBLE).Describe context and summary for this change. Highlight if any code was copied.
Impact
fix the incorrectness after change data type from float to double
Risk level (write none, low medium or high below)
low
Documentation Update
none
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist