[MINOR] Fix flaky test testCreateNewInstantTimes

[yihua]

Change Logs

As above.

Impact

Fixes test flakiness.

Risk level

none

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[nsivabalan]

10 sec is not sufficient is it? :(

[yihua]

10 sec is not sufficient is it? :(

Seems like it. I've added logs also in case it fails again so we know the exact instant times from the test.

[codope]

Looks like the test is passing. Curious about why generating time is taking long.

[codope]

Is InProcess lock taking so long?

[geserdugarov]

I've added corresponding comment below #13032 (comment).

[yihua]

I rebased the branch on top of the latest master with CI improvements.

[hudi-bot]

CI report:

• dcaba03 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[yihua]

Flink IT failure is irrelevant.

[geserdugarov]

@yihua , @codope , @nsivabalan , @danny0405 , I've also spent some time on this flaky test before I found this PR. I propose to clarify the situation with this UT by merge of #13042

I've added clarification in the comment to this test in mentioned PR:

// Note, that `metaClient` is initialized here with default `HoodieTimeGeneratorConfig#timeGeneratorConfig`,
// which means that `HoodieTimeGeneratorConfig#MAX_EXPECTED_CLOCK_SKEW_MS` with default value will be used (probably, it's 200 ms).
// It leads to corresponding thread sleep in `SkewAdjustingTimeGenerator::generateTime` for each call
// after all preliminary initialization costs in the following loop, so we don't want to iterate a lot here.

and reduced number of iterations and max permitted diff.

The failure could be easily reproduced locally by changing to 10 s max and i < 100. After I've found the reason, it was clear that it would be enough to pass HoodieTimeGeneratorConfig#MAX_EXPECTED_CLOCK_SKEW_MS or explicit set of in-process lock provider to HoodieTableMetaClient::build to support large number of iterations. But I couldn't configure properly this case from TestHoodieTableMetaClient, because of current test classes inheritance. And also we don't need to retry so much in a unit test.

I've checked locally, that if I change default value of HoodieTimeGeneratorConfig#MAX_EXPECTED_CLOCK_SKEW_MS to 1 ms, then I could set i < 100 without any failures.

[geserdugarov]

Interesting, that correct direction for research was revealed by profiling:

Not all runs were caught in this jfr, but it's obvious that we just sleep periodically.