[HUDI-9219] Add callbacks for CloseableIterators and eager close data readers

[the-other-tim-brown]

Change Logs

• Adds a helper for adding a task completion listener to close iterators if applicable
• Ensure HoodieData abstractions are calling close when required
• Add listener in spark code that does not use the HoodieData abstraction
• Eager close the file readers when there is no more data to read

Impact

• Ensures the readers are being closed as soon as they are no longer needed. This will help with memory usage and avoid memory leaks
• Making sure close is called at the HoodieData level also helps ensure that the resources are closed, especially in the face of an error

Risk level (write none, low medium or high below)

Low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• 981d33e UNKNOWN
• 0a0055e UNKNOWN
• f63a531 UNKNOWN
• c847a78 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[nsivabalan]

LGTM. just 1 minor suggestion.
good job chasing down all the resource closing gaps.

[nsivabalan]

hwo about CloseValidationIterator

[the-other-tim-brown]

updated

[hudi-bot]

CI report:

• 981d33e UNKNOWN
• 0a0055e UNKNOWN
• f63a531 UNKNOWN
• d0c019c Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

we have these eagerly close, do we still need to invoke the Iterator.close outside? Looks the logic is redundant.

[the-other-tim-brown]

This helps proactively free up resources if all elements are read, especially in places that try with resources was missed or if the close is in a later callback. If the iterator is closed without going through all of the elements, it will hit the close method to ensure that the input stream is still closed

[danny0405]

yeah, understand, but eager close is not often used in code, I suggest we remove it and unify the code to AutoClosable and try-resource for resource releasing.

[the-other-tim-brown]

I don't see any issue in doing both of these things. What is the main concern for this added level of safety?

[danny0405]

The iterator is designed to be no side effect actually, we breaks it already for introducing ClosableIterator, and not we breaks the rule silently during the iteration which I think is not a very good convention to follow.