Skip to content

async Init file function: streams and fileAdded events #319

Open
@drzraf

Description

@drzraf

This is a follow-up of #296
In that PR, support was introduced for async initFileFn which allows to read from a stream and/or perform other async operation before Flow.js proceed with a given file.

One very important aspect of the FlowFile bootstrap is size determination (by default, from the regular file.size) in order to compute the number of chunks needed. A initFileFn could affect this computation.

FlowFile initialization is followed by the fileAdded event which serves two purposes:

  • As a hook, which allow consumer to reject this file
  • As a notification, which inform consumer that bootstrap is fully done and further processing [eg: flow.upload()] can be called (Flow.upload() would fails if the FlowFile.chunks array was not computed)

One issue is that an async initFileFn could finish after fileAdded fires.
So one question is: Should the initFileFn function really run if a file is going to be rejected?

If yes (eg, because rejection depends upon the number of chunks/file.size), then a PR could be made so that the event is fired after initFileFn (even if async). It would move fileAdded inside FlowFile's bootstrap function unless a nicer and higher-level solution exists (like of more pipeline-like vision of the process). I tend to think there should be less assumptions about file.size at the beginning of the process (eg: FlowChunks could be built on-demand later) but this hook in its current shape forbids this.

Side note about Events: I think that having events to expect a return value makes them more hooks than events. It's a bit counter intuitive and keeps the code from moving to native Events. Not sure they are worth it.
The fileAdded native event could just pass the corresponding (read-only) reference to Flow and the FlowFile and the consumer could delete it as well (accessing to the global flow object). It'd be more flexible.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions