Multiple "lanes" on a pipeline? (one per type of data: logs, metrics, traces)

[linkdd]

The OpenTelemetry Specification identifies 3 types of data:

• logs: textual data
• metrics: numeric data
• traces: calendar data (spans have a start and an end)

All of those with optional attributes (key/value pairs).

At the moment, FlowG only processes logs and represents them as a mapping (set of key/value pairs, all textual).

It might be interesting in the distant future to also be able to support processing metrics and traces. But what would be the proper way?

One idea is to have "lanes" on the pipeline. Instead of connecting nodes via only one slot, we would have multiple slots, one for each data type:

In the diagram above, we would have:

• the "DIRECT" source node would output 3 lanes, one for logs, one for metrics, one for traces
• the "SYSLOG" source node would only output 1 lane, for logs
• the stream nodes would only accept a "logs" lane as input
• the forwarder nodes would accept 3 lanes as input
• there would be 3 kind of transformer nodes, one for each lanes
• ...

NB: This is just an example and is in no way what we plan to implement. In fact, there is no roadmap for this yet, hence this discussion.

This opens the door for many other kinds of nodes:

• a node that accepts a log lane as input, and outputs a metric lane, in order to produce metrics from logs
• dedicated forwarder nodes for every type of lanes (for example, the syslog forwarder would only accept logs)
• ...

This would require a huge refactoring, and this opens many questions:

• could a stream store metrics/traces? if yes, should we implement charts in the stream viewer to view said metrics?
• should we focus on the 3 data types OpenTelemetry specifies or open the door for many more data types, and possibly infinitely many lanes?
• is this a good design?

I opened this discussion to gather your thoughts, use cases, and ideas. Feel free to comment 🙂