Skip to content

Warnings when using MicrometerMetrics with PrometheusMeterRegistry #3038

@marcinpeck

Description

@marcinpeck

Bug Report

What did you do?

Using operator with metrics. Implementation of Metrics (io.javaoperatorsdk.operator.monitoring.micrometer.MicrometerMetrics) is using
io.micrometer.prometheusmetrics.PrometheusMeterRegistry.

This produces a lot of warnings in logs:

Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'operator_sdk_events_received' containing tag keys [action, event, group, kind, name, namespace, scope, version]. The meter you are attempting to register has keys [event, group, kind, name, namespace, scope, version].

It happens in the following method:

Mentioned implementation allows registration of the same metric but with different set of tags. This is not allowed by PrometheusMeterRegistry - see as prometheus treats this as separate time series see

What did you expect to see?

I expect to see all metrics without plenty of warnings in logs (Splunk).

Possible Solution

As operator uses 2 types of events - resource related event with action and non-resource related - I would expect to have the same set of tags, but in case of non-resource related maybe something general.

  public void receivedEvent(Event event, Map<String, Object> metadata) {
    if (event instanceof ResourceEvent) {
      incrementCounter(
          event.getRelatedCustomResourceID(),
          EVENTS_RECEIVED,
          metadata,
          Tag.of(EVENT, event.getClass().getSimpleName()),
          Tag.of(ACTION, ((ResourceEvent) event).getAction().toString()));
    } else {
      incrementCounter(
          event.getRelatedCustomResourceID(),
          EVENTS_RECEIVED,
          metadata,
          Tag.of(EVENT, event.getClass().getSimpleName()));
          Tag.of(ACTION, "NA");
    }
  }

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions