Skip to content

kubernetes druid job status reporting issue #17848

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
vadimprt opened this issue Mar 31, 2025 · 4 comments
Open

kubernetes druid job status reporting issue #17848

vadimprt opened this issue Mar 31, 2025 · 4 comments
Labels
Uncategorized problem report

Comments

@vadimprt
Copy link

vadimprt commented Mar 31, 2025 •
edited
Loading

hi, i have druid version 32.0.1 deployed on kubernetes version 1.29.6 used with druid-operator version 1.3.0

task are running but on successful finish reported as failed

Image

2025-03-31T09:00:06,127 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_kafka_account_statistics_5843c7683a014fd_lgbmackd",
  "status" : "SUCCESS",
  "duration" : 1788220,
  "errorMsg" : null,
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}
2025-03-31T09:00:06,131 INFO [main] org.apache.druid.cli.CliPeon - Thread [Thread[HTTP-Dispatcher,5,main]] is non daemon.
2025-03-31T09:00:06,132 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2025-03-31T09:00:06,137 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2025-03-31T09:00:06,141 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@65f3f9e2{HTTP/1.1, (http/1.1)}{0.0.0.0:8100}
2025-03-31T09:00:06,142 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging
2025-03-31T09:00:06,143 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@468eff41{/,null,STOPPED}
2025-03-31T09:00:06,191 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_kafka_account_statistics_5843c7683a014fd_lgbmackd].
2025-03-31T09:00:06,192 INFO [main] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Stopping forcefully (status: [PUBLISHING])
2025-03-31T09:00:06,192 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2025-03-31T09:00:06,192 INFO [main] org.apache.druid.server.coordination.SegmentBootstrapper - Stopping...
2025-03-31T09:00:06,192 INFO [main] org.apache.druid.server.coordination.SegmentBootstrapper - Stopped.
2025-03-31T09:00:06,193 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2025-03-31T09:00:06,194 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopping.
2025-03-31T09:00:06,194 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopped.
2025-03-31T09:00:06,194 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopping.
2025-03-31T09:00:06,194 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopped.
2025-03-31T09:00:06,194 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider - stopping
2025-03-31T09:00:06,194 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Stopping NodeRoleWatcher for role[OVERLORD]...
2025-03-31T09:00:06,195 ERROR [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcheroverlord] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Error while watching role[OVERLORD]
java.lang.RuntimeException: IO Exception during hasNext method.
	at io.kubernetes.client.util.Watch.hasNext(Watch.java:183) ~[?:?]
	at org.apache.druid.k8s.discovery.DefaultK8sApiClient$2.hasNext(DefaultK8sApiClient.java:132) ~[?:?]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.keepWatching(K8sDruidNodeDiscoveryProvider.java:266) ~[?:?]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.watch(K8sDruidNodeDiscoveryProvider.java:236) ~[?:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.io.InterruptedIOException
	at okhttp3.internal.http2.Http2Stream.waitForIo$okhttp(Http2Stream.kt:660) ~[?:?]
	at okhttp3.internal.http2.Http2Stream$FramingSource.read(Http2Stream.kt:376) ~[?:?]
	at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:281) ~[?:?]
	at okio.RealBufferedSource.exhausted(RealBufferedSource.kt:200) ~[?:?]
	at io.kubernetes.client.util.Watch.hasNext(Watch.java:181) ~[?:?]
	... 8 more
2025-03-31T09:00:06,200 ERROR [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcheroverlord] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Exception while watching for role[OVERLORD].
java.lang.RuntimeException: java.lang.InterruptedException
	at org.apache.druid.concurrent.LifecycleLock$Sync.awaitStarted(LifecycleLock.java:144) ~[druid-processing-32.0.1.jar:32.0.1]
	at org.apache.druid.concurrent.LifecycleLock.awaitStarted(LifecycleLock.java:245) ~[druid-processing-32.0.1.jar:32.0.1]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.keepWatching(K8sDruidNodeDiscoveryProvider.java:255) ~[?:?]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.watch(K8sDruidNodeDiscoveryProvider.java:236) ~[?:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081) ~[?:?]
	at org.apache.druid.concurrent.LifecycleLock$Sync.awaitStarted(LifecycleLock.java:139) ~[druid-processing-32.0.1.jar:32.0.1]
	... 8 more
2025-03-31T09:00:16,201 INFO [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcheroverlord] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Exited Watch for role[OVERLORD].
2025-03-31T09:00:16,201 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Stopped NodeRoleWatcher for role[OVERLORD].
2025-03-31T09:00:16,201 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Stopping NodeRoleWatcher for role[COORDINATOR]...
2025-03-31T09:00:16,202 ERROR [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatchercoordinator] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Error while watching role[COORDINATOR]
java.lang.RuntimeException: IO Exception during hasNext method.
	at io.kubernetes.client.util.Watch.hasNext(Watch.java:183) ~[?:?]
	at org.apache.druid.k8s.discovery.DefaultK8sApiClient$2.hasNext(DefaultK8sApiClient.java:132) ~[?:?]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.keepWatching(K8sDruidNodeDiscoveryProvider.java:266) ~[?:?]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.watch(K8sDruidNodeDiscoveryProvider.java:236) ~[?:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.io.InterruptedIOException
	at okhttp3.internal.http2.Http2Stream.waitForIo$okhttp(Http2Stream.kt:660) ~[?:?]
	at okhttp3.internal.http2.Http2Stream$FramingSource.read(Http2Stream.kt:376) ~[?:?]
	at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:281) ~[?:?]
	at okio.RealBufferedSource.exhausted(RealBufferedSource.kt:200) ~[?:?]
	at io.kubernetes.client.util.Watch.hasNext(Watch.java:181) ~[?:?]
	... 8 more
2025-03-31T09:00:16,203 ERROR [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatchercoordinator] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Exception while watching for role[COORDINATOR].
java.lang.RuntimeException: java.lang.InterruptedException
	at org.apache.druid.concurrent.LifecycleLock$Sync.awaitStarted(LifecycleLock.java:144) ~[druid-processing-32.0.1.jar:32.0.1]
	at org.apache.druid.concurrent.LifecycleLock.awaitStarted(LifecycleLock.java:245) ~[druid-processing-32.0.1.jar:32.0.1]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.keepWatching(K8sDruidNodeDiscoveryProvider.java:255) ~[?:?]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.watch(K8sDruidNodeDiscoveryProvider.java:236) ~[?:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081) ~[?:?]
	at org.apache.druid.concurrent.LifecycleLock$Sync.awaitStarted(LifecycleLock.java:139) ~[druid-processing-32.0.1.jar:32.0.1]
	... 8 more
2025-03-31T09:00:26,204 INFO [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatchercoordinator] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Exited Watch for role[COORDINATOR].
2025-03-31T09:00:26,204 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Stopped NodeRoleWatcher for role[COORDINATOR].
2025-03-31T09:00:26,204 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider - stopped
2025-03-31T09:00:26,215 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task
@gianm
Copy link
Contributor

gianm commented Apr 3, 2025

Did the Overlord log anything interesting about index_kafka_account_statistics_5843c7683a014fd_lgbmackd? Generally in this situation (task thinks it succeeded, but is marked as failed) the failure marking was done on the Overlord.

@vadimprt
Copy link
Author

vadimprt commented Apr 4, 2025 •
edited
Loading 

2025-04-04T06:25:13,185 WARN [KafkaSupervisor-account_statistics-Worker-0] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Ignoring task [index_kafka_account_statistics_ea980b593535b44_bjifongi], as probably it is not started running yet
2025-04-04T06:25:21,640 ERROR [TaskQueue-OnComplete-4] org.apache.druid.k8s.overlord.common.KubernetesPeonClient - Error watching logs from task: [ index_kafka_account_statistics_staging_88d2c2b11e4e98c_nhdlmkcm, indexkafkaaccountstatisticssta-abf6c2ceb0f8993c08decefac1d98a1c]
2025-04-04T06:25:21,649 INFO [TaskQueue-OnComplete-4] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_account_statistics_staging_88d2c2b11e4e98c_nhdlmkcm] from activeTasks
2025-04-04T06:25:21,659 INFO [TaskQueue-OnComplete-4] org.apache.druid.indexing.overlord.TaskLockbox - Deleted [0] entries from pendingSegments table for taskAllocatorId[index_kafka_account_statistics_staging_88d2c2b11e4e98c].
2025-04-04T06:25:21,659 INFO [TaskQueue-OnComplete-4] org.apache.druid.indexing.overlord.TaskQueue - Completed task[index_kafka_account_statistics_staging_88d2c2b11e4e98c_nhdlmkcm] with status[FAILED] in [-1]ms.
2025-04-04T06:25:31,738 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_kafka_account_statistics_ea980b593535b44_bjifongi
2025-04-04T06:25:31,739 INFO [TaskQueue-Manager] org.apache.druid.k8s.overlord.KubernetesTaskRunner - Shutdown [index_kafka_account_statistics_staging_88d2c2b11e4e98c_odblbjmg] because [Task is not in knownTaskIds]
2025-04-04T06:25:31,739 INFO [TaskQueue-Manager] org.apache.druid.k8s.overlord.common.KubernetesPeonClient - Not cleaning up job [ index_kafka_account_statistics_staging_88d2c2b11e4e98c_odblbjmg, indexkafkaaccountstatisticssta-e0ea00754260549edbbd94a4df376c8a] due to flag: debugJobs=true
2025-04-04T06:25:31,739 INFO [TaskQueue-Manager] org.apache.druid.k8s.overlord.KubernetesTaskRunner - Shutdown [index_kafka_account_statistics_staging_88d2c2b11e4e98c_agioecop] because [Task is not in knownTaskIds]
2025-04-04T06:25:31,739 INFO [TaskQueue-Manager] org.apache.druid.k8s.overlord.common.KubernetesPeonClient - Not cleaning up job [ index_kafka_account_statistics_staging_88d2c2b11e4e98c_agioecop, indexkafkaaccountstatisticssta-017e17110ec7cec571d5c3fec069f4d6] due to flag: debugJobs=true
2025-04-04T06:25:31,739 INFO [TaskQueue-Manager] org.apache.druid.k8s.overlord.KubernetesTaskRunner - Shutdown [index_kafka_account_statistics_ea980b593535b44_pnibaceb] because [Task is not in knownTaskIds]
2025-04-04T06:25:31,739 INFO [TaskQueue-Manager] org.apache.druid.k8s.overlord.common.KubernetesPeonClient - Not cleaning up job [ index_kafka_account_statistics_ea980b593535b44_pnibaceb, indexkafkaaccountstatisticsea9-c86b7814f10534825e5bf234d0d11747] due to flag: debugJobs=true
2025-04-04T06:25:31,739 INFO [TaskQueue-Manager] org.apache.druid.k8s.overlord.KubernetesTaskRunner - Shutdown [index_kafka_account_statistics_staging_88d2c2b11e4e98c_pnkpilmc] because [Task is not in knownTaskIds]
2025-04-04T06:25:31,739 INFO [TaskQueue-Manager] org.apache.druid.k8s.overlord.common.KubernetesPeonClient - Not cleaning up job [ index_kafka_account_statistics_staging_88d2c2b11e4e98c_pnkpilmc, indexkafkaaccountstatisticssta-8bba2b9e59a248419e04fc1161cfbcd6] due to flag: debugJobs=true
2025-04-04T06:37:46,343 INFO [k8s-task-runner-2] org.apache.druid.k8s.overlord.KubernetesPeonLifecycle - Peon for task [[ index_kafka_account_statistics_ea980b593535b44_ncabiepa, indexkafkaaccountstatisticsea9-32ebca4d1cfc2455a1b824abe41a3986]] did not push its task status. Check k8s logs and events for the pod to see what happened.
2025-04-04T06:37:46,344 INFO [k8s-task-runner-1] org.apache.druid.k8s.overlord.KubernetesPeonLifecycle - Peon for task [[ index_kafka_account_statistics_ea980b593535b44_hpnchnjm, indexkafkaaccountstatisticsea9-2d8ab78c1e856cb2191d0cee6c60ce60]] did not push its task status. Check k8s logs and events for the pod to see what happened.
2025-04-04T06:37:46,345 INFO [k8s-task-runner-0] org.apache.druid.k8s.overlord.KubernetesPeonLifecycle - Peon for task [[ index_kafka_account_statistics_ea980b593535b44_bemdgoen, indexkafkaaccountstatisticsea9-05f940838ebf13691c013013f1d25308]] did not push its task status. Check k8s logs and events for the pod to see what happened.
2025-04-04T06:37:46,356 INFO [k8s-task-runner-10] org.apache.druid.k8s.overlord.KubernetesPeonLifecycle - Peon for task [[ index_kafka_account_statistics_staging_88d2c2b11e4e98c_fpibenbg, indexkafkaaccountstatisticssta-90cc72d7a26d8156fb6d3a9fe87dcd1d]] did not push its task status. Check k8s logs and events for the pod to see what happened.
2025-04-04T06:37:46,661 INFO [k8s-task-runner-10] org.apache.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: /druid/data/indexing-logs/index_kafka_account_statistics_staging_88d2c2b11e4e98c_fpibenbg.index_kafka_account_statistics_staging_88d2c2b11e4e98c_fpibenbg11046542241980584732log
2025-04-04T06:37:46,661 INFO [k8s-task-runner-1] org.apache.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: /druid/data/indexing-logs/index_kafka_account_statistics_ea980b593535b44_hpnchnjm.index_kafka_account_statistics_ea980b593535b44_hpnchnjm7544857200429693543log
2025-04-04T06:37:46,661 INFO [k8s-task-runner-0] org.apache.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: /druid/data/indexing-logs/index_kafka_account_statistics_ea980b593535b44_bemdgoen.index_kafka_account_statistics_ea980b593535b44_bemdgoen3404686932929332383log
2025-04-04T06:37:46,661 INFO [k8s-task-runner-2] org.apache.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: /druid/data/indexing-logs/index_kafka_account_statistics_ea980b593535b44_ncabiepa.index_kafka_account_statistics_ea980b593535b44_ncabiepa16657618095290491263log

{
  "id": "index_kafka_account_statistics_e08e5ce85376c60_jimnpifn",
  "groupId": "index_kafka_account_statistics",
  "type": "index_kafka",
  "createdTime": "2025-04-04T11:43:38.982Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "FAILED",
  "status": "FAILED",
  "runnerStatusCode": "WAITING",
  "duration": 969000,
  "location": {
    "host": "x.x.x.x",
    "port": 8100,
    "tlsPort": -1,
    "k8sPodName": "indexkafkaaccountstatisticse08-91e0faa350ee5f06781270086dc54jmx"
  },
  "dataSource": "account_statistics",
  "errorMsg": "Peon did not report status successfully."
}

@vadimprt
Copy link
Author

vadimprt commented Apr 8, 2025 

private TaskStatus getTaskStatus(long duration)
  {
    TaskStatus taskStatus;
    try {
      Optional<InputStream> maybeTaskStatusStream = taskLogs.streamTaskStatus(taskId.getOriginalTaskId());
      if (maybeTaskStatusStream.isPresent()) {
        taskStatus = mapper.readValue(
            IOUtils.toString(maybeTaskStatusStream.get(), StandardCharsets.UTF_8),
            TaskStatus.class
        );
      } else {
        log.info(
            "Peon for task [%s] did not push its task status. Check k8s logs and events for the pod to see what happened.",
            taskId
        );
        taskStatus = TaskStatus.failure(taskId.getOriginalTaskId(), "Peon did not report status successfully.");
      }
    }
    catch (IOException e) {
      log.error(e, "Failed to load task status for task [%s]", taskId.getOriginalTaskId());
      taskStatus = TaskStatus.failure(
          taskId.getOriginalTaskId(),
          StringUtils.format("error loading status: %s", e.getMessage())
      );
    }

    return taskStatus.withDuration(duration);
  }

something around https://github.com/apache/druid/blob/5764183d4e3c3d8267e98eced05858d5524816c9/extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesPeonLifecycle.java

can you please check or instruct how to get more logs around this process?

FYI i am getting this type of behavior/issue since druid version 27.0.0 version 26.0.0 working as expected

@vadimprt
Copy link
Author

vadimprt commented Apr 20, 2025 •
edited
Loading

@gianm @adarshsanjeev can someone have a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Uncategorized problem report
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gianm @vadimprt