ClassCastException when grouping on virtualColumn with nvl expression

[pmmag]

Affected Version

31.0.0 (have not tried with 32.0.0 yet)

Description

The following error is produced when running a groupBy query where context.vectorize=true and the grouping dimension is a virtual column of the form nvl(col1, col2) where both columns are string typed, col1 exists in at least some segments and col2 does not exist in all segments:

{
  "error": "Unknown exception",
  "errorClass": "java.lang.RuntimeException",
  "host": "...",
  "errorCode": "legacyQueryException",
  "persona": "OPERATOR",
  "category": "RUNTIME_FAILURE",
  "errorMessage": "java.util.concurrent.ExecutionException: java.lang.ClassCastException",
  "context": {
    "host": "...",
    "errorClass": "java.lang.RuntimeException",
    "legacyErrorCode": "Unknown exception"
  }
}

Logs do not seem to offer much help:

2025-03-19T11:19:51,063 ERROR [processing-5] org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunner - Exception with one of the sequences!
java.lang.ClassCastException: null

org.apache.druid.query.QueryException: java.util.concurrent.ExecutionException: java.lang.ClassCastException
	at jdk.internal.reflect.GeneratedConstructorAccessor132.newInstance(Unknown Source) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
	at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:291) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:288) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:202) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:455) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at com.fasterxml.jackson.databind.deser.std.ThrowableDeserializer.deserializeFromObject(ThrowableDeserializer.java:65) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:196) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4569) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2798) ~[jackson-databind-2.12.7.1.jar:2.12.7.1]
	at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:196) [druid-server-31.0.0.jar:31.0.0]
	at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:102) [druid-server-31.0.0.jar:31.0.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:70) [druid-processing-31.0.0.jar:31.0.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) [druid-processing-31.0.0.jar:31.0.0]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$ResultBatch.fromSequence(ParallelMergeCombiningSequence.java:932) [druid-processing-31.0.0.jar:31.0.0]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.block(ParallelMergeCombiningSequence.java:984) [druid-processing-31.0.0.jar:31.0.0]
	at java.base/java.util.concurrent.ForkJoinPool.compensatedBlock(ForkJoinPool.java:3451) [?:?]
	at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434) [?:?]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.getBatchYielder(ParallelMergeCombiningSequence.java:972) [druid-processing-31.0.0.jar:31.0.0]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.block(ParallelMergeCombiningSequence.java:1107) [druid-processing-31.0.0.jar:31.0.0]
	at java.base/java.util.concurrent.ForkJoinPool.compensatedBlock(ForkJoinPool.java:3451) [?:?]
	at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434) [?:?]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$BatchedResultsCursor.nextBatch(ParallelMergeCombiningSequence.java:1021) [druid-processing-31.0.0.jar:31.0.0]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.initialize(ParallelMergeCombiningSequence.java:1081) [druid-processing-31.0.0.jar:31.0.0]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$PrepareMergeCombineInputsAction.compute(ParallelMergeCombiningSequence.java:772) [druid-processing-31.0.0.jar:31.0.0]
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194) [?:?]
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) [?:?]
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) [?:?]
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) [?:?]
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) [?:?]
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) [?:?]

For context, in our case, we have two datasources, error-state and help-state, which contain columns errorCode and helpCode, respectively, and the query looks like this (actually it is more complex, this is just a minimal reproducing example):

{
  "queryType": "groupBy",
  "dataSource": {
    "type": "union",
    "dataSources": [ "error-state", "help-state" ]
  },
  "dimensions": [ "code" ],
  "granularity": "all",
  "virtualColumns": [
    {
      "type": "expression",
      "name": "code",
      "expression": "nvl(errorCode, helpCode)",
      "outputType": "STRING"
    }
  ],
  "intervals": [  "2025-03/2025-04" ],
  "context": {"vectorize": true}
}

If we change the expression to nvl(errorCode, nvl(helpCode, '')) or even nvl(errorCode, nvl(helpCode, null)), the query works as expected (this is our workaround for now).

The same can be reproduced with any datasource (can be a single table as well), by grouping on an nvl expression where the second argument is a reference to a column that doesn't exist in every segment.

This is not a big issue for us because of the workaround, however, we would expect such queries to either A just work, or B return a useful error message telling the user if there is something in the query that is not supported.

[gianm]

I tried to repro this and wasn't able to. It is possible it's been fixed since Druid 31.0.0. There have been changes in the area of vectorized expressions since then.

Here is what I did:

1. Populated error-state with errorCode

REPLACE INTO "error-state"
OVERWRITE ALL
SELECT * FROM (VALUES ('1'), ('2')) t (errorCode)
PARTITIONED BY ALL

2. Populated help-state with helpCode

REPLACE INTO "help-state"
OVERWRITE ALL
SELECT * FROM (VALUES ('3'), ('4')) t (helpCode)
PARTITIONED BY ALL

3. Ran this query (same as yours, but with a broader time interval, since the datasets I inserted would use the default timestamp 1970-01-01)

{
  "queryType": "groupBy",
  "dataSource": {
    "type": "union",
    "dataSources": [ "error-state", "help-state" ]
  },
  "dimensions": [ "code" ],
  "granularity": "all",
  "virtualColumns": [
    {
      "type": "expression",
      "name": "code",
      "expression": "nvl(errorCode, helpCode)",
      "outputType": "STRING"
    }
  ],
  "intervals": [  "0000/3000" ],
  "context": {"vectorize": true}
}

I got good results:

{"code":"1"}
{"code":"2"}
{"code":"3"}
{"code":"4"}

[pmmag]

I tried your example in a fresh docker compose with both 31.0.0 and 32.0.1 (with the sample compose file and environment file linked in the docs).

For me when I run the groupBy query right after the data has been inserted, I actually do get

{
  "error": "Unknown exception",
  "errorClass": "java.lang.RuntimeException",
  "host": "172.21.0.5:8083",
  "errorCode": "legacyQueryException",
  "persona": "OPERATOR",
  "category": "RUNTIME_FAILURE",
  "errorMessage": "java.util.concurrent.ExecutionException: java.lang.ClassCastException: class [J cannot be cast to class [Ljava.lang.Object; ([J and [Ljava.lang.Object; are in module java.base of loader 'bootstrap')",
  "context": {
    "host": "172.21.0.5:8083",
    "errorClass": "java.lang.RuntimeException",
    "legacyErrorCode": "Unknown exception"
  }
}

However, if I run the groupBy even once with vectorize: false, then afterwards all subsequent queries with vectorize: true will work (which is not the case with our actual Druid 31.0.0 installation, where we get the error every time)... Perhaps this is some kind of user error or I just need to wait longer after inserting the data?

I also tried to change the help and error codes to contain non-numeric characters (as they do in our case), as well as tweak the timestamps and partitioning (e.g., "by month") to see if it matters when the data is not in the same time range. But it made no difference, I still always only get the error until I make one query with vectorize: false. Yet, in our real world datasources, the errors persist.

So yes, it seems to be difficult to reproduce, and I guess there must be some other factor at play in our real world setup where it happens every time (?). The original datasources in question are using Kafka-based ingestion, in case that matters.