Upgrade to kafka 3.9.0 container images

[davsclaus]

Description

Target

• I checked that the commit is targeting the correct branch (Camel 4 uses the main branch)

Tracking

• If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

• I checked that each commit in the pull request has a meaningful subject line and body.

• I have run mvn clean install -DskipTests locally from root folder and I have committed all auto-generated changes.

[davsclaus]

something weird is happening - anyone welcome to try locally if they can have 3.9.0 containers working.

[claudio4j]

I'm looking at why the 3.9.0 is failing to start.

[claudio4j]

The error when starting the kafka container, from target/camel-kafka-test.log

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: advertised.listeners cannot use the nonroutable meta-address 0.0.0.0. Use a routable IP address. 

There is an open bug to fix it KAFKA-18281 and fix is ongoing.

[apupier]

There is an open bug to fix it KAFKA-18281 and fix is ongoing.

issue marked as fixed in 3.9.1 and 4.0.0
3.9.1 not yet released. Planned for April. Code freeze on 9th April https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.1
4.0.0 released (but maybe a too big version jump?)

Kafka 3.8.1 is still used as java dependencies

camel/parent/pom.xml

Line 325 in 484c814

<kafka-version>3.8.1</kafka-version>

previously discarded PR upgrade of Kafka 3.9.0 #16209

[claudio4j]

I did local tests with the 4.0.0 container image and all tests passed. It should be ok to update container image and kafka api lib to 4.x.

[davsclaus]

Even if kafka 4 is out now, then IMHO it would be better to use 3.x container images for testing so it stays together with the camel-kafka client versions.

The big v4 upgrade will come later. CSB and CEQ need to be confortable with kafka v4 as well before we upgrade as default in camel too.

[davsclaus]

Anyone may want to help with this on a linux or an older mac system to give this a go. The M4 seems to not work with these images for me when I resumed this work a while back