Configuring Kafka Properties in Spring Boot’s application.yml

When integrating Apache Kafka with a Spring Boot application, configuring the essential Kafka properties is pivotal for seamless communication between producers and consumers. The application.yml file, a popular choice for configuration in Spring Boot, provides a clean and structured approach for setting up these properties.

Example —

kafka:
  producer:
    bootstrap-servers: localhost:9092
    key-serializer: org.apache.kafka.common.serialization.StringSerializer
    value-serializer: org.apache.kafka.common.serialization.StringSerializer

  consumer:
    bootstrap-servers: localhost:9092
    key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

For the producer, key-serializer and value-serializer are configured using StringSerializer, which converts keys and values to strings. Conversely, for the consumer, StringDeserializer is employed to deserialize keys and values from the byte format sent by the producer.

To handle more complex data structures like JSON, additional configuration is needed. For Example —

spring:
  kafka:
    consumer:
      enable-auto-commit: true
      bootstrap-servers: localhost:9092
      group-id: himanshu_consumer_group
      auto-offset-reset: earliest
      key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring:
           json:
             trusted:
               packages: '*'
        spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer


    producer:
      bootstrap-servers: localhost:9092
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

In this configuration:

For the producer:

Get bectorhimanshu’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

-> key-serializer: Uses StringSerializer to serialize keys as strings.
-> value-serializer: Employs JsonSerializer to serialize values as JSON.

For the consumer:

-> key-deserializer: Deserializes keys as strings using StringDeserializer.
-> value-deserializer: Uses ErrorHandlingDeserializer to deserialize values.

This deserializer can handle exceptions during deserialization, providing more robust error handling.

Additionally, for JSON deserialization in the consumer:

-> value-deserializer property uses JsonDeserializer to deserialize values as JSON.
-> properties are specified to configure the trusted packages for JSON deserialization.

This comprehensive configuration allows efficient communication between Kafka producers and consumers, supporting error handling during deserialization and enabling flexible handling of data in JSON format.

✌️If you guys enjoyed my this tech blog post and found it useful in your daily programming life, please consider buying me a coffee 😊

✌️Don’t forget to check out my other articles on Medium

✌️Please don’t forget to leave some claps for this story and follow me…. See you until next time guys. Happy Learning and keep innovating!!