Kafka Stream

最新推荐文章于 2025-12-10 08:45:00 发布

原创最新推荐文章于 2025-12-10 08:45:00 发布 · 285 阅读

本内容遵循CC 4.0 BY-SA版权协议

本文详细介绍了KafkaStream的使用，包括启动Kafka服务、创建消息队列及消费者，然后展示了如何利用KafkaStream进行数据处理，如单词计数、流处理逻辑，并输出结果到新的主题。通过KafkaStream，可以实现高效的数据流处理和分析。

Kafka Stream

一、kafka Stream是什么
二、代码

一、kafka Stream是什么

https://abelyang.blog.csdn.net/article/details/53648757

二、代码

1.开启kafka相关应用

//后台启动kafka服务
kafkserver-start.sh  -daemon /opt/software/kafka211/config/server.properties

//查看kafka已有消息队列
kafka-topics.sh --zookeeper 192.168.186.100:2181 --list

//生成kafka消息生产队列
kafka-console-producer.sh --topic wordCount1 --broker-list 192.168.186.100:9092

//Consumer wordCount1 消费队列
kafka-console-producer.sh --topic wordCount1 --broker-list 192.168.186.100:9092

//Consumer wordCount2 消费队列
kafka-console-producer.sh --topic wordCount2 --broker-list 192.168.186.100:9092

2.kafka stream 处理逻辑

public class WordCount {
    public static void main(String[] args) {
        Properties prop = new Properties();
        prop.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount");
        prop.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.186.100:9092");
        prop.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 3000);
        prop.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        //earliest latest none
        prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        prop.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        prop.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.IntegerSerde.class);

        StreamsBuilder builder = new StreamsBuilder();

        KTable<String, Long> count = builder.stream("wordCount1")
                .flatMapValues((value) -> {
                    String[] split = value.toString().split("\\s+");
                    List<String> strings = Arrays.asList(split);
                    return strings;
                })       //[null hello, null world]
                .map((k, v) -> {
                    KeyValue<String, Integer> keyValue = new KeyValue<>(v, 1);
                    return keyValue;     // hell0:1 , hello:1, world:1
                })
                .groupByKey()   //hello:(1,1), world:(1)
                .count();//hello:2, world:1

        count.toStream().foreach((key, value) -> {
            System.out.println("key:" + key + ",  value:" + value);
        });

        count.toStream().map((key, value) -> {
            return new KeyValue<>(key, key +":"+value);
        }).to("wordCount2");

        Topology topo = builder.build();
        final KafkaStreams streams = new KafkaStreams(topo, prop);

        final CountDownLatch latch = new CountDownLatch(1);
        Runtime.getRuntime().addShutdownHook(new Thread("wordcount") {
            @Override
            public void run() {
                streams.close();
                latch.countDown();
            }
        });


        try {
            streams.start();
            latch.await();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        System.exit(0);
    }
}