Flink1.12 使用WatermarkStrategy生成时间戳

原创已于 2022-11-01 03:50:13 修改 · 5.6k 阅读

9 ·

CC 4.0 BY-SA版权

文章标签：

#apache #java #zookeeper

于 2021-03-05 11:21:46 首次发布

大数据同时被 2 个专栏收录

28 篇文章

订阅专栏

flink

10 篇文章

订阅专栏

本文介绍如何在Flink 1.12版本中使用WatermarkStrategy来生成watermark，包括具体实现代码示例。通过该方法可以更好地处理乱序数据，并确保窗口操作的准确性。

Flink1.12版本后建议使用assignTimestampsAndWatermarks(WatermarkStrategy)的方式生成watermark，之前使用的assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks)以及assignTimestampsAndWatermarks(timestampAndWatermarkAssigner)均不再被推荐使用，这里是根据官网给出的小例子，仅供参考：

package demo;

import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.util.Collector;
import java.time.Duration;


public class WatermarkWCDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> inputStream = env.socketTextStream("localhost", 7777);
        SerializableTimestampAssigner<String> timestampAssigner = new SerializableTimestampAssigner<String>() {
            @Override
            public long extractTimestamp(String element, long recordTimestamp) {
                String[] fields = element.split(",");
                Long aLong = new Long(fields[0]);
                return aLong * 1000L;
            }
        };
        SingleOutputStreamOperator<String> watermarkStream = inputStream.assignTimestampsAndWatermarks(WatermarkStrategy.<String>forBoundedOutOfOrderness(Duration.ofSeconds(10)).withTimestampAssigner(timestampAssigner));
        watermarkStream.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
            @Override
            public void flatMap(String value, Collector<Tuple2<String, Integer>> out) throws Exception {
                String[] fields = value.split(",");
                out.collect(new Tuple2<>(fields[1], 1));
            }
        }).keyBy(data -> data.f0).window(TumblingEventTimeWindows.of(Time.seconds(10))).sum(1).print();
        env.execute("run watermark wc");
    }
}

程序中，输入的nc数据形如：

1614909710,java
1614909701,python
1614909700,python
1614909702,python