Flink多流共用一个方法,状态编程报错

在Flink作业执行时遇到IllegalArgumentException,报错信息指出存在Hash碰撞,原因是用户指定的IDasdfasdfasfdfLiveCartInfoMergeProcessUid不唯一。解决方法包括确保上游不同流调用不同的方法以避免uid冲突,或者在被多个流调用的方法中避免使用状态编程,从而消除uid在同一拓扑中出现多次的情况。

报错信息

xception in thread "main" java.lang.IllegalArgumentException: Hash collision on user-specified ID "asdfasdfasfdfLiveCartInfoMergeProcessUid". Most likely cause is a non-unique ID. Please check that all IDs specified via `uid(String)` are unique.
	at org.apache.flink.streaming.api.graph.StreamGraphHasherV2.generateNodeHash(StreamGraphHasherV2.java:185)
	at org.apache.flink.streaming.api.graph.StreamGraphHasherV2.traverseStreamGraphAndGenerateHashes(StreamGraphHasherV2.java:110)
	at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:171)
	at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:117)
	at org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:934)
	at org.apache.flink.client.StreamGraphTranslator.translateToJobGraph(StreamGraphTranslator.java:50)
	at org.apache.flink.client.FlinkPipelineTranslationUtil.getJobGraph(FlinkPipelineTranslationUtil.java:39)
	at org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:56)
	at org.apache.flink.client.deployment.executors.LocalExecutor.getJobGraph(LocalExecutor.java:104)
	at org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:82)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1957)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1848)
	at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:69)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1834)
	at com.ywwl.ywdc.live.main.LiveFansCart.main(LiveFansCart.java:90)
Disconnected from the target VM, address: 'javadebug', transport: 'shared memory'

原因分析

上游消费不同的kafka数据,下游同时调用一个方法;在拓扑关系中,就是两个流程;但是在方法中,使用了状态编程,用到了uid;在拓扑关系中,就是在两个流中存在同样的uid,导致uid不唯一,报出上述错误

解决

①上游不同的流,调用不同的方法,排除uid相同的问题;

②被上游不同流调用的方法中,不使用状态编程;

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值