Spark Yarn Client模式spark.driver.host导致 Cannot assign requested address: Service ‘sparkDriver‘ failed

最新推荐文章于 2024-05-05 05:25:47 发布

原创最新推荐文章于 2024-05-05 05:25:47 发布 · 2.8k 阅读

收录于

大数据-spark 专栏收录该内容

204 篇文章 ¥19.90 ¥99.00

订阅专栏

本文探讨了在Spark Yarn Client模式运行时遇到的'Cannot assign requested address: Service 'sparkDriver' failed'错误。通过调整配置参数，参照相关博客文章，找到了问题的解决方法。

1.背景

在这里插入图片描述

[deploy@fideploy cheetah-node]$  /opt/cloudera/parcels/SPARK2/bin/spark2-submit  --keytab /etc/security/keytabs/cluster_user.keytab --principal cluster_user@HADOOP.COM --class com.dtwave.cheetah.node.spark.structured.streaming.StructureStreamingExecutor --name test --master yarn --deploy-mode client --queue root.default /opt/workspace/cheetah-node/libs/spark-structured-streaming.jar  deploy-mode=client  task-protocol=/data/cheetah-node/codes/T_6167372758075432_20190829211641_2_1/stream_info   yarn-address=xxx:8032


WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera4-1.cdh5.13.3.p0.603055/lib/spark2) overrides d

了解本专栏