spark集群任务遇到的问题

原创已于 2023-10-20 15:29:11 修改 · 245 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

GEO检测

于 2023-10-20 15:15:21 首次发布

文章讲述了在Spark的YARN集群环境下，用户遇到任务提交时依赖找不到的问题，涉及到NoClassDefFoundError，解决方法是将依赖的jar包上传至HDFS并确保所有节点访问权限。

spark的yarn集群提交任务，遇到依赖找不到问题

Job aborted due to stage failure: Task 21 in stage 15.0 failed 4 times, most recent failure: Lost task 21.3 in stage 15.0 (TID 1631) (IOT-147 executor 1): org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] Failed to execute user defined function (UDFRegistration $KaTeX parse error: Can't use function '$' in math mode at position 7: Lambda$̲1220/0x00000008\dots$ anon $1. ha s N e x t (Wh o l e St a g e C o d e g e n E x ec . sc a l a : 760) a t or g . a p a c h e . s p a r k . s ql . e x ec u t i o n . d a t a so u rces . v 2. W r i t in g Sp a r k T a s k .$ anonfun$run $1 (W r i t e T oD a t a S o u rce V 2 E x ec . sc a l a : 464) a t or g . a p a c h e . s p a r k . u t i l . U t i l s$ .tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1563)
at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run(WriteToDataSourceV2Exec.scala:509)
at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run $(W r i t e T oD a t a S o u rce V 2 E x ec . sc a l a : 448) a t or g . a p a c h e . s p a r k . s ql . e x ec u t i o n . d a t a so u rces . v 2. D a t aW r i t in g Sp a r k T a s k$ .run(WriteToDataSourceV2Exec.scala:514)
at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec. $an o n f u n$ writeWithV2 $2 (W r i t e T oD a t a S o u rce V 2 E x ec . sc a l a : 411) a t or g . a p a c h e . s p a r k . sc h e d u l er . R es u ltT a s k . r u n T a s k (R es u ltT a s k . sc a l a : 92) a t or g . a p a c h e . s p a r k . T a s k C o n t e x t . r u n T a s kWi t h L i s t e n ers (T a s k C o n t e x t . sc a l a : 161) a t or g . a p a c h e . s p a r k . sc h e d u l er . T a s k . r u n (T a s k . sc a l a : 139) a t or g . a p a c h e . s p a r k . e x ec u t or . E x ec u t or$ TaskRunner. $an o n f u n$ run $3 (E x ec u t or . sc a l a : 554) a t or g . a p a c h e . s p a r k . u t i l . U t i l s$ .tryWithSafeFinally(Utils.scala:1529)
at org.apache.spark.executor.Executor $T a s k R u nn er . r u n (E x ec u t or . sc a l a : 557) a t ja v a . ba se / ja v a . u t i l . co n c u rre n t . T h re a d P oo lE x ec u t or . r u nW or k er (T h re a d P oo lE x ec u t or . ja v a : 1128) a t ja v a . ba se / ja v a . u t i l . co n c u rre n t . T h re a d P oo lE x ec u t or$ Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NoClassDefFoundError: org/lionsoul/ip2region/xdb/Searcher
at spark.util.IpToCityConverter2.convertToCity(IpToCityConverter2.java:19)
at spark.ZxyDiagrams.lambda $main$ c94c15e5 $1 (Z x yD ia g r am s . ja v a : 201) a t or g . a p a c h e . s p a r k . s ql . U D FR e g i s t r a t i o n .$ anonfun$register$352(UDFRegistration.scala:752)
… 19 more