InTDS ArchivebySergey Kotlov·Nov 7, 2024Adopting Spark ConnectHow we use a shared Spark server to make our Spark infrastructure more efficientA response icon3A response icon3
Sergey Kotlov·Jul 8, 2022Spark on AWS using Spot Instances: ensuring capacity and optimizing costsIn this post, I will share our experience of using AWS Spot Instances for Spark applications. We will talk about the problems you may…
InTDS ArchivebySergey Kotlov·May 17, 2022Monitoring of Spark ApplicationsUsing custom metrics to detect problemsA response icon3A response icon3
Sergey Kotlov·Mar 29, 2022Unit Testing of Spark ApplicationsIn this post, we’ll look at one of the ways to unit test Spark applications and prepare test datasets. The motivation for using the…
Sergey Kotlov·Feb 23, 2022Logging for Spark on KubernetesIn 2021 at Joom, we migrated our Spark setup from EMR to Kubernetes/EKS. With Spark on EMR, we had support for logging out of the box. But…A response icon1A response icon1
