Sergey Ivanychev·Aug 8, 2024Simple trick to debug stuck Python jobsTL;DR: in separate thread run a function that periodically records and saves stack traces of all threads and write them to external…
InConstructor EngineeringbySergey Ivanychev·Nov 28, 2023Observe and record performance of Databricks jobsUsing Victoria Metrics for Databricks Spark monitoring performance
InConstructor EngineeringbySergey Ivanychev·Aug 31, 2023How to optimize AWS S3 costs via granular visualisationBuild a report on how much every folder costs, who accessed it, and whether it should be removed to optimize spending.A response icon1A response icon1
InConstructor EngineeringbySergey Ivanychev·Jun 24, 2022Faster PySpark Unit TestsTL;DR: A PySpark unit test setup for pytest that uses efficient default settings and utilizes all CPU cores via pytest-xdist is available…A response icon1A response icon1
InJoombySergey Ivanychev·Dec 1, 2021Building data platform in PySpark. Part 1 — Python and Scala interopAccording to multiple recent surveys (as of 2021), Python is among the most widely used programming languages and its share of users is…