PinnedRavindra Kumar·Sep 28, 2025Optimizing Databricks/Spark Batch Jobs: How We Cut CDC Ingestion Runtime in Half with S3 Events…Every data engineer knows this pain: your batch jobs take hours to run, but most of that time isn’t spent on useful work — it’s just…A response icon1A response icon1
PinnedInPython in Plain EnglishbyRavindra Kumar·Oct 8, 2023Implementing the Open-Closed Principle in Data Processing: Organizing Code with pythonic Modular…In the world of software engineering, adhering to solid design principles is essential for building maintainable, scalable, and efficient…
PinnedInPython in Plain EnglishbyRavindra Kumar·Oct 2, 2023How to Automate Databricks Job Triggers Pythonically with Apache Airflow: A Custom Operator…
PinnedInPython in Plain EnglishbyRavindra Kumar·Aug 24, 2023Crafting Robust Data Pipelines with SOLID Principles: Python’s Approach to Data EngineeringIn the bustling world of data engineering, the art of crafting robust, maintainable, and scalable pipelines is a feat not easily achieved…
Ravindra Kumar·Jul 20, 2024Unlocking the Hidden Power of Databricks: The _metadata ColumnIn the ever-evolving world of data engineering, finding hidden gems that can streamline your workflows is always exciting. One such gem in…
Ravindra Kumar·Apr 13, 2024Streamline PySpark/Spark Debugging and Job Management with Custom Job DescriptionsIntroduction
Ravindra Kumar·Apr 8, 2024Optimizing Apache Spark with External Shuffle ServiceApache Spark’s shuffle operation is a critical yet resource-intensive process required for a wide range of data transformation tasks…
Ravindra Kumar·Nov 13, 2023Data Engineering and Design Patterns: Simplifying the Workflow with the Factory PatternIn the intricate world of data engineering, managing diverse data formats and sources is a common challenge. The factory pattern offers a…A response icon1A response icon1
InPython in Plain EnglishbyRavindra Kumar·Nov 10, 2023Python UDTFs in PySpark: A Practical Guide to Crafting and Utilizing Advanced Data Solutions
InPython in Plain EnglishbyRavindra Kumar·Oct 26, 2023Mastering PySpark’s Elegance: Navigating Efficient Resource Management with the Singleton Design…Introduction