PinnedInData Engineer ThingsbyKerrache Massipssa·Dec 20, 2023Why Apache Iceberg Changes the GameIf you’ve had experience with data lakes, you likely faced significant challenges related to executing updates and deletes. Managing the…A response icon2A response icon2
PinnedInData Engineer ThingsbyKerrache Massipssa·Jan 7, 2024Why You Should Avoid Using UDFs in PySpark?A response icon5A response icon5
PinnedKerrache Massipssa·Dec 25, 2023How does Adaptive Query Execution fix your Spark performance issues?In Spark versions prior to 3.0, the common performance issues encountered are:A response icon2A response icon2
PinnedInTowards Data EngineeringbyKerrache Massipssa·Dec 12, 2023Data Quality with Great Expectations and PySparkBoost Your Data Quality !A response icon1A response icon1
PinnedInData Engineer ThingsbyKerrache Massipssa·Dec 14, 2023Apache Spark Partitioning and BucketingLearn the Partitioning and Bucketing with Apache Spark (PySpark) and understand how and when to use each of them.A response icon2A response icon2
Kerrache Massipssa·Feb 28, 2025Setup dbt in local with PostgreSQLIn this article, we’ll walk through installing dbt locally using a Python virtual environment. Then, we’ll provision a local PostgreSQL…
Kerrache Massipssa·Jan 8, 2024How Does Apache Spark Manage Executor Memory?On-heap memory (Spark Executor Memory): The size is configured by the — executor-memory or spark.executor.memory parameter at Spark…
InData Engineer ThingsbyKerrache Massipssa·Dec 28, 2023Exciting New Feature in Spark: “Spark Connect API”Spark introduced Spark Connect in version 3.4.0, an exciting feature that adds significant capabilities to the platform. In this article…