PinnedArthur Caron·Jan 5, 2024Optimizing Pyspark code for Delta formatOptimising Python code for handling data in Delta format, especially when working with large datasets, requires a blend of efficient coding…A response icon2A response icon2
PinnedArthur Caron·Mar 9, 2024Pyspark OptimizationOptimizing PySpark applications involves several strategies to improve performance and efficiency, especially when dealing with large-scale…
Arthur Caron·Dec 12, 2023Index in a Medallion Architecture Using Delta Lake with PySparkIn this article, we explore the integration of Delta Lake in a medallion architecture, focusing on the use of PySpark’s row_number function…
Arthur Caron·Dec 8, 2023Delta Lake: The Future of Data Lake ManagementA discussion on the benefits offered by Data Lake compare to the traditional Data Lake.