PinnedLucianosantosg·Dec 3, 2024Decoupling nested data in PySpark dataframesI suppose this is a frequent challenge for data engineers working with PySpark. Many times I got a dataframe with a column with a list data…
PinnedLucianosantosg·Jan 7, 2025Creating a Star Schema with Pandas DataframesIt’s usual to load pandas dataframes into our target database, just as a direct integration step, so start working on analysis in the SQL…
PinnedLucianosantosg·Feb 20, 2025Pandasql: Manipulating dataframes using SQL in Python.Pandas lib is already awesome for data handling in Python. We can turn a dataset into a dataframe table and have all the data manipulation…
Lucianosantosg·Feb 24, 2025Medallion Architecture pipeline in an OOP structureIn this brief article, we are going to explore the steps of a standard Spark Medallion architecture pipeline in an Object-Oriented…