PinnedInAWS in Plain EnglishbySagadevan K·Jun 11, 2023ETL: MS SQL Server to Redshift via S3 using Pandas and PyArrowIn this article, we’ll see how we can load data from an MS SQL Server instance to a Redshift cluster using pandas, pyarrow and pg8000. Code…A response icon3A response icon3
PinnedSagadevan K·Mar 21, 2023Ways to load data from source to targetIn an ETL process, there are multiple ways to load data from a source system to a target system. Let’s have a look at some of them.A response icon2A response icon2
Sagadevan K·Mar 10Spark Cluster Setup in EMR on EC2: Practical Production GuidelinesMost Spark mysteries are configuration issues in disguise.
Sagadevan K·Mar 10Navigating the EMR on EC2 ConsoleLet’s navigate the different options available on the EMR on EC2 console.
Sagadevan K·Feb 15A Practical Data Engineering Checklist I Actually UseOver time, most data engineering failures stop being mysterious. Pipelines break for the same reasons, backfills are painful usually…
Sagadevan K·Feb 12Data Engineering Terminologies Explained Using Everyday AnalogiesThe fastest way to understand data engineering is not definitions, but analogies.
Sagadevan K·Feb 4Where’d all the time go?Whenever I come across a quote, saying or an anecdote I like, I have a habit of writing it down in my notes. Recently, when I found one, I…
InPython in Plain EnglishbySagadevan K·May 27, 2024Schema Evolution Made Simple with Pandas and PostgreSQLHandling schema changes using Pandas and PostgreSQL
Sagadevan K·Jan 8, 2023Test AWS Glue jobs locally using Docker and motoWe’ll find out how to run tests locally for an AWS Glue job which reads data from a Postgres instance and writes to S3 using Docker and…