Member-only story
Automating Data Pipelines with Python & GitHub Actions
A simple (and free) way to run data workflows
This is the 4th article in a larger series on Full Stack Data Science (FSDS). In the last post, I shared a concrete example of how to build data pipelines for machine learning projects. One limitation of the example, however, was that the data pipeline had to be run manually. While this might be fine for some applications, more often than not, it’s better to automate than delegate this process to a person. In this article, I will walk through a simple way to do this using Python and GitHub Actions.
A few years ago, Andrej Karpathy gave a talk describing “Operation Vacation” [1]. This was what Tesla’s full self-driving engineering team called their goal of completely automating the improvement of the self-driving model.
Although this goal was somewhat facetious, it illustrates an aspiration I’ve seen in most data scientists and engineers: the desire to build a system that can operate autonomously (so they can go on vacation).
Here, I will discuss how to automate a key element of any machine learning system—the data pipeline.
