@@ -56,7 +56,7 @@ store, but in order to support our desired multi-repo approach DAGs, we needed
56
56
to build our own tooling to coordinate synchronizing the local DAG store with
57
57
S3 objects from the multiple DAG repositories.
58
58
59
- The tool we build , [ objinsync] ( https://github.com/scribd/objinsync ) [ ^ 1 ] , is a
59
+ The tool we built , [ objinsync] ( https://github.com/scribd/objinsync ) [ ^ 1 ] , is a
60
60
stateless DAG sync daemon, which is deployed as a sidecar container. From
61
61
Airflow’s point of view, the DAG folder is just a magical local folder that
62
62
always contains the up to date DAG definitions assembled from multiple Git
@@ -106,12 +106,12 @@ environment variables injected by the CI/CD system.
106
106
Our airflow clusters are orchestrated using both ECS fargate and EKS. ECS is
107
107
used to run Airflow web server and scheduler while EKS is what’s powering
108
108
Airflow’s Kubernetes executor. Due to differences in different Airflow
109
- components, we need to run ` objinsync ` binary in two container orchestration
109
+ components, we need to run the ` objinsync ` binary in two container orchestration
110
110
platforms with slightly different setups.
111
111
112
112
For daemon Airflow components like web server and scheduler, we run
113
113
` objinsync ` in a continuous sync mode where it pulls incremental updates from
114
- S3 to local filesystem every 5 seconds. This is implemented using sidecar
114
+ S3 to local filesystem every 5 seconds. This is implemented usingt the sidecar
115
115
container pattern. The DAG folder is mounted as a shared volume between the
116
116
Airflow web/scheduler container and objinsync container. The sidecar
117
117
objinsync container is setup to run the following command:
@@ -157,7 +157,7 @@ primitives from the Go runtime for better performance.
157
157
Engineering is all about making the right trade-offs. I won’t claim what we have
158
158
is the perfect solution for everyone, but I do believe it strikes a good
159
159
balance between productivity, operability, and availability. If you have any
160
- question regarding the setup, I am available in Airflow’s
160
+ questions regarding the setup, I am available in Airflow’s
161
161
[ #airflow-creative] ( https://apache-airflow.slack.com/messages/airflow-creative )
162
162
slack channel under the handle "QP." If you are not already part of the Airflow
163
163
Slack community, you can get access via
0 commit comments