You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _posts/2021-04-26-integrating-airflow-and-okta.md
+25-7Lines changed: 25 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -10,18 +10,23 @@ team: Core Platform
10
10
---
11
11
12
12
13
-
At Scribd we use Airflow as a scheduler for most of our batch workloads, this blog is not about Airflow so we are not getting into why Airflow. This is about one of the biggest challenge that we faced while using Airflow and finally conquer. That is how to do authentication and authorisation for Airflow. Of course Airflow does support LDAP and at Scribd we started using LDAP with Airflow initially, but as the organisation grow and more and more user started using Airflow, it became imperative that we integrate Airflow with our SSO provider that is OKTA. This write up will describe the journey of integrating Airflow with Okta from the earlier LDAP setup.
13
+
At Scribd we use Airflow as a scheduler for most of our batch workloads, this blog is not about Airflow so we are not getting into why Airflow. This is about one of the biggest challenge that we faced while using Airflow and finally conquer. That is how to do authentication and authorisation for Airflow. Of course Airflow does support LDAP and at Scribd we started using LDAP with Airflow initially, but as the organisation grow and more and more user started using Airflow, it became imperative that we integrate Airflow with our SSO provider that is OKTA.
14
14
15
+
Sadly there is a lack of resources on how to implement airflow with OKTA specifically. We are going to use Flask app builder along with some additional packages to integrate it via OKTA.
15
16
16
-
## Prerequisite:
17
-
This section will describe the minimum setup that will require to enable this integration,
17
+
This write up will describe the journey of integrating Airflow with Okta from the earlier LDAP setup.
18
+
19
+
20
+
## Prerequisite
21
+
This section will describe the minimum setup that will require to enable this integration.
18
22
1. Okta with [API Access Management](https://developer.okta.com/docs/concepts/api-access-management/) enabled. Without this feature enabled in OKTA we will not be able to integrate Airflow with OKTA
23
+
24
+
In Scribd we use a custom build docker image for Airflow, we install the following libraries in that docker image to make Airflow integration work with Okta
19
25
1.[Flask-AppBuilder
20
26
3.2.2](https://github.com/dpgaspar/Flask-AppBuilder/tree/v3.2.2). Official
21
27
Airflow repo has a
22
28
[constraint](https://github.com/apache/airflow/blob/master/setup.cfg#L97) on
23
-
`flask-appbuilder~=3.1,>=3.1.1`, so we might need to use a fork to get this
24
-
integration going.
29
+
`flask-appbuilder~=3.1,>=3.1.1`, so adding this additionally to the docker image helps us bypass that constraint
25
30
1.`sqlalchemy>=1.3.18, <1.4.0` --> This is because of some python dependency for Flask-AppBuilder
26
31
1.`authlib==0.15.3` --> authlib needs to installed along with Airflow to enable flask-appbuilder integration with Okta via OIDC
27
32
@@ -57,7 +62,7 @@ This section will describe the minimum setup that will require to enable this in
57
62
}
58
63
]
59
64
60
-
### Special Steps:
65
+
### Special Steps
61
66
62
67
63
68
1. We started with Flask-AppBuilder 3.2.1, however it had a bug that needs to
@@ -82,10 +87,23 @@ This section will describe the minimum setup that will require to enable this in
1. Currently in the audit log, any action triggered on Airflow has Okta user id. Airflow needs to be patched to write out audit log entries with human readable user identifiers instead.
88
93
94
+
## Final Stage
95
+
96
+
Once the setup is complete, you will find the similar tiles on your okta dashboard,
Hope this doc will help you setting it up, This journey was a bit tricky for us but we finally make it happen and we do hope that this doc will help a lot of folks to integrate Airflow with OKTA successfully.
89
107
---
90
108
91
109
Within Scribd's Platform Engineering group we have a *lot* more services than
0 commit comments