Skip to content

Commit b18411e

Browse files
kuntalkumarbasurtyler
authored andcommitted
incorporating review comments
1 parent 1a89ac8 commit b18411e

File tree

3 files changed

+25
-7
lines changed

3 files changed

+25
-7
lines changed

_posts/2021-04-26-integrating-airflow-and-okta.md

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,18 +10,23 @@ team: Core Platform
1010
---
1111

1212

13-
At Scribd we use Airflow as a scheduler for most of our batch workloads, this blog is not about Airflow so we are not getting into why Airflow. This is about one of the biggest challenge that we faced while using Airflow and finally conquer. That is how to do authentication and authorisation for Airflow. Of course Airflow does support LDAP and at Scribd we started using LDAP with Airflow initially, but as the organisation grow and more and more user started using Airflow, it became imperative that we integrate Airflow with our SSO provider that is OKTA. This write up will describe the journey of integrating Airflow with Okta from the earlier LDAP setup.
13+
At Scribd we use Airflow as a scheduler for most of our batch workloads, this blog is not about Airflow so we are not getting into why Airflow. This is about one of the biggest challenge that we faced while using Airflow and finally conquer. That is how to do authentication and authorisation for Airflow. Of course Airflow does support LDAP and at Scribd we started using LDAP with Airflow initially, but as the organisation grow and more and more user started using Airflow, it became imperative that we integrate Airflow with our SSO provider that is OKTA.
1414

15+
Sadly there is a lack of resources on how to implement airflow with OKTA specifically. We are going to use Flask app builder along with some additional packages to integrate it via OKTA.
1516

16-
## Prerequisite:
17-
This section will describe the minimum setup that will require to enable this integration,
17+
This write up will describe the journey of integrating Airflow with Okta from the earlier LDAP setup.
18+
19+
20+
## Prerequisite
21+
This section will describe the minimum setup that will require to enable this integration.
1822
1. Okta with [API Access Management](https://developer.okta.com/docs/concepts/api-access-management/) enabled. Without this feature enabled in OKTA we will not be able to integrate Airflow with OKTA
23+
24+
In Scribd we use a custom build docker image for Airflow, we install the following libraries in that docker image to make Airflow integration work with Okta
1925
1. [Flask-AppBuilder
2026
3.2.2](https://github.com/dpgaspar/Flask-AppBuilder/tree/v3.2.2). Official
2127
Airflow repo has a
2228
[constraint](https://github.com/apache/airflow/blob/master/setup.cfg#L97) on
23-
`flask-appbuilder~=3.1,>=3.1.1`, so we might need to use a fork to get this
24-
integration going.
29+
`flask-appbuilder~=3.1,>=3.1.1`, so adding this additionally to the docker image helps us bypass that constraint
2530
1. `sqlalchemy>=1.3.18, <1.4.0` --> This is because of some python dependency for Flask-AppBuilder
2631
1. `authlib==0.15.3` --> authlib needs to installed along with Airflow to enable flask-appbuilder integration with Okta via OIDC
2732

@@ -57,7 +62,7 @@ This section will describe the minimum setup that will require to enable this in
5762
}
5863
]
5964

60-
### Special Steps:
65+
### Special Steps
6166

6267

6368
1. We started with Flask-AppBuilder 3.2.1, however it had a bug that needs to
@@ -82,10 +87,23 @@ This section will describe the minimum setup that will require to enable this in
8287
airflow users add-role -r Admin -u okta_00u1046sqzJprt1hZ4x6
8388
```
8489
85-
## Known Issue:
90+
## Known Issue
8691
8792
1. Currently in the audit log, any action triggered on Airflow has Okta user id. Airflow needs to be patched to write out audit log entries with human readable user identifiers instead.
8893
94+
## Final Stage
95+
96+
Once the setup is complete, you will find the similar tiles on your okta dashboard,
97+
98+
![Sample Okta Tiles](/post-images/2021-04-okta-airflow/okta-tiles.png)
99+
<font size="3"><center><i>Sample Okta Tiles </i></center></font>
100+
101+
Once you select the tiles, it should redirect you to the below page
102+
103+
![Sample Okta Login Page](/post-images/2021-04-okta-airflow/airflow-login.png)
104+
<font size="3"><center><i>Okta Login Page </i></center></font>
105+
106+
Hope this doc will help you setting it up, This journey was a bit tricky for us but we finally make it happen and we do hope that this doc will help a lot of folks to integrate Airflow with OKTA successfully.
89107
---
90108
91109
Within Scribd's Platform Engineering group we have a *lot* more services than
Loading
22.8 KB
Loading

0 commit comments

Comments
 (0)