You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _posts/2021-04-26-integrating-airflow-and-okta.md
+41-23Lines changed: 41 additions & 23 deletions
Original file line number
Diff line number
Diff line change
@@ -1,37 +1,46 @@
1
1
---
2
2
layout: post
3
-
title: "Integrating Airflow with OKTA"
4
-
author: Kuntalb, Greg Reznik
3
+
title: "Integrating Airflow with Okta"
4
+
author: kuntalb
5
5
tags:
6
-
- OKTA
7
-
- Airflow
6
+
- okta
7
+
- airflow
8
8
- featured
9
-
team: Core Platform, IT
9
+
team: Core Platform
10
10
---
11
-
At Scribd we decided to pair Airflow with OKTA. Earlier we were using LDAP for authentication. This write up will describe the journey of integrating Airflow with OKTA from the earlier LDAP setup.
11
+
12
+
13
+
At Scribd we decided to pair Airflow with Okta. Earlier we were using LDAP for
14
+
authentication. This write up will describe the journey of integrating Airflow
15
+
with Okta from the earlier LDAP setup.
12
16
13
17
14
18
## Prerequisite:
15
-
1. OKTA with api access management
16
-
2.[Flask-AppBuilder 3.2.2](https://github.com/dpgaspar/Flask-AppBuilder/tree/v3.2.2). Official Airflow repo has a [constraint](https://github.com/apache/airflow/blob/master/setup.cfg#L97) on flask-appbuilder~=3.1,>=3.1.1 , so we might need to use a fork to get this integration going.
17
-
3. sqlalchemy>=1.3.18, <1.4.0
18
-
4. authlib==0.15.3
19
+
1. Okta with [API Access Management](https://developer.okta.com/docs/concepts/api-access-management/) enabled.
20
+
1.[Flask-AppBuilder
21
+
3.2.2](https://github.com/dpgaspar/Flask-AppBuilder/tree/v3.2.2). Official
22
+
Airflow repo has a
23
+
[constraint](https://github.com/apache/airflow/blob/master/setup.cfg#L97) on
24
+
`flask-appbuilder~=3.1,>=3.1.1`, so we might need to use a fork to get this
1. Create an OIDC Web application. Give it a name and leave the values under the “Configure OpenID Connect” section empty.
26
-
2. Make note of the Client ID and the Client Secret, as you will need them for configuring the airflow webserver.
27
-
3. In the “Allowed Grant Types” section, make sure you check all of the boxes.
28
-
4. For the Login redirect URIs field, you will enter: https://your-airflow-url-goes-here.com/oauth-authorized/okta
29
-
5. For the Initiate login URI field, you will enter: https://your-airflow-url-goes-here.com/login
35
+
1. Make note of the Client ID and the Client Secret, as you will need them for configuring the airflow webserver.
36
+
1. In the “Allowed Grant Types” section, make sure you check all of the boxes.
37
+
1. For the Login redirect URIs field, you will enter: `https://your-airflow-url-goes-here.com/oauth-authorized/okta`
38
+
1. For the Initiate login URI field, you will enter: `https://your-airflow-url-goes-here.com/login`
30
39
31
40
## Airflow Configuration
32
41
33
-
### conf/webserver_config.py
34
-
42
+
`conf/webserver_config.py`
43
+
35
44
AUTH_TYPE = AUTH_OAUTH
36
45
OAUTH_PROVIDERS = [
37
46
{'name': 'okta', 'icon': 'fa-circle-o',
@@ -52,14 +61,23 @@ At Scribd we decided to pair Airflow with OKTA. Earlier we were using LDAP for a
52
61
### Special Steps:
53
62
54
63
55
-
1. We started with Flask-AppBuilder 3.2.1,however it had a bug that needs to be fixed, we raised a [PR] (https://github.com/dpgaspar/Flask-AppBuilder/pull/1589)Flask-AppBuilder: fix: load user info for oktaCLOSED to resolve that issue. That PR got merged and now we can use the new release, Flask-AppBuilder 3.2.2
64
+
1. We started with Flask-AppBuilder 3.2.1, however it had a bug that needs to
65
+
be fixed, we raised a [PR](https://github.com/dpgaspar/Flask-AppBuilder/pull/1589) for Flask-AppBuilder:
66
+
fix: load user info for oktaCLOSED to resolve that issue. That PR got
67
+
merged and now we can use the new release, Flask-AppBuilder 3.2.2
56
68
57
-
2. As we were migrating from LDAP, we will already have user info populated, however OKTA generates a new user id something like thisokta_00u1046sqzJprt1hZ4x6 , but as the email id corresponding to that user id is already present we got the below error. To prevent this we logged into the underlying database for Airflow and cleaned up the ab_user and ab_user_role table and let OKTA integration recreate the user during first sign up.
69
+
2. As we were migrating from LDAP, we will already have user info populated,
70
+
however Okta generates a new user id something like
71
+
`thisokta_00u1046sqzJprt1hZ4x6`, but as the email id corresponding to that
72
+
user id is already present we got the below error. To prevent this we logged
73
+
into the underlying database for Airflow and cleaned up the ab_user and
74
+
ab_user_role table and let Okta integration recreate the user during first
75
+
sign up.
58
76
59
77
```
60
78
[2021-03-19 16:32:28,559] {manager.py:215} ERROR - Error adding new user to database. (sqlite3.IntegrityError) UNIQUE constraint failed: ab_user.email
[2021-03-19 16:32:28,560] {manager.py:1321} ERROR - Error creating a new OAuth user okta_00u1046sqzJprt1hZ4x6
80
+
[2021-03-19 16:32:28,560] {manager.py:1321} ERROR - Error creating a new OAuth user okta_00u1046sqzJprt1hZ4x6
63
81
```
64
82
3. Because we have deleted all the existing user and role, once the users logged in for the first time, especially for the first admin user we did the following from the airflow cli. This will create the first admin user after that if needed we can propagate other user and roles from the Airflow web console from this admin user account.
65
83
```
@@ -68,7 +86,7 @@ At Scribd we decided to pair Airflow with OKTA. Earlier we were using LDAP for a
68
86
69
87
## Known Issue:
70
88
71
-
1. Currently in the audit log, any action triggered on Airflow has OKTA user id. Airflow needs to be patched to write out audit log entries with human readable user identifiers instead.
89
+
1. Currently in the audit log, any action triggered on Airflow has Okta user id. Airflow needs to be patched to write out audit log entries with human readable user identifiers instead.
0 commit comments