Skip to content
This repository was archived by the owner on Mar 12, 2025. It is now read-only.

Commit 9dc9069

Browse files
author
Sachin Maheshwari
committed
intial version
0 parents  commit 9dc9069

File tree

13 files changed

+3869
-0
lines changed

13 files changed

+3869
-0
lines changed

README.md

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# UBahn - User Reconciliation Processor
2+
3+
Sync a v3 user
4+
5+
## Dependencies
6+
7+
- Nodejs(v12+)
8+
- Kafka
9+
10+
## Configuration
11+
12+
Configuration for the user reconciliation processor is at `config/default.js`.
13+
The following parameters can be set in config files or in env variables:
14+
15+
- LOG_LEVEL: the log level; default value: 'debug'
16+
- KAFKA_URL: comma separated Kafka hosts; default value: 'localhost:9092'
17+
- KAFKA_CLIENT_CERT: Kafka connection certificate, optional; default value is undefined;
18+
if not provided, then SSL connection is not used, direct insecure connection is used;
19+
if provided, it can be either path to certificate file or certificate content
20+
- KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional; default value is undefined;
21+
if not provided, then SSL connection is not used, direct insecure connection is used;
22+
if provided, it can be either path to private key file or private key content
23+
- KAFKA_GROUP_ID: the Kafka group id, default value is 'user-reconciliation-processor'
24+
- USER_RECONCILATION_TOPIC: Kafka topic, default value is 'backgroundjob.reconcile.user'
25+
- UBAHN_API_URL: The ubahn api url, default value: 'https://api.topcoder-dev.com/v5'
26+
- MEMBERS_API_URL: The topcoder member api url, default value: 'https://api.topcoder-dev.com/v5/members'
27+
- AUTH0_URL: The auth0 url, default value: 'https://topcoder-dev.auth0.com/oauth/token'
28+
- AUTH0_UBAHN_AUDIENCE: The auth0 audience for accessing ubahn api(s), default value: 'https://u-bahn.topcoder-dev.com'
29+
- AUTH0_TOPCODER_AUDIENCE: The auth0 audience for accessing ubahn api(s), default value: 'https://m2m.topcoder-dev.com/'
30+
- AUTH0_CLIENT_ID: The auth0 client id
31+
- AUTH0_CLIENT_SECRET: The auth0 client secret
32+
- AUTH0_PROXY_SERVER_URL: The auth0 proxy server url
33+
- SLEEP_TIME: The pause time between two create operations, default value: 1000 ms
34+
35+
There is a `/health` endpoint that checks for the health of the app. This sets up an expressjs server and listens on the environment variable `PORT`. It's not part of the configuration file and needs to be passed as an environment variable
36+
37+
## Local Kafka setup
38+
39+
### Install bin
40+
41+
- `http://kafka.apache.org/quickstart` contains details to setup and manage Kafka server,
42+
below provides details to setup Kafka server in Linux/Mac, Windows will use bat commands in bin/windows instead
43+
44+
### Local install with Docker
45+
46+
- Navigate to the directory `docker-kafka`
47+
- Run the command `docker-compose up -d`
48+
49+
## Local deployment
50+
51+
1. Make sure that Kafka is running.
52+
53+
2. From the project root directory, run the following command to install the dependencies
54+
55+
```bash
56+
npm install
57+
```
58+
59+
3. To run linters if required
60+
61+
```bash
62+
npm run lint
63+
```
64+
65+
To fix possible lint errors:
66+
67+
```bash
68+
npm run lint:fix
69+
```
70+
71+
4. Start the processor and health check dropin
72+
73+
```bash
74+
npm start
75+
```
76+
77+
## Local Deployment with Docker
78+
79+
To run this processor using docker, follow the below steps
80+
81+
1. Navigate to the directory `docker`
82+
83+
2. Rename the file `sample.api.env` to `api.env`
84+
85+
3. Set the auth0 config in the file `api.env`
86+
87+
4. Once that is done, run the following command
88+
89+
```bash
90+
docker-compose up
91+
```
92+
93+
5. When you are running the application for the first time, It will take some time initially to download the image and install the dependencies
94+
95+
## Verification
96+
97+
1. config `AUTH0_CLIENT_ID`, `AUTH0_CLIENT_SECRET`
98+
2. start kafka-console-producer to write messages to `backgroundjob.reconcile.user`
99+
`docker exec -it user-reconciliation-processor_kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic backgroundjob.reconcile.user`
100+
3. write message:
101+
`{ "topic": "backgroundjob.reconcile.user", "originator": "zapier", "timestamp": "2021-05-08T00:00:00.000Z", "mime-type": "application/json", "payload": {"handle":"billsedison"} }`
102+
4. Watch the app console, It will show message successfully handled.
103+
5. write non handle message:
104+
`{ "topic": "backgroundjob.reconcile.user", "originator": "backgroundjob.service", "timestamp": "2021-05-08T00:00:00.000Z", "mime-type": "application/json", "payload": {"id":"88774616","firstName":"Sachin1","lastName":"Kumar1"} }`
105+
6. watch the app console, it will show the ignoring message

build.sh

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
set -eo pipefail
3+
APP_NAME=$1
4+
UPDATE_CACHE=""
5+
echo "" > docker/api.env
6+
docker-compose -f docker/docker-compose.yml build $APP_NAME
7+
docker create --name app $APP_NAME:latest
8+
9+
if [ -d node_modules ]
10+
then
11+
mv package-lock.json old-package-lock.json
12+
docker cp app:/$APP_NAME/package-lock.json package-lock.json
13+
set +eo pipefail
14+
UPDATE_CACHE=$(cmp package-lock.json old-package-lock.json)
15+
set -eo pipefail
16+
else
17+
UPDATE_CACHE=1
18+
fi
19+
20+
if [ "$UPDATE_CACHE" == 1 ]
21+
then
22+
docker cp app:/$APP_NAME/node_modules .
23+
fi

config/default.js

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
/**
2+
* The default configuration file.
3+
*/
4+
5+
module.exports = {
6+
LOG_LEVEL: process.env.LOG_LEVEL || 'debug',
7+
8+
KAFKA_URL: process.env.KAFKA_URL || 'localhost:9092',
9+
// below are used for secure Kafka connection, they are optional
10+
// for the local Kafka, they are not needed
11+
KAFKA_CLIENT_CERT: process.env.KAFKA_CLIENT_CERT,
12+
KAFKA_CLIENT_CERT_KEY: process.env.KAFKA_CLIENT_CERT_KEY,
13+
14+
// Kafka group id
15+
KAFKA_GROUP_ID: process.env.KAFKA_GROUP_ID || 'u-bahn-user-reconciliation-processor',
16+
USER_RECONCILATION_TOPIC: process.env.USER_RECONCILATION_TOPIC || 'backgroundjob.reconcile.user',
17+
18+
UBAHN_API_URL: process.env.UBAHN_API_URL || 'https://api.topcoder-dev.com/v5',
19+
MEMBERS_API_URL: process.env.MEMBERS_API_URL || 'https://api.topcoder-dev.com/v5/members',
20+
21+
AUTH0_URL: process.env.AUTH0_URL || 'https://topcoder-dev.auth0.com/oauth/token', // Auth0 credentials
22+
AUTH0_UBAHN_AUDIENCE: process.env.AUTH0_UBAHN_AUDIENCE || 'https://u-bahn.topcoder.com',
23+
AUTH0_TOPCODER_AUDIENCE: process.env.AUTH0_TOPCODER_AUDIENCE || 'https://m2m.topcoder-dev.com/',
24+
AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
25+
AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
26+
AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL,
27+
28+
SLEEP_TIME: process.env.SLEEP_TIME ? parseInt(process.env.SLEEP_TIME, 10) : 1000
29+
}

docker-kafka/docker-compose.yml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
version: '3'
2+
services:
3+
zookeeper:
4+
image: wurstmeister/zookeeper
5+
container_name: user-reconciliation-processor_zookeeper
6+
ports:
7+
- "2181:2181"
8+
kafka:
9+
image: wurstmeister/kafka
10+
container_name: user-reconciliation-processor_kafka
11+
ports:
12+
- "9092:9092"
13+
environment:
14+
KAFKA_ADVERTISED_HOST_NAME: localhost
15+
KAFKA_CREATE_TOPICS: "backgroundjob.reconcile.user:1:1"
16+
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

docker/Dockerfile

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Use the base image with Node.js 12.16.3
2+
FROM node:12.16.3
3+
4+
# Copy the current directory into the Docker image
5+
COPY . /u-bahn-user-reconciliation-processor
6+
7+
# Set working directory for future use
8+
WORKDIR /u-bahn-user-reconciliation-processor
9+
10+
# Install the dependencies from package.json
11+
RUN npm install
12+
CMD npm start

docker/docker-compose.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
version: '3'
2+
services:
3+
u-bahn-user-reconciliation-processor:
4+
image: u-bahn-user-reconciliation-processor:latest
5+
build:
6+
context: ../
7+
dockerfile: docker/Dockerfile
8+
env_file:
9+
- api.env
10+
network_mode: "host"

docker/sample.api.env

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
AUTH0_CLIENT_ID=<Auth0 Client ID>
2+
AUTH0_CLIENT_SECRET=<Auth0 Client Secret>

0 commit comments

Comments
 (0)