intial version

Author: Sachin Maheshwari

README.md

@@ -0,0 +1,105 @@
+# UBahn - User Reconciliation Processor
+
+Sync a v3 user  
+
+## Dependencies
+
+- Nodejs(v12+)
+- Kafka
+
+## Configuration
+
+Configuration for the user reconciliation processor is at `config/default.js`.
+The following parameters can be set in config files or in env variables:
+
+- LOG_LEVEL: the log level; default value: 'debug'
+- KAFKA_URL: comma separated Kafka hosts; default value: 'localhost:9092'
+- KAFKA_CLIENT_CERT: Kafka connection certificate, optional; default value is undefined;
+    if not provided, then SSL connection is not used, direct insecure connection is used;
+    if provided, it can be either path to certificate file or certificate content
+- KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional; default value is undefined;
+    if not provided, then SSL connection is not used, direct insecure connection is used;
+    if provided, it can be either path to private key file or private key content
+- KAFKA_GROUP_ID: the Kafka group id, default value is 'user-reconciliation-processor'
+- USER_RECONCILATION_TOPIC: Kafka topic, default value is 'backgroundjob.reconcile.user'
+- UBAHN_API_URL: The ubahn api url, default value: 'https://api.topcoder-dev.com/v5'
+- MEMBERS_API_URL: The topcoder member api url, default value: 'https://api.topcoder-dev.com/v5/members'
+- AUTH0_URL: The auth0 url, default value: 'https://topcoder-dev.auth0.com/oauth/token'
+- AUTH0_UBAHN_AUDIENCE: The auth0 audience for accessing ubahn api(s), default value: 'https://u-bahn.topcoder-dev.com'
+- AUTH0_TOPCODER_AUDIENCE: The auth0 audience for accessing ubahn api(s), default value: 'https://m2m.topcoder-dev.com/'
+- AUTH0_CLIENT_ID: The auth0 client id
+- AUTH0_CLIENT_SECRET: The auth0 client secret
+- AUTH0_PROXY_SERVER_URL: The auth0 proxy server url
+- SLEEP_TIME: The pause time between two create operations, default value: 1000 ms
+
+There is a `/health` endpoint that checks for the health of the app. This sets up an expressjs server and listens on the environment variable `PORT`. It's not part of the configuration file and needs to be passed as an environment variable
+
+## Local Kafka setup
+
+### Install bin
+
+- `http://kafka.apache.org/quickstart` contains details to setup and manage Kafka server,
+  below provides details to setup Kafka server in Linux/Mac, Windows will use bat commands in bin/windows instead
+
+### Local install with Docker
+
+- Navigate to the directory `docker-kafka`
+- Run the command `docker-compose up -d`
+
+## Local deployment
+
+1. Make sure that Kafka is running.
+
+2. From the project root directory, run the following command to install the dependencies
+
+    ```bash
+    npm install
+    ```
+
+3. To run linters if required
+
+    ```bash
+    npm run lint
+    ```
+
+    To fix possible lint errors:
+
+    ```bash
+    npm run lint:fix
+    ```
+
+4. Start the processor and health check dropin
+
+    ```bash
+    npm start
+    ```
+
+## Local Deployment with Docker
+
+To run this processor using docker, follow the below steps
+
+1. Navigate to the directory `docker`
+
+2. Rename the file `sample.api.env` to `api.env`
+
+3. Set the auth0 config in the file `api.env`
+
+4. Once that is done, run the following command
+
+    ```bash
+    docker-compose up
+    ```
+
+5. When you are running the application for the first time, It will take some time initially to download the image and install the dependencies
+
+## Verification
+
+1. config `AUTH0_CLIENT_ID`, `AUTH0_CLIENT_SECRET`
+2. start kafka-console-producer to write messages to `backgroundjob.reconcile.user`
+  `docker exec -it user-reconciliation-processor_kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic backgroundjob.reconcile.user`
+3. write message:
+  `{ "topic": "backgroundjob.reconcile.user", "originator": "zapier", "timestamp": "2021-05-08T00:00:00.000Z", "mime-type": "application/json", "payload": {"handle":"billsedison"} }`
+4. Watch the app console, It will show message successfully handled.
+5. write non handle message:
+  `{ "topic": "backgroundjob.reconcile.user", "originator": "backgroundjob.service", "timestamp": "2021-05-08T00:00:00.000Z", "mime-type": "application/json", "payload": {"id":"88774616","firstName":"Sachin1","lastName":"Kumar1"} }`
+6. watch the app console, it will show the ignoring message 

build.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+set -eo pipefail
+APP_NAME=$1
+UPDATE_CACHE=""
+echo "" > docker/api.env
+docker-compose -f docker/docker-compose.yml build $APP_NAME
+docker create --name app $APP_NAME:latest
+
+if [ -d node_modules ]
+then
+  mv package-lock.json old-package-lock.json
+  docker cp app:/$APP_NAME/package-lock.json package-lock.json
+  set +eo pipefail
+  UPDATE_CACHE=$(cmp package-lock.json old-package-lock.json)
+  set -eo pipefail
+else
+  UPDATE_CACHE=1
+fi
+
+if [ "$UPDATE_CACHE" == 1 ]
+then
+  docker cp app:/$APP_NAME/node_modules .
+fi

config/default.js

@@ -0,0 +1,29 @@
+/**
+ * The default configuration file.
+ */
+
+module.exports = {
+  LOG_LEVEL: process.env.LOG_LEVEL || 'debug',
+
+  KAFKA_URL: process.env.KAFKA_URL || 'localhost:9092',
+  // below are used for secure Kafka connection, they are optional
+  // for the local Kafka, they are not needed
+  KAFKA_CLIENT_CERT: process.env.KAFKA_CLIENT_CERT,
+  KAFKA_CLIENT_CERT_KEY: process.env.KAFKA_CLIENT_CERT_KEY,
+
+  // Kafka group id
+  KAFKA_GROUP_ID: process.env.KAFKA_GROUP_ID || 'u-bahn-user-reconciliation-processor',
+  USER_RECONCILATION_TOPIC: process.env.USER_RECONCILATION_TOPIC || 'backgroundjob.reconcile.user',
+
+  UBAHN_API_URL: process.env.UBAHN_API_URL || 'https://api.topcoder-dev.com/v5',
+  MEMBERS_API_URL: process.env.MEMBERS_API_URL || 'https://api.topcoder-dev.com/v5/members',
+
+  AUTH0_URL: process.env.AUTH0_URL || 'https://topcoder-dev.auth0.com/oauth/token', // Auth0 credentials
+  AUTH0_UBAHN_AUDIENCE: process.env.AUTH0_UBAHN_AUDIENCE || 'https://u-bahn.topcoder.com',
+  AUTH0_TOPCODER_AUDIENCE: process.env.AUTH0_TOPCODER_AUDIENCE || 'https://m2m.topcoder-dev.com/',
+  AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
+  AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
+  AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL,
+
+  SLEEP_TIME: process.env.SLEEP_TIME ? parseInt(process.env.SLEEP_TIME, 10) : 1000
+}

docker-kafka/docker-compose.yml

@@ -0,0 +1,16 @@
+version: '3'
+services:
+  zookeeper:
+    image: wurstmeister/zookeeper
+    container_name: user-reconciliation-processor_zookeeper
+    ports:
+      - "2181:2181"
+  kafka:
+    image: wurstmeister/kafka
+    container_name: user-reconciliation-processor_kafka
+    ports:
+      - "9092:9092"
+    environment:
+      KAFKA_ADVERTISED_HOST_NAME: localhost
+      KAFKA_CREATE_TOPICS: "backgroundjob.reconcile.user:1:1"
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

docker/Dockerfile

@@ -0,0 +1,12 @@
+# Use the base image with Node.js 12.16.3
+FROM node:12.16.3
+
+# Copy the current directory into the Docker image
+COPY . /u-bahn-user-reconciliation-processor
+
+# Set working directory for future use
+WORKDIR /u-bahn-user-reconciliation-processor
+
+# Install the dependencies from package.json
+RUN npm install
+CMD npm start

docker/docker-compose.yml

@@ -0,0 +1,10 @@
+version: '3'
+services:
+  u-bahn-user-reconciliation-processor:
+    image: u-bahn-user-reconciliation-processor:latest
+    build:
+      context: ../
+      dockerfile: docker/Dockerfile
+    env_file:
+      - api.env
+    network_mode: "host"

docker/sample.api.env

@@ -0,0 +1,2 @@
+AUTH0_CLIENT_ID=<Auth0 Client ID>
+AUTH0_CLIENT_SECRET=<Auth0 Client Secret>