Use leaderboard api instead of directly interacting with database. Also store the groupId of the contest in the database

Author: callmekatootie

README.md

@@ -4,7 +4,6 @@
 
 - nodejs https://nodejs.org/en/ (v10)
 - Kafka (v2)
-- Mongodb (v4)
 
 ## Configuration
 
@@ -20,15 +19,25 @@ The following parameters can be set in config files or in env variables:
 - KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional;
     if not provided, then SSL connection is not used, direct insecure connection is used;
     if provided, it can be either path to private key file or private key content
+- KAFKA_GROUP_ID: Kafka group id
 - CREATE_DATA_TOPIC: Kafka topic related to creation
 - UPDATE_DATA_TOPIC: Kafka topic related to update
 - DELETE_DATA_TOPIC: Kafka topic related to deletion
-- GROUP_IDS: List of Group IDs which should be used for filtering
+- LEADERBOARD_API_URL: Leaderboard API URL
 - SUBMISSION_API_URL: Submission API URL
-- CHALLENGE_API_URL: Challenge API URL
-- MEMBER_API_URL: Member API URL
-- All variables starting with prefix `AUTH0` corresponds to Auth0 related credentials
-- MONGODB_URL: Mongo DB URL
+- AUTH0_URL: Auth0 URL, used to get TC M2M token
+- AUTH0_AUDIENCE: Auth0 audience, used to get TC M2M token
+- TOKEN_CACHE_TIME: Auth0 token cache time, used to get TC M2M token
+- AUTH0_CLIENT_ID: Auth0 client id, used to get TC M2M token
+- AUTH0_CLIENT_SECRET: Auth0 client secret, used to get TC M2M token
+- AUTH0_PROXY_SERVER_URL: Proxy Auth0 URL, used to get TC M2M token
+
+Also note that there is a `/health` endpoint that checks for the health of the app. This sets up an expressjs server and listens on the environment variable `PORT`. It's not part of the configuration file and needs to be passed as an environment variable
+
+Configuration for the tests is at `config/test.js`, only add such new configurations different from `config/default.js`
+- WAIT_TIME: wait time used in test, default is 1000 or one second
+- LEADERBOARD_API_URL: Leaderboard API URL used in testing
+- SUBMISSION_API_URL: Submission API URL used in testing
 
 ## Local Kafka setup
 
@@ -47,7 +56,7 @@ The following parameters can be set in config files or in env variables:
 ```bash
   bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic submission.notification.create
   bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic submission.notification.update
-  bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic notifications.autopilot.events
+  bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic submission.notification.delete
 ```
 
 - verify that the topics are created:
@@ -58,7 +67,7 @@ The following parameters can be set in config files or in env variables:
 - In the console, write some message, one message per line:
 
 ```bash
-  { "topic":"submission.notification.create", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"review", "id": "d34d4180-65aa-42ec-a945-5fd21dec0502", "score": 92.0, "typeId": "c56a4180-65aa-42ec-a945-5fd21dec0501", "reviewerId": "c23a4180-65aa-42ec-a945-5fd21dec0503", "scoreCardId": "b25a4180-65aa-42ec-a945-5fd21dec0503", "submissionId": "fad49103-37ac-4a04-8294-c840483178a5", "created": "2018-05-20T07:00:30.123Z", "updated": "2018-06-01T07:36:28.178Z", "createdBy": "admin", "updatedBy": "admin" } }
+  { "topic":"submission.notification.create", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0508", "submissionId": "a34e1158-2c27-4d38-b079-5e5cca1bdcf7", "aggregateScore": 88, "scoreCardId": "b25a4180-65aa-42ec-a945-5fd21dec0503", "isPassing": true, "created": "2018-05-20T07:00:30.123Z", "updated": "2018-06-01T07:36:28.178Z", "createdBy": "admin", "updatedBy": "admin" } }
 ```
 
 - optionally, use another terminal, go to same directory, start a consumer to view the messages:
@@ -97,14 +106,43 @@ npm start
 - git add .
 - git commit -m init
 - heroku create
-- heroku config:set KAFKA_URL=... AUTH0_URL=... 
+- heroku config:set KAFKA_URL=... AUTH0_URL=...
 - git push heroku master
 
 ## Verification
 
-1. Ensure that Kafka is up and running and the topics `submission.notification.create, submission.notification.update and notifications.autopilot.events` are created in Kafka
+1. Ensure that Kafka is up and running and the topics `submission.notification.create, submission.notification.update and submission.notification.delete` are created in Kafka
+
+2. Refer `README.md` in `leaderboard-api` to start leaderboard api, all operations are under `leaderboard-api` project root folder
+
+- Start mock app and it will listen on 3001 PORT.
+
+```bash
+npm run mock-api
+```
+
+- Ensure you have start MongoDB and properly configure `MONGODB_URL`. Run the following commands to clear and insert test data, step up environment variables and start the app(it will listen on 3002 PORT!).
+
+```bash
+npm run init-db
+npm run test-data
+export CHALLENGE_API_URL=http://localhost:3001/challenges
+export MEMBER_API_URL=http://localhost:3001/users
+export PORT=3002
+npm start
+```
 
-2. Ensure that MONGODB_URL configured is correct and ensure that Submission API URL is pointing to `https://api.topcoder-dev.com/v5/submissions` (To verify based on the data present in Dev)
+3. Run the following command to start processor app(Under this project's root folder)
+```bash
+export AUTH0_URL=https://topcoder-dev.auth0.com/oauth/token
+export AUTH0_AUDIENCE=https://m2m.topcoder-dev.com/
+export TOKEN_CACHE_TIME=90
+export AUTH0_CLIENT_ID=8QovDh27SrDu1XSs68m21A1NBP8isvOt
+export AUTH0_CLIENT_SECRET=3QVxxu20QnagdH-McWhVz0WfsQzA1F8taDdGDI4XphgpEYZPcMTF4lX3aeOIeCzh
+export LEADERBOARD_API_URL=http://localhost:3002/v5/leaderboard
+export SUBMISSION_API_URL=http://localhost:3001/submissions
+npm start
+```
 
 3. Attach to the topic `submission.notification.create` using Kafka console producer
 
@@ -115,47 +153,50 @@ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.no
 4. Write the following message to the Console
 
 ```bash
-{ "topic":"submission.notification.create", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0507", "aggregateScore": 87.5, "isPassing": true, "scoreCardId": "b25a4180-65aa-42ec-a945-5fd21dec0503", "submissionId": "fad49103-37ac-4a04-8294-c840483178a5", "created": "2018-05-20T07:00:30.123Z", "updated": "2018-06-01T07:36:28.178Z", "createdBy": "admin", "updatedBy": "admin" } }
+{ "topic":"submission.notification.create", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0508", "submissionId": "a34e1158-2c27-4d38-b079-5e5cca1bdcf7", "aggregateScore": 90, "scoreCardId": "b25a4180-65aa-42ec-a945-5fd21dec0503", "isPassing": true, "metadata": { "assertions": { "pending": 0, "failed": 1, "total": 10}, "tests": { "total": 10 } }, "created": "2018-05-20T07:00:30.123Z", "updated": "2018-06-01T07:36:28.178Z", "createdBy": "admin", "updatedBy": "admin" } }
 ```
 
-5. You could see in the console that message will be processed
+5. You could see in the console that message will be processed, and find the following message: `Record with Challenge ID # 30051825 and Member ID # 8547899 does not exists in database. Creating the record`. Also check the leaderboard-api console for more information(Console in step 2)
 
-6. Data in the Database could be verified directly or by using the `/leaderboard` end point
+6. Attach to the topic `submission.notification.update` using Kafka console producer
 
-7. Open Postman and import the environment, collection in `docs` directory
-
-8. Trigger the `leaderboard` end point to the verify the data present in Mongo DB
+```bash
+bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.update
+```
 
-9. Attach to the topic `submission.notifcation.update` using Kafka console producer and write the below message
+7. Write the following message to the Console
 
 ```bash
-{ "topic":"submission.notification.update", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0507", "aggregateScore": 85.83 } }
+{ "topic":"submission.notification.update", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0508", "submissionId": "a34e1158-2c27-4d38-b079-5e5cca1bdcf7", "aggregateScore": 80, "scoreCardId": "b25a4180-65aa-42ec-a945-5fd21dec0503", "isPassing": true, "metadata": { "assertions": { "pending": 0, "failed": 1, "total": 5}, "tests": { "total": 5 } }, "created": "2018-05-20T07:00:30.123Z", "updated": "2018-06-01T07:36:28.178Z", "createdBy": "admin", "updatedBy": "admin" } }
 ```
 
-10. Validate the data using Postman
+8. You could see in the console that message will be processed, and find the following message: `Record with Challenge ID # 30051825 and Member ID # 8547899 exists in database. Updating the score`. Also check the leaderboard-api console for more information(Console in step 2)
 
-11. Attach to the topic `submission.notifcation.delete` using Kafka console producer and write the below message
+9. Attach to the topic `submission.notification.delete` using Kafka console producer
 
 ```bash
-{ "topic":"submission.notification.delete", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0507" } }
+bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.delete
 ```
 
-12. Validate the data using Postman
+10. Write the following message to the Console. Also check the leaderboard-api console for more information(Console in step 2)
 
-## Running unit tests and coverage
+```bash
+{ "topic":"submission.notification.delete", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0508" } }
+```
 
-To run tests, following Environment variables need to be set up
+11. You could see in the console that message will be processed
 
-- TEST_MONGODB_URL MongoDB URL pointing to Test DB instance
+### Tests
+Note: you need to stop the processor app before execute test.
 
-To run unit tests alone
+- Run the following command to execute unit test and generate coverage report
 
 ```bash
 npm run test
 ```
 
-To run unit tests with coverage report
+- Run the following command to execute e2e test and generate coverage report
 
 ```bash
-npm run cov
+npm run e2e
 ```

config/default.js

@@ -12,23 +12,20 @@ module.exports = {
   KAFKA_CLIENT_CERT: process.env.KAFKA_CLIENT_CERT,
   KAFKA_CLIENT_CERT_KEY: process.env.KAFKA_CLIENT_CERT_KEY,
 
+  KAFKA_GROUP_ID: process.env.KAFKA_GROUP_ID || 'tc-leaderboard-processor',
+
   // Kafka topics related to Creation, Update and Delete
   CREATE_DATA_TOPIC: process.env.CREATE_DATA_TOPIC || 'submission.notification.create',
   UPDATE_DATA_TOPIC: process.env.UPDATE_DATA_TOPIC || 'submission.notification.update',
   DELETE_DATA_TOPIC: process.env.DELETE_DATA_TOPIC || 'submission.notification.delete',
 
-  GROUP_IDS: process.env.GROUP_IDS || '202343,20000000', // Comma separated string of Group IDs
-
+  LEADERBOARD_API_URL: process.env.LEADERBOARD_API_URL || 'https://api.topcoder-dev.com/v5/leaderboard',
   SUBMISSION_API_URL: process.env.SUBMISSION_API_URL || 'https://api.topcoder-dev.com/v5/submissions',
-  REVIEW_SUMMATION_API_URL: process.env.REVIEW_SUMMATION_API_URL || 'https://api.topcoder-dev.com/v5/reviewSummations',
-  CHALLENGE_API_URL: process.env.CHALLENGE_API_URL || 'https://api.topcoder-dev.com/v3/challenges',
-  MEMBER_API_URL: process.env.MEMBER_API_URL || 'https://api.topcoder-dev.com/v3/users',
 
-  AUTH0_URL: process.env.AUTH0_URL, // Auth0 credentials for Leaderboard Service
+  AUTH0_URL: process.env.AUTH0_URL,
   AUTH0_AUDIENCE: process.env.AUTH0_AUDIENCE,
   TOKEN_CACHE_TIME: process.env.TOKEN_CACHE_TIME,
   AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
   AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
-  AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL,
-  MONGODB_URL: process.env.MONGODB_URL || 'mongodb://localhost:27017/leaderboardDB'
+  AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL
 }

config/test.js

@@ -1,7 +1,9 @@
 /**
- * Configuration file to be used while running tests
+ * The configuration file.
  */
 
 module.exports = {
-  MONGODB_URL: process.env.TEST_MONGODB_URL || 'mongodb://localhost:27017/leaderboardTestDB'
+  LEADERBOARD_API_URL: 'https://api.topcoder-dev.com/v5/leaderboard',
+  SUBMISSION_API_URL: 'https://api.topcoder-dev.com/v5/submissions',
+  WAIT_TIME: 1000
 }