Skip to content
This repository was archived by the owner on Mar 13, 2025. It is now read-only.

Commit 0245dd6

Browse files
callmekatootiecallmekatootie
committed
#20 - Use review payloads instead of review summation payloads
1 parent cb41f6f commit 0245dd6

File tree

6 files changed

+86
-80
lines changed

6 files changed

+86
-80
lines changed

‎README.md

Lines changed: 32 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## Dependencies
44

5-
- nodejs https://nodejs.org/en/ (v10)
5+
- Nodejs (v10)
66
- Kafka (v2)
77

88
## Configuration
@@ -35,6 +35,7 @@ The following parameters can be set in config files or in env variables:
3535
Also note that there is a `/health` endpoint that checks for the health of the app. This sets up an expressjs server and listens on the environment variable `PORT`. It's not part of the configuration file and needs to be passed as an environment variable
3636

3737
Configuration for the tests is at `config/test.js`, only add such new configurations different from `config/default.js`
38+
3839
- WAIT_TIME: wait time used in test, default is 1000 or one second
3940
- LEADERBOARD_API_URL: Leaderboard API URL used in testing
4041
- SUBMISSION_API_URL: Submission API URL used in testing
@@ -67,7 +68,7 @@ Configuration for the tests is at `config/test.js`, only add such new configurat
6768
- In the console, write some message, one message per line:
6869

6970
```bash
70-
{ "topic":"submission.notification.create", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0508", "submissionId": "a34e1158-2c27-4d38-b079-5e5cca1bdcf7", "aggregateScore": 88, "scoreCardId": "b25a4180-65aa-42ec-a945-5fd21dec0503", "isPassing": true, "created": "2018-05-20T07:00:30.123Z", "updated": "2018-06-01T07:36:28.178Z", "createdBy": "admin", "updatedBy": "admin" } }
71+
{"topic":"submission.notification.create","originator":"submission-api","timestamp":"2018-08-06T15:46:05.575Z","mime-type":"application/json","payload":{"resource":"review","id":"49871146-eb0a-4d0e-ab9a-adc94018c5da","submissionId":"6ff0c009-51ee-4c8e-aa0d-159c20503cc2","score":-1,"scoreCardId":30001852,"metadata":{"testType":"provisional","assertions":{"pending":0,"failed":1,"total":10},"tests":{"total":10}},"created":"2019-11-06T15:02:35.539Z","updated":"2019-11-06T15:02:35.539Z","createdBy":"I3etJtTqlz1XHgCXduBN1us705ufrykl@clients","updatedBy":"I3etJtTqlz1XHgCXduBN1us705ufrykl@clients","status":"completed","reviewerId":"0301619c-3d9e-44c3-85cb-c20311100f7f","typeId":"52c91e85-745f-4e62-b592-9879a2dfe9fd"}}
7172
```
7273

7374
- optionally, use another terminal, go to same directory, start a consumer to view the messages:
@@ -115,78 +116,52 @@ npm start
115116

116117
2. Refer `README.md` in `leaderboard-api` to start leaderboard api, all operations are under `leaderboard-api` project root folder
117118

118-
- Start mock app and it will listen on 3001 PORT.
119-
120-
```bash
121-
npm run mock-api
122-
```
123-
124-
- Ensure you have start MongoDB and properly configure `MONGODB_URL`. Run the following commands to clear and insert test data, step up environment variables and start the app(it will listen on 3002 PORT!).
125-
126-
```bash
127-
npm run init-db
128-
npm run test-data
129-
export CHALLENGE_API_URL=http://localhost:3001/challenges
130-
export MEMBER_API_URL=http://localhost:3001/users
131-
export PORT=3002
132-
npm start
133-
```
119+
3. Set the necessary environment variables and then run `npm start` command to start processor app(Under this project's root folder)
134120

135-
3. Run the following command to start processor app(Under this project's root folder)
136-
```bash
137-
export AUTH0_URL=https://topcoder-dev.auth0.com/oauth/token
138-
export AUTH0_AUDIENCE=https://m2m.topcoder-dev.com/
139-
export TOKEN_CACHE_TIME=90
140-
export AUTH0_CLIENT_ID=8QovDh27SrDu1XSs68m21A1NBP8isvOt
141-
export AUTH0_CLIENT_SECRET=3QVxxu20QnagdH-McWhVz0WfsQzA1F8taDdGDI4XphgpEYZPcMTF4lX3aeOIeCzh
142-
export LEADERBOARD_API_URL=http://localhost:3002/v5/leaderboard
143-
export SUBMISSION_API_URL=http://localhost:3001/submissions
144-
npm start
145-
```
121+
4. Attach to the topic `submission.notification.create` using Kafka console producer
146122

147-
3. Attach to the topic `submission.notification.create` using Kafka console producer
123+
```bash
124+
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.create
125+
```
148126

149-
```bash
150-
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.create
151-
```
127+
5. Write the following message to the Console
152128

153-
4. Write the following message to the Console
129+
```bash
130+
{"topic":"submission.notification.create","originator":"submission-api","timestamp":"2018-08-06T15:46:05.575Z","mime-type":"application/json","payload":{"resource":"review","id":"49871146-eb0a-4d0e-ab9a-adc94018c5da","submissionId":"6ff0c009-51ee-4c8e-aa0d-159c20503cc2","score":-1,"scoreCardId":30001852,"metadata":{"testType":"provisional","assertions":{"pending":0,"failed":1,"total":10},"tests":{"total":10}},"created":"2019-11-06T15:02:35.539Z","updated":"2019-11-06T15:02:35.539Z","createdBy":"I3etJtTqlz1XHgCXduBN1us705ufrykl@clients","updatedBy":"I3etJtTqlz1XHgCXduBN1us705ufrykl@clients","status":"completed","reviewerId":"0301619c-3d9e-44c3-85cb-c20311100f7f","typeId":"52c91e85-745f-4e62-b592-9879a2dfe9fd"}}
131+
```
154132

155-
```bash
156-
{ "topic":"submission.notification.create", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0508", "submissionId": "a34e1158-2c27-4d38-b079-5e5cca1bdcf7", "aggregateScore": 90, "scoreCardId": "b25a4180-65aa-42ec-a945-5fd21dec0503", "isPassing": true, "metadata": { "assertions": { "pending": 0, "failed": 1, "total": 10}, "tests": { "total": 10 } }, "created": "2018-05-20T07:00:30.123Z", "updated": "2018-06-01T07:36:28.178Z", "createdBy": "admin", "updatedBy": "admin" } }
157-
```
133+
6. You could see in the console that message will be processed, and find the following message: `Record with Challenge ID # 30051825 and Member ID # 8547899 does not exists in database. Creating the record`. Also check the leaderboard-api console for more information(Console in step 2)
158134

159-
5. You could see in the console that message will be processed, and find the following message: `Record with Challenge ID # 30051825 and Member ID # 8547899 does not exists in database. Creating the record`. Also check the leaderboard-api console for more information(Console in step 2)
135+
7. Attach to the topic `submission.notification.update` using Kafka console producer
160136

161-
6. Attach to the topic `submission.notification.update` using Kafka console producer
137+
```bash
138+
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.update
139+
```
162140

163-
```bash
164-
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.update
165-
```
141+
8. Write the following message to the Console
166142

167-
7. Write the following message to the Console
143+
```bash
144+
{"topic":"submission.notification.update","originator":"submission-api","timestamp":"2018-08-06T15:46:05.575Z","mime-type":"application/json","payload":{"resource":"review","id":"49871146-eb0a-4d0e-ab9a-adc94018c5da","submissionId":"6ff0c009-51ee-4c8e-aa0d-159c20503cc2","score":52,"scoreCardId":30001852,"metadata":{"testType":"provisional","assertions":{"pending":0,"failed":2,"total":10},"tests":{"total":10}},"created":"2019-11-06T15:02:35.539Z","updated":"2019-11-06T15:02:35.539Z","createdBy":"I3etJtTqlz1XHgCXduBN1us705ufrykl@clients","updatedBy":"I3etJtTqlz1XHgCXduBN1us705ufrykl@clients","status":"completed","reviewerId":"0301619c-3d9e-44c3-85cb-c20311100f7f","typeId":"52c91e85-745f-4e62-b592-9879a2dfe9fd"}}
145+
```
168146

169-
```bash
170-
{ "topic":"submission.notification.update", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0508", "submissionId": "a34e1158-2c27-4d38-b079-5e5cca1bdcf7", "aggregateScore": 80, "scoreCardId": "b25a4180-65aa-42ec-a945-5fd21dec0503", "isPassing": true, "metadata": { "assertions": { "pending": 0, "failed": 1, "total": 5}, "tests": { "total": 5 } }, "created": "2018-05-20T07:00:30.123Z", "updated": "2018-06-01T07:36:28.178Z", "createdBy": "admin", "updatedBy": "admin" } }
171-
```
147+
9. You could see in the console that message will be processed, and find the following message: `Record with Challenge ID # 30051825 and Member ID # 8547899 exists in database. Updating the score`. Also check the leaderboard-api console for more information(Console in step 2)
172148

173-
8. You could see in the console that message will be processed, and find the following message: `Record with Challenge ID # 30051825 and Member ID # 8547899 exists in database. Updating the score`. Also check the leaderboard-api console for more information(Console in step 2)
149+
10. Attach to the topic `submission.notification.delete` using Kafka console producer
174150

175-
9. Attach to the topic `submission.notification.delete` using Kafka console producer
151+
```bash
152+
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.delete
153+
```
176154

177-
```bash
178-
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.delete
179-
```
155+
11. Write the following message to the Console. Also check the leaderboard-api console for more information(Console in step 2)
180156

181-
10. Write the following message to the Console. Also check the leaderboard-api console for more information(Console in step 2)
157+
```bash
158+
{ "topic":"submission.notification.delete", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"review", "id": "49871146-eb0a-4d0e-ab9a-adc94018c5da" } }
159+
```
182160
183-
```bash
184-
{ "topic":"submission.notification.delete", "originator":"submission-api", "timestamp":"2018-08-06T15:46:05.575Z", "mime-type":"application/json", "payload":{ "resource":"reviewSummation", "id": "d24d4180-65aa-42ec-a945-5fd21dec0508" } }
185-
```
161+
12. You could see in the console that message will be processed
186162
187-
11. You could see in the console that message will be processed
163+
## Tests
188164
189-
### Tests
190165
Note: you need to stop the processor app before execute test.
191166
192167
- Run the following command to execute unit test and generate coverage report

‎src/app.js

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@ const dataHandler = (messageSet, topic, partition) => Promise.each(messageSet, (
4040
return
4141
}
4242

43-
if (messageJSON.payload.resource !== 'reviewSummation') {
44-
logger.debug(`Ignoring Non review Summation payloads from topic ${messageJSON.topic}.`)
43+
if (messageJSON.payload.resource !== 'review') {
44+
logger.debug(`Ignoring Non review payloads from topic ${messageJSON.topic}.`)
4545
// ignore the message
4646
return
4747
}

‎src/services/ProcessorService.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ upsert.schema = {
4242
* @param {Object} message the Kafka message in JSON format
4343
*/
4444
async function remove (message) {
45-
await helper.reqToAPI('DELETE', `${config.LEADERBOARD_API_URL}/reviewSummation/${message.payload.id}`)
45+
await helper.reqToAPI('DELETE', `${config.LEADERBOARD_API_URL}/review/${message.payload.id}`)
4646
}
4747

4848
remove.schema = {

‎test/common/prepare.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ prepare(function (done) {
2525
.reply(200)
2626
.patch('/v5/leaderboard/challenge/30051825/member/22688726')
2727
.reply(200)
28-
.delete('/v5/leaderboard/reviewSummation/d24d4180-65aa-42ec-a945-5fd21dec0510')
28+
.delete('/v5/leaderboard/review/49871146-eb0a-4d0e-ab9a-adc94018c5da')
2929
.reply(204)
3030

3131
done()

‎test/common/testData.js

Lines changed: 49 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -9,19 +9,32 @@ const testTopics = {
99
testMessage: {
1010
topic: 'submission.notification.create',
1111
originator: 'submission-api',
12-
timestamp: '2018-02-03T00:00:00',
12+
timestamp: '2018-08-06T15:46:05.575Z',
1313
'mime-type': 'application/json',
1414
payload: {
15-
resource: 'reviewSummation',
16-
id: 'd24d4180-65aa-42ec-a945-5fd21dec0508',
15+
resource: 'review',
16+
id: '49871146-eb0a-4d0e-ab9a-adc94018c5da',
1717
submissionId: 'a34e1158-2c27-4d38-b079-5e5cca1bdcf7',
18-
aggregateScore: 88,
19-
scoreCardId: 'b25a4180-65aa-42ec-a945-5fd21dec0503',
20-
isPassing: true,
21-
created: '2018-01-01T00:00:00',
22-
updated: '2018-01-02T00:00:00',
23-
createdBy: 'admin',
24-
updatedBy: 'user'
18+
score: -1,
19+
scoreCardId: 30001852,
20+
metadata: {
21+
testType: 'provisional',
22+
assertions: {
23+
pending: 0,
24+
failed: 1,
25+
total: 10
26+
},
27+
tests: {
28+
total: 10
29+
}
30+
},
31+
created: '2019-11-06T15:02:35.539Z',
32+
updated: '2019-11-06T15:02:35.539Z',
33+
createdBy: 'I3etJtTqlz1XHgCXduBN1us705ufrykl@clients',
34+
updatedBy: 'I3etJtTqlz1XHgCXduBN1us705ufrykl@clients',
35+
status: 'completed',
36+
reviewerId: '0301619c-3d9e-44c3-85cb-c20311100f7f',
37+
typeId: '52c91e85-745f-4e62-b592-9879a2dfe9fd'
2538
}
2639
}
2740
},
@@ -31,14 +44,32 @@ const testTopics = {
3144
testMessage: {
3245
topic: 'submission.notification.update',
3346
originator: 'submission-api',
34-
timestamp: '2018-02-03T00:00:00',
47+
timestamp: '2018-08-06T15:46:05.575Z',
3548
'mime-type': 'application/json',
3649
payload: {
37-
resource: 'reviewSummation',
38-
id: 'd24d4180-65aa-42ec-a945-5fd21dec0509',
50+
resource: 'review',
51+
id: '49871146-eb0a-4d0e-ab9a-adc94018c5da',
3952
submissionId: 'a34e1158-2c27-4d38-b079-5e5cca1bdcf8',
40-
aggregateScore: 93.5,
41-
isPassing: true
53+
score: 52,
54+
scoreCardId: 30001852,
55+
metadata: {
56+
testType: 'provisional',
57+
assertions: {
58+
pending: 0,
59+
failed: 2,
60+
total: 10
61+
},
62+
tests: {
63+
total: 10
64+
}
65+
},
66+
created: '2019-11-06T15:02:35.539Z',
67+
updated: '2019-11-06T15:02:35.539Z',
68+
createdBy: 'I3etJtTqlz1XHgCXduBN1us705ufrykl@clients',
69+
updatedBy: 'I3etJtTqlz1XHgCXduBN1us705ufrykl@clients',
70+
status: 'completed',
71+
reviewerId: '0301619c-3d9e-44c3-85cb-c20311100f7f',
72+
typeId: '52c91e85-745f-4e62-b592-9879a2dfe9fd'
4273
}
4374
}
4475
},
@@ -51,8 +82,8 @@ const testTopics = {
5182
timestamp: '2018-02-03T00:00:00',
5283
'mime-type': 'application/json',
5384
payload: {
54-
resource: 'reviewSummation',
55-
id: 'd24d4180-65aa-42ec-a945-5fd21dec0510'
85+
resource: 'review',
86+
id: '49871146-eb0a-4d0e-ab9a-adc94018c5da'
5687
}
5788
}
5889
}
@@ -79,7 +110,7 @@ const submissionAPIResponse = [
79110
'created': '2018-08-22T02:41:07.576Z',
80111
'isFileSubmission': true,
81112
'type': 'challengesubmission',
82-
'url': 'https://topcoder-dev-submissions-dmz.s3.amazonaws.com/a34e1158-2c27-4d38-b079-%3Cspan%20class="x x-first x-last">5e5cca1bdcf7',
113+
'url': 'https://topcoder-dev-submissions-dmz.s3.amazonaws.com/a34e1158-2c27-4d38-b079-%3Cspan%20class="x x-first x-last">5e5cca1bdcf8',
83114
'challengeId': 30051825,
84115
'filename': 'CORE-TopcoderEventBus-190618-1413.pdf.zip',
85116
'createdBy': 'lazybaer',

‎test/e2e/test.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,7 @@ describe('Topcoder - Leaderboard Processor E2E Test', () => {
224224
await waitJob()
225225

226226
expect(errorLogs.length).to.equal(0)
227-
expect(debugLogs[debugLogs.length - 1]).to.equal(`Ignoring Non review Summation payloads from topic ${message.topic}.`)
227+
expect(debugLogs[debugLogs.length - 1]).to.equal(`Ignoring Non review payloads from topic ${message.topic}.`)
228228
})
229229

230230
for (const op of ['create', 'update', 'delete']) {

0 commit comments

Comments
 (0)