DynamoDB updates

Author: jmgasper

TopcoderXDeploy.md

@@ -1,6 +1,6 @@
 # Topcoder X overview
 
-Topcoder-X is a set of applications and services that allow a copilot or Topcoder customer to manage work directly through Gitlab or Github.  When an issue is created in a Gitlab or Github project set up in Topcoder-X, Topcoder-X will create a Topcoder challenge to mirror the Gitlab or Github issue, and it will ensure that the challenge has the correct prize, copilot, assignee, description, and title.  As the Gitlab or Github issue is updated, Topcoder-X will ensure that the Topcoder challenge associated with the issue is updated appropriately.  When the Gitlab or Github issue is closed, Topcoder-X will activate and close the Topcoder challenge, ensuring that the members get paid as expected.  
+Topcoder-X is a set of applications and services that allow a copilot or Topcoder customer to manage work directly through Gitlab or Github.  When an issue is created in a Gitlab or Github project set up in Topcoder-X, Topcoder-X will create a Topcoder challenge to mirror the Gitlab or Github issue, and it will ensure that the challenge has the correct prize, copilot, assignee, description, and title.  As the Gitlab or Github issue is updated, Topcoder-X will ensure that the Topcoder challenge associated with the issue is updated appropriately.  When the Gitlab or Github issue is closed, Topcoder-X will activate and close the Topcoder challenge, ensuring that the members get paid as expected.
 
 At each step of the process, Topcoder-X will add comments to the Gitlab or Github project, ensuring that the members know where the Topcoder challenge is and what the status of the challenge is.
 
@@ -10,7 +10,7 @@ The information is updated in real time based on webhook integrations with Gitla
 
 ## Dependencies
 * NodeJS 8+
-* MongoDB 3.2
+* DynamoDB
 * Kafka
 * nodemon (for local development)
 
@@ -25,18 +25,21 @@ Topcoder-X comprises 3 pieces:
 All 3 pieces will be configured to use the same MongoDB and Kafka installations.
 
 
-## MongoDB
+## DynamoDB
 
-The MongoDB can be created using default options.  Just make sure that it is configured properly as `MONGODB_URI` in all 3 pieces:
+The DynamoDB can be created using default options.  Just make sure that it is configured properly as `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `IS_LOCAL` in all 3 pieces:
 
 * Topcoder-X processor
 * Topcoder-X receiver
 * Topcoder-X UI
 
-Sample from our development environment:
-
-`MONGODB_URI: mongodb://heroku_k366sw2n:<password>@ds135552.mlab.com:35552/heroku_k366sw2n`
-
+Sample from our local development environment:
+```
+AWS_ACCESS_KEY_ID: 'FAKE_ACCESS_KEY_ID'
+AWS_SECRET_ACCESS_KEY: 'FAKE_SECRET_ACCESS_KEY'
+AWS_REGION: 'FAKE_REGION'
+IS_LOCAL: true
+```
 ## Kafka
 
 Installing Kafka can be done either locally or using a cloud service.  You'll need to note how the service is configured and will have to update the configuration appropriately for the receiver, processor, and site.
@@ -87,7 +90,7 @@ To this:
 
 For login to work, your local Topcoder-X-UI deployment needs to have a `*.topcoder-dev.com` DNS name.  Our development environment uses `x.topcoder-dev.com`
 
-You can make this change in your local `/etc/hosts` file.  
+You can make this change in your local `/etc/hosts` file.
 
 ```
 127.0.0.1   x.topcoder-dev.com
@@ -100,7 +103,7 @@ You can login with one of these sample accounts:
 
 ## Local webhook setup
 
-The hardest part of the setup may be ensuring that Gitlab and Github can make callbacks to your local environment.  You will have to ensure that your Topcoder-X receiver is publicly accessible on the public internet.  
+The hardest part of the setup may be ensuring that Gitlab and Github can make callbacks to your local environment.  You will have to ensure that your Topcoder-X receiver is publicly accessible on the public internet.
 
 If your ISP dynamically configures your IP address, you can use a dyndns service:
 
@@ -118,7 +121,7 @@ http://<publicly accessible domain name>:<port>/webhooks/gitlab
 #### Github
 
 ```
-https://<publicly accessible domain name>:<port>/webhooks/github  
+https://<publicly accessible domain name>:<port>/webhooks/github
 ```
 
 ## Account setup
@@ -153,7 +156,7 @@ Contest https://www.topcoder-dev.com/challenges/30052039 has been updated - it h
 
 ## Sample development configs
 
-For reference, this is what the sample configs look like in our development environment, which should closely match your local deployment environment. 
+For reference, this is what the sample configs look like in our development environment, which should closely match your local deployment environment.
 
 You can use the Gitlab and Github keys and secrets below, but you are also welcome to create your own.
 
@@ -174,12 +177,15 @@ MAILGUN_SMTP_LOGIN:           [email protected]
 MAILGUN_SMTP_PASSWORD:        c8aefb446e76febdbc31d57ef30b9c10
 MAILGUN_SMTP_PORT:            587
 MAILGUN_SMTP_SERVER:          smtp.mailgun.org
-MONGODB_URI:                  mongodb://heroku_k366sw2n:<password>@ds135552.mlab.com:35552/heroku_k366sw2n
 NODE_DEBUG:                   app
 NODE_ENV:                     development
 NODE_MODULES_CACHE:           false
 NODE_TLS_REJECT_UNAUTHORIZED: 0
 TOPIC:                        topcoder-x
+AWS_ACCESS_KEY_ID:  FAKE_ACCESS_KEY_ID
+AWS_SECRET_ACCESS_KEY: FAKE_SECRET_ACCESS_KEY
+AWS_REGION: FAKE_REGION
+IS_LOCAL: true
 ```
 
 #### Topcoder-X receiver
@@ -191,10 +197,13 @@ KAFKA_CLIENT_CERT: <cert>
 KAFKA_CLIENT_CERT_KEY: <key>
 KAFKA_HOST:                   silver-craft-01.srvs.cloudkafka.com:9093,silver-craft-01.srvs.cloudkafka.com:9094
 LOG_LEVEL:                    debug
-MONGODB_URI:                  mongodb://heroku_k366sw2n:<password>@ds135552.mlab.com:35552/heroku_k366sw2n
 NODE_ENV:                     development
 NODE_TLS_REJECT_UNAUTHORIZED: 0
 TOPIC:                        topcoder-x
+AWS_ACCESS_KEY_ID:  FAKE_ACCESS_KEY_ID
+AWS_SECRET_ACCESS_KEY: FAKE_SECRET_ACCESS_KEY
+AWS_REGION: FAKE_REGION
+IS_LOCAL: true
 ```
 
 #### Topcoder-X UI
@@ -211,11 +220,14 @@ HOOK_BASE_URL:         https://topcoder-x-receiver-dev.herokuapp.com
 KAFKA_CLIENT_CERT:  <cert>
 KAFKA_CLIENT_CERT_KEY: <key>
 KAFKA_HOST:            silver-craft-01.srvs.cloudkafka.com:9093,silver-craft-01.srvs.cloudkafka.com:9094
-MONGODB_URI:           mongodb://heroku_k366sw2n:<password>@ds135552.mlab.com:35552/heroku_k366sw2n
 NPM_CONFIG_PRODUCTION: false
 SESSION_SECRET:        kjsdfkj34857
 TC_LOGIN_URL:          https://accounts.topcoder-dev.com/member
 TC_USER_PROFILE_URL:   http://api.topcoder-dev.com/v2/user/profile
 TOPIC:                 topcoder-x
 WEBSITE:               https://x.topcoder-dev.com
-```
+AWS_ACCESS_KEY_ID:  FAKE_ACCESS_KEY_ID
+AWS_SECRET_ACCESS_KEY: FAKE_SECRET_ACCESS_KEY
+AWS_REGION: FAKE_REGION
+IS_LOCAL: true
+```

configuration.md

@@ -15,7 +15,6 @@ The following config parameters are supported, they are defined in `src/config.j
 | GITLAB_CLIENT_SECRET                   | the GitLab client secret                   | No default - needs to be set up using the instructions below                                                             |
 | WEBSITE                                | used as base to construct various URLs     | http://topcoderx.topcoder-dev.com/ |
 | GITLAB_API_BASE_URL                    | The Gitlab API base URL                    | https://gitlab.com|
-| MONGODB_URI                            | The MongoDB URI.  This needs to be the same MongoDB used by topcoder-x-receiver, topcoder-x-processor, and topcoder-x-site                           | mongodb://127.0.0.1:27017/topcoderx |
 |TOPIC  | The Kafka topic where events are published.  This must be the same as the configured value for topcoder-x-processor| |
 |KAFKA_OPTIONS | Kafka connection options| |
 |KAFKA_HOST | The Kafka host to connect to| localhost:9092 |
@@ -28,6 +27,10 @@ The following config parameters are supported, they are defined in `src/config.j
 |COPILOT_ROLE| The role to identify copilot|'copilot'|
 |HELP_LINK| The link for help| 'https://github.com/topcoder-platform/topcoder-x-ui/wiki'|
 |ADMINISTRATOR_ROLES| The array of roles to be considered as admin| `['administrator', 'admin']`|
+|AWS_ACCESS_KEY_ID | The Amazon certificate key to use when connecting. Use local dynamodb you can set fake value|FAKE_ACCESS_KEY_ID |
+|AWS_SECRET_ACCESS_KEY | The Amazon certificate access key to use when connecting. Use local dynamodb you can set fake value|FAKE_SECRET_ACCESS_KEY |
+|AWS_REGION | The Amazon certificate region to use when connecting. Use local dynamodb you can set fake value|FAKE_REGION |
+|IS_LOCAL | Use Amazon DynamoDB Local or server. |true |
 
 ## GitHub OAuth App Setup
 

package.json

@@ -49,6 +49,7 @@
         "core-js": "^2.4.1",
         "cors": "^2.8.4",
         "debug": "~2.6.3",
+        "dynamoose": "^1.1.0",
         "express": "^4.15.4",
         "express-jwt": "^5.3.0",
         "express-session": "^1.15.5",
@@ -66,7 +67,7 @@
         "jwt-decode": "^2.2.0",
         "lodash": "^4.17.4",
         "metismenu": "~2.0.2",
-        "mongoose": "^4.11.9",
+        "moment": "^2.22.2",
         "no-kafka": "^3.2.10",
         "octonode": "^0.9.2",
         "pace-js": "~1.0.2",
@@ -75,7 +76,7 @@
         "superagent": "^3.6.0",
         "superagent-promise": "^1.1.0",
         "typescript": "~2.3.3",
-        "uuid": "^3.1.0",
+        "uuid": "^3.3.2",
         "winston": "^2.3.1"
     },
     "devDependencies": {

src/common/db-helper.js

@@ -0,0 +1,131 @@
+/*
+ * Copyright (c) 2018 TopCoder, Inc. All rights reserved.
+ */
+/**
+ * This module contains the database helper methods.
+ *
+ * @version 1.0
+ */
+
+/**
+ * Get Data by model id
+ * @param {Object} model The dynamoose model to query
+ * @param {String} id The id value
+ * @returns {Promise<void>}
+ */
+async function getById(model, id) {
+  return await new Promise((resolve, reject) => {
+    model.query('id').eq(id).exec((err, result) => {
+      if (err) {
+        reject(err);
+      }
+
+      return resolve(result[0]);
+    });
+  });
+}
+
+/**
+ * Get data collection by scan parameters
+ * @param {Object} model The dynamoose model to scan
+ * @param {Object} scanParams The scan parameters object
+ * @returns {Promise<void>}
+ */
+async function scan(model, scanParams) {
+  return await new Promise((resolve, reject) => {
+    model.scan(scanParams).exec((err, result) => {
+      if (err) {
+        reject(err);
+      }
+
+      return resolve(result.count === 0 ? [] : result);
+    });
+  });
+}
+
+/**
+ * Get single data by scan parameters
+ * @param {Object} model The dynamoose model to scan
+ * @param {Object} scanParams The scan parameters object
+ * @returns {Promise<void>}
+ */
+async function scanOne(model, scanParams) {
+  return await new Promise((resolve, reject) => {
+    model.scan(scanParams).exec((err, result) => {
+      if (err) {
+        reject(err);
+      }
+
+      return resolve(result.count === 0 ? null : result[0]);
+    });
+  });
+}
+
+/**
+ * Create item in database
+ * @param {Object} Model The dynamoose model to query
+ * @param {Object} data The create data object
+ * @returns {Promise<void>}
+ */
+async function create(Model, data) {
+  return await new Promise((resolve, reject) => {
+    const dbItem = new Model(data);
+    dbItem.save((err) => {
+      if (err) {
+        reject(err);
+      }
+
+      return resolve(dbItem);
+    });
+  });
+}
+
+/**
+ * Update item in database
+ * @param {Object} Model The dynamoose model to update
+ * @param {String} id The id of item
+ * @param {Object} data The updated data object
+ * @returns {Promise<void>}
+ */
+async function update(Model, id, data) {
+  const dbItem = await getById(Model, id);
+  Object.keys(data).forEach((key) => {
+    dbItem[key] = data[key];
+  });
+  return await new Promise((resolve, reject) => {
+    dbItem.save((err) => {
+      if (err) {
+        reject(err);
+      }
+
+      return resolve(dbItem);
+    });
+  });
+}
+
+/**
+ * Delete item in database
+ * @param {Object} Model The dynamoose model to delete
+ * @param {Object} queryParams The query parameters object
+ */
+async function remove(Model, queryParams) {
+  const dbItem = await scanOne(Model, queryParams);
+  await new Promise((resolve, reject) => {
+    dbItem.delete((err) => {
+      if (err) {
+        reject(err);
+      }
+
+      resolve(dbItem);
+    });
+  });
+}
+
+module.exports = {
+  getById,
+  scan,
+  scanOne,
+  create,
+  update,
+  remove,
+};

src/common/helper.js

@@ -8,12 +8,13 @@
  * @author TCSCODER
  * @version 1.0
  */
+
 const util = require('util');
 const _ = require('lodash');
+const uuid = require('uuid/v4');
 const Joi = require('joi');
 const getParams = require('get-parameter-names');
 const bluebird = require('bluebird');
-const uuid = require('uuid/v4');
 const bcrypt = require('bcryptjs');
 const parseDomain = require('parse-domain');
 const config = require('../config');
@@ -22,6 +23,7 @@ const errors = require('./errors');
 const constants = require('./constants');
 const NotFoundError = require('./errors').NotFoundError;
 const ValidationError = require('./errors').ValidationError;
+const dbHelper = require('./db-helper');
 
 bluebird.promisifyAll(bcrypt);
 bluebird.promisifyAll(parseDomain);
@@ -85,7 +87,7 @@ function _decorateWithValidators(service) {
     const params = getParams(method);
     service[name] = async function serviceMethodWithValidation(...args) {
       const value = _combineObject(params, args);
-      const normalized = Joi.attempt(value, method.schema, { abortEarly: false });
+      const normalized = Joi.attempt(value, method.schema, {abortEarly: false});
       // Joi will normalize values
       // for example string number '1' to 1
       // if schema type is number
@@ -167,24 +169,25 @@ function convertGitLabError(err, message) {
  * Ensure entity exists for given criteria. Return error if no result.
  * @param {Object} Model the mongoose model to query
  * @param {Object|String|Number} criteria the criteria (if object) or id (if string/number)
+ * @param {String} modelName the name of model
  * @returns {Object} the found entity
  */
-async function ensureExists(Model, criteria) {
+async function ensureExists(Model, criteria, modelName) {
   let query;
   let byId = true;
   if (_.isObject(criteria)) {
     byId = false;
-    query = Model.findOne(criteria);
+    query = dbHelper.scanOne(Model, criteria);
   } else {
-    query = Model.findById(criteria);
+    query = dbHelper.getById(Model, criteria);
   }
   const result = await query;
   if (!result) {
     let msg;
     if (byId) {
-      msg = util.format('%s not found with id: %s', Model.modelName, criteria);
+      msg = util.format('%s not found with id: %s', modelName, criteria);
     } else {
-      msg = util.format('%s not found with criteria: %j', Model.modelName, criteria);
+      msg = util.format('%s not found with criteria: %j', modelName, criteria);
     }
     throw new NotFoundError(msg);
   }
@@ -212,19 +215,18 @@ async function getProviderType(repoUrl) {
  * @returns {Object} the owner/copilot for the project
  */
 async function getProjectCopilot(models, project, provider) {
-  const userMapping = await models.UserMapping.findOne({
+  const userMapping = await dbHelper.scanOne(models.UserMapping, {
     topcoderUsername: project.copilot,
   });
 
   if (!userMapping || (provider === 'github' && !userMapping.githubUserId) || (provider === 'gitlab' && !userMapping.gitlabUserId)) {
     throw new Error(`Couldn't find owner username for '${provider}' for this repository.`);
   }
 
-  const copilot = await models.User.findOne({
+  return await dbHelper.scanOne(models.User, {
     username: provider === 'github' ? userMapping.githubUsername : userMapping.gitlabUsername,
     type: provider,
   });
-  return copilot;
 }