#1 - Update code base to sync user skills b/w member api and u-bahn

Author: callmekatootie

.gitignore

@@ -0,0 +1,122 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+
+# Diagnostic reports (https://nodejs.org/api/report.html)
+report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+*.lcov
+
+# nyc test coverage
+.nyc_output
+
+# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+bower_components
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (https://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# Snowpack dependency directory (https://snowpack.dev/)
+web_modules/
+
+# TypeScript cache
+*.tsbuildinfo
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Microbundle cache
+.rpt2_cache/
+.rts2_cache_cjs/
+.rts2_cache_es/
+.rts2_cache_umd/
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variables file
+.env
+.env.test
+
+# parcel-bundler cache (https://parceljs.org/)
+.cache
+.parcel-cache
+
+# Next.js build output
+.next
+out
+
+# Nuxt.js build / generate output
+.nuxt
+dist
+
+# Gatsby files
+.cache/
+# Comment in the public line in if your project uses Gatsby and not Next.js
+# https://nextjs.org/blog/next-9-1#public-directory-support
+# public
+
+# vuepress build output
+.vuepress/dist
+
+# Serverless directories
+.serverless/
+
+# FuseBox cache
+.fusebox/
+
+# DynamoDB Local files
+.dynamodb/
+
+# TernJS port file
+.tern-port
+
+# Stores VSCode versions used for testing VSCode extensions
+.vscode-test
+
+# yarn v2
+.yarn/cache
+.yarn/unplugged
+.yarn/build-state.yml
+.yarn/install-state.gz
+.pnp.*
+
+# api.env
+api.env
+
+# macOS files
+.DS_Store

.nvmrc

@@ -1,78 +1,120 @@
 # member-skills-sync-lambda
 
 ## Prerequisites
-1. Amazon DynamoDB table for the streaming and sending member skills data to Amazon Lambda exists and the ARN is known (please enable the streaming on DynamoDB with View type "New and old images" - see screen-shot in Verification Document if required).
-2. Amazon ElasticSearch domain for posting the member skills data already exists and endpoint is known.
-3. AWS Lambda IAM role that has the privilege (permission policies) to read the stream from DynamodDB already exists and ARN is known.
-4. Amazon ElasticSearch domain access policy has been configured so that AWS Lambda IAM role can do necessary action to post data to ElasticSearch and for unit test purpose, either using IP address or Serverless Agent IAM User below is given access to post data to ElasticSearch.
-5. Serverless Framework has been installed. If not yet done, please follow this [link](https://serverless.com/framework/docs/providers/aws/guide/installation/).
-6. Serverless Agent IAM User has been configured properly and its access key ID and its agent secret access key is known, please follow this [link](https://serverless.com/framework/docs/providers/aws/guide/credentials/) to setup one if not yet done.
-7. All the instructions below are using Node v10.16.3.
-
-## Environment Variables
-Please follow this instruction to setup the required environment variables. We are using Linux terminal in this example.
-
-First you need to setup environment variables for AWS credentials. For review purpose, actually you don't need to create a new user for severless agent, just use your AWS account's access key and secret access key which already have full-access to all AWS services.
-```
-export AWS_ACCESS_KEY_ID=<your access key id>
-export AWS_SECRET_ACCESS_KEY=<your secret access key>
-```
-For the sake of unit testing to run, running your lambda function locally, and deploying your lambda's environment variables, please also setup 9 more environment variables below in your terminal (using export command same as above).
-1. ES_URL                         --> Endpoint of your elastic search
-2. ES_REGION                      --> Region of your elastic search e.g. us-east-1
-3. ES_SKILLS_INDEX                --> ElasticSearch Index for member entered skills documents
-4. ES_SKILLS_MAPPING              --> ElasticSearch document type for member entered skills index
-7. DB_ENTERED_SKILLS_STREAM       --> ARN of DynamoDB table MemberEnteredSkills stream
-8. DB_AGGREGATED_SKILLS_STREAM    --> ARE of DynamoDB table MemberAggregatedSkills stream
-9. LAMBDA_ROLE                    --> ARN of IAM Role to be used when executing Lambda
-
-## Change the serverless.yml to fit your environment
-Please open serverless.yml from the project and modify following value in case you don't want to use environment variables:
-1. role (under index.syncSkills)    --> ARN of your AWS Lambda IAM Role
-2. arn (under stream)               --> ARN of your DynamoDB table stream
-3. ES_* (under environment)         --> This is by default populated by environment variables, feel free to change if needed
-4. DB_* (under environment)         --> This is by default populated by environment variables, feel free to change if needed
-5. Please also modify stage and region to fit your needs
-
-## Install Dependencies and Run Lint
-1. run the following command to install the dependencies
-```
+
+- Nodejs(v12+)
+
+## Configuration
+
+Configuration for the skill record processor is at `config/default.js`.
+The following parameters can be set in config files or in env variables:
+
+- UBAHN_API_URL: The ubahn api url, default value: 'https://api.topcoder-dev.com/v5'
+- TC_API_URL: The topcoder api url, default value: 'https://api.topcoder-dev.com'
+- AUTH0_URL: The auth0 url, default value: 'https://topcoder-dev.auth0.com/oauth/token'
+- AUTH0_UBAHN_AUDIENCE: The auth0 audience for accessing ubahn api(s), default value: 'https://u-bahn.topcoder-dev.com'
+- AUTH0_TOPCODER_AUDIENCE: The auth0 audience for accessing tc api(s), default value: 'https://m2m.topcoder-dev.com/'
+- AUTH0_CLIENT_ID: The auth0 client id
+- AUTH0_CLIENT_SECRET: The auth0 client secret
+- VALID_ISSUERS: The valid issuer of tokens, a json array contains valid issuer.
+- TOKEN_CACHE_TIME: Auth0 token cache time, used to get TC M2M token
+- AUTH0_PROXY_SERVER_URL: Proxy Auth0 URL, used to get TC M2M token
+- SKILL_PROVIDER_NAME: The skill provider name, default value: Topcoder
+- SLEEP_TIME: The pause time between two create operations, default value: 1000 ms
+
+## Local Test
+
+- install the dependencies
+
+``` bash
 npm install
 ```
-2. To run linter if required
-```
-npm run lint
 
+- run linters if required
+
+``` bash
+npm run lint
 npm run lint:fix # To fix possible lint errors
 ```
 
-## Unit Test
-Make sure you have properly setup the [environment variables](#environment-variables) and [installing dependencies](#install-dependencies-and-run-lint). Run `npm run test` to run the unit test.
+- run test
 
-## Run lambda locally
-1. Make sure you have properly setup the [environment variables](#environment-variables) and [installing dependencies](#install-dependencies-and-run-lint).
-
-2. Run command `npm run test-data` to generate test data.(Important step!! The provided test data have different eventSourceARN which are different with your environment variables. The lambda function need to use eventSourceARN to distinguish ES index.)
-
-3. Then, Run the following to invoke lambda locally.
-```
-serverless invoke local -f sync -p test_data/createEnteredSkillsEvent.json
-serverless invoke local -f sync -p test_data/updateEnteredSkillsEvent.json
-serverless invoke local -f sync -p test_data/deleteEnteredSkillsEvent.json
-serverless invoke local -f sync -p test_data/createMemberAggregatedSkillsEvent.json
-serverless invoke local -f sync -p test_data/updateMemberAggregatedSkillsEvent.json
-serverless invoke local -f sync -p test_data/deleteMemberAggregatedSkillsEvent.json
+``` bash
+npm test
 ```
-Please note that even if the lambda is invoked locally using test data, the data is still being posted to ElasticSearch in AWS based on environment variables above. In order for the local invocation to work properly, the serverless agent that is used by serverless command has to have privilege to post the data to ElasticSearch or the access policy for the ElasticSearch is configured to allow any AWS principal to post the data but restricted by condition e.g. by using IpAddress.
 
 ## Deploy
-When all the things mentioned above have been done, please run the following:
-```
-serverless deploy
-```
-Feel free to use flag `--verbose` after `serverless deploy` if you want to know more information about your deployment.
 
-You can also modify the lambda code if required and the redeployment is as simple as running `serverless deploy` again.
+- make sure you have installed the dependencies
+- make sure you have aws credentials in you environment
+- update the `.env` file, set `AUTH0_CLIENT_ID` and `AUTH0_CLIENT_SECRET` and other configurations that you want to change
+
+``` bash
+npm run deploy
+```
 
 ## Verification
-Please read verification document for screen shot of enabling DynamoDB table stream, unit test result, deployment results, and results of manual testing on the lambda.
+
+After deploy, you can open [aws lambda config](https://console.aws.amazon.com/lambda/home?region=us-east-1#/functions/member-v5user-skill-sync?tab=testing) to test, here is the example event json `{
+  "Records":[
+    {
+      "eventID":"904156e21aa4f1085f970bfdbcbfb483",
+      "eventName":"MODIFY",
+      "eventVersion":"1.1",
+      "eventSource":"aws:dynamodb",
+      "awsRegion":"us-east-1",
+      "dynamodb":{
+        "ApproximateCreationDateTime":1618587935,
+        "Keys":{
+          "userId":{
+            "N":"40153932"
+          }
+        },
+        "NewImage":{
+          "skills":{
+            "S":"{\"110\":{\"tagName\":null,\"hidden\":false,\"score\":1.0,\"sources\":[]},\"117\":{\"tagName\":null,\"hidden\":false,\"score\":1.0,\"sources\":[]}}"
+          },
+          "userHandle":{
+            "S":"sachin-kumar"
+          },
+          "updatedBy":{
+            "S":"40153932"
+          },
+          "userId":{
+            "N":"40153932"
+          },
+          "handleLower":{
+            "S":"sachin-kumar"
+          },
+          "updatedAt":{
+            "N":"1618587934734"
+          }
+        },
+        "OldImage":{
+          "skills":{
+            "S":"{\"110\":{\"tagName\":null,\"hidden\":false,\"score\":1.0,\"sources\":[]}}"
+          },
+          "userHandle":{
+            "S":"sachin-kumar"
+          },
+          "updatedBy":{
+            "S":"40153932"
+          },
+          "userId":{
+            "N":"40153932"
+          },
+          "handleLower":{
+            "S":"sachin-kumar"
+          },
+          "updatedAt":{
+            "N":"1618586677911"
+          }
+        },
+        "SequenceNumber":"9298848500000000010313715543",
+        "SizeBytes":394,
+        "StreamViewType":"NEW_AND_OLD_IMAGES"
+      },
+      "eventSourceARN":":table/MemberEnteredSkills/stream/2018-09-21T08:58:50.405"
+    }
+  ]
+}`