|
| 1 | +# member-skills-sync-lambda |
| 2 | + |
| 3 | +## Prerequisites |
| 4 | +1. Amazon DynamoDB table for the streaming and sending member skills data to Amazon Lambda exists and the ARN is known (please enable the streaming on DynamoDB with View type "New and old images" - see screen-shot in Verification Document if required). |
| 5 | +2. Amazon ElasticSearch domain for posting the member skills data already exists and endpoint is known. |
| 6 | +3. AWS Lambda IAM role that has the privilege (permission policies) to read the stream from DynamodDB already exists and ARN is known. |
| 7 | +4. Amazon ElasticSearch domain access policy has been configured so that AWS Lambda IAM role can do necessary action to post data to ElasticSearch and for unit test purpose, either using IP address or Serverless Agent IAM User below is given access to post data to ElasticSearch. |
| 8 | +5. Serverless Framework has been installed. If not yet done, please follow this [link](https://serverless.com/framework/docs/providers/aws/guide/installation/). |
| 9 | +6. Serverless Agent IAM User has been configured properly and its access key ID and its agent secret access key is known, please follow this [link](https://serverless.com/framework/docs/providers/aws/guide/credentials/) to setup one if not yet done. |
| 10 | +7. All the instructions below are using Node v10.16.3. |
| 11 | + |
| 12 | +## Environment Variables |
| 13 | +Please follow this instruction to setup the required environment variables. We are using Linux terminal in this example. |
| 14 | + |
| 15 | +First you need to setup environment variables for AWS credentials. For review purpose, actually you don't need to create a new user for severless agent, just use your AWS account's access key and secret access key which already have full-access to all AWS services. |
| 16 | +``` |
| 17 | +export AWS_ACCESS_KEY_ID=<your access key id> |
| 18 | +export AWS_SECRET_ACCESS_KEY=<your secret access key> |
| 19 | +``` |
| 20 | +For the sake of unit testing to run, running your lambda function locally, and deploying your lambda's environment variables, please also setup 9 more environment variables below in your terminal (using export command same as above). |
| 21 | +1. ES_URL --> Endpoint of your elastic search |
| 22 | +2. ES_REGION --> Region of your elastic search e.g. us-east-1 |
| 23 | +3. ES_SKILLS_INDEX --> ElasticSearch Index for member entered skills documents |
| 24 | +4. ES_SKILLS_MAPPING --> ElasticSearch document type for member entered skills index |
| 25 | +7. DB_ENTERED_SKILLS_STREAM --> ARN of DynamoDB table MemberEnteredSkills stream |
| 26 | +8. DB_AGGREGATED_SKILLS_STREAM --> ARE of DynamoDB table MemberAggregatedSkills stream |
| 27 | +9. LAMBDA_ROLE --> ARN of IAM Role to be used when executing Lambda |
| 28 | + |
| 29 | +## Change the serverless.yml to fit your environment |
| 30 | +Please open serverless.yml from the project and modify following value in case you don't want to use environment variables: |
| 31 | +1. role (under index.syncSkills) --> ARN of your AWS Lambda IAM Role |
| 32 | +2. arn (under stream) --> ARN of your DynamoDB table stream |
| 33 | +3. ES_* (under environment) --> This is by default populated by environment variables, feel free to change if needed |
| 34 | +4. DB_* (under environment) --> This is by default populated by environment variables, feel free to change if needed |
| 35 | +5. Please also modify stage and region to fit your needs |
| 36 | + |
| 37 | +## Install Dependencies and Run Lint |
| 38 | +1. run the following command to install the dependencies |
| 39 | +``` |
| 40 | +npm install |
| 41 | +``` |
| 42 | +2. To run linter if required |
| 43 | +``` |
| 44 | +npm run lint |
| 45 | +
|
| 46 | +npm run lint:fix # To fix possible lint errors |
| 47 | +``` |
| 48 | + |
| 49 | +## Unit Test |
| 50 | +Make sure you have properly setup the [environment variables](#environment-variables) and [installing dependencies](#install-dependencies-and-run-lint). Run `npm run test` to run the unit test. |
| 51 | + |
| 52 | +## Run lambda locally |
| 53 | +1. Make sure you have properly setup the [environment variables](#environment-variables) and [installing dependencies](#install-dependencies-and-run-lint). |
| 54 | + |
| 55 | +2. Run command `npm run test-data` to generate test data.(Important step!! The provided test data have different eventSourceARN which are different with your environment variables. The lambda function need to use eventSourceARN to distinguish ES index.) |
| 56 | + |
| 57 | +3. Then, Run the following to invoke lambda locally. |
| 58 | +``` |
| 59 | +serverless invoke local -f sync -p test_data/createEnteredSkillsEvent.json |
| 60 | +serverless invoke local -f sync -p test_data/updateEnteredSkillsEvent.json |
| 61 | +serverless invoke local -f sync -p test_data/deleteEnteredSkillsEvent.json |
| 62 | +serverless invoke local -f sync -p test_data/createMemberAggregatedSkillsEvent.json |
| 63 | +serverless invoke local -f sync -p test_data/updateMemberAggregatedSkillsEvent.json |
| 64 | +serverless invoke local -f sync -p test_data/deleteMemberAggregatedSkillsEvent.json |
| 65 | +``` |
| 66 | +Please note that even if the lambda is invoked locally using test data, the data is still being posted to ElasticSearch in AWS based on environment variables above. In order for the local invocation to work properly, the serverless agent that is used by serverless command has to have privilege to post the data to ElasticSearch or the access policy for the ElasticSearch is configured to allow any AWS principal to post the data but restricted by condition e.g. by using IpAddress. |
| 67 | + |
| 68 | +## Deploy |
| 69 | +When all the things mentioned above have been done, please run the following: |
| 70 | +``` |
| 71 | +serverless deploy |
| 72 | +``` |
| 73 | +Feel free to use flag `--verbose` after `serverless deploy` if you want to know more information about your deployment. |
| 74 | + |
| 75 | +You can also modify the lambda code if required and the redeployment is as simple as running `serverless deploy` again. |
| 76 | + |
| 77 | +## Verification |
| 78 | +Please read verification document for screen shot of enabling DynamoDB table stream, unit test result, deployment results, and results of manual testing on the lambda. |
0 commit comments