Serverless is a revolution for interpreted, single threaded languages such as Node.js or Python. JavaScript engineers have got possibility to write scalable and fast architecture without going too deep into multi threaded issues. Initialisation of Node.js and Python processes is fast, which makes them ideal choice to use for AWS Lambda. Functions can be run in many instances in parallel with autoscaling (based on real traffic). You can control that and eg. prevent spinning up too many instances (reserved concurrency) or set min. instances available to prevent cold starts (provisioned concurrency). One part of your application can be run eg. in 200 concurrent processes, when other in 10. And you pay only for what you’re using.

Here are some notes and tips, which might be helpful, if you are starting your journey with AWS Lambda and Dynamo DB.

Async architecture with AWS Lambda

Split your logic

The tricky part is to proper organise architecture — split logic into multiple functions and transfer data between them. That splitting makes sense especially if one part of logic is invoked more often than other, or if one part is more complicated, takes more time / memory to compute. Another reason could be, that for one function you would like to have concurrency limit— for example to avoid too many requests to external API or DB (throttling issues).

Keep in mind, that Lambda initialisation can take some time (not so much as compiled languages, though), so it might not be necessary to split logic into Lambdas, which computation time takes less < 1s.

Know the difference between sync and async Lambda

You can invoke Lambda synchronously or asynchronously. If your logic is simple and you expect direct response from your function, invoke Lambda synchronously (InvocationType = RequestResponse). For example Lambda calls via API Gateway or Elastic Load Balancer are synchronous, as you expect response.

For more complicated logic it’s recommend to split it between functions and invoke them asynchronously (InvocationType = Event). If you would like to call Lambda from other Lambda synchronously — keep in mind, that you may pay twice, as time of both Lambdas will be billed separately.

Example of async / event driven architecture: one Lambda sends message to topic (SNS), or Event Bus (EventBridge), or for example updates record in database, and another Lambda is subscribed to that specific event / stream. Communication in that case is event driven - there is no other Lambda waiting for response as it could be in sync model.

You can orchestrate logic using AWS Step Functions, so your event driven application will be easier to maintain. If you would like to orchestrate async workflow, but expect sync response after workflow will be finished (eg. by API Gateway) — consider Synchronous Express Workflow. For some applications AWS Step Functions might be too costly because of double-billing issue, so use it carefully. Sometimes it might be better to stick with EventBridge or Lambda destinations instead.

Async Lambda has its own event queue (don’t confuse with SQS), so you can implement retry logic in case event won’t be delivered (max 2 retries) and it can keep events for max. 6 hours, eg. when function doesn’t have enough capacity to handle all incoming requests (throttling errors). Keep in mind that event queue is eventually consistent (event can be sometimes delivered more than once). If you expect throttling issues, consider implementing SQS queue (instead of relying on event queue) to have guaranteed, that all events will be delivered.

Do not forget about handling errors — you can set SQS queue or SNS topic as dead letter queue or as an on failure destination. In first case you will have access to all discarded events, in second case you will have access to events, but also to errors responses.

Think about multiple ways of triggering Lambda

You can trigger Lambdas by events emitted from different sources. List of possible event sources is long.

The most popular ways to trigger Lambda are:

• SNS — pub/sub (push) functionality; one or many Lambdas send message to topic, which others Lambdas/services can be subscribed to;
• SQS — queues (pull) are ideal solution when you expect throttling problem — for example if your traffic is very dynamic and you would like to avoid to lose any messages, or if you want to optimize Lambda autoscaling (60 additional instances per minute to a maximum of 1,000 concurrent invocations);
• EventBridge (CloudWatch Events) — used for more complex events management, where you can filter by event patterns, subscribe to scheduled job (cron), 3rd party emitters and more — such as communication between accounts;
• Kinesis — dedicated for streaming or data driven applications;
• S3 — you can trigger Lambda based on changes in S3 bucket eg. one service upload file, other service does some operation on it;
• DynamoDB — Lambda can read records from DB stream, so you can react each time, when data changes;

By default SNS and SQS don’t guarantee, that messages will be delivered in the same order, as they were published. Occasionally there could be situation that messages will be delivered more than once, so you might need deduplication mechanism. If you would like to prevent duplicate messages from delivered and need order to be guaranteed, think about SNS FIFO or SQS FIFO.

Get some inspirations from others

There are many different ways to deal with events in AWS, so I encourage you to read about how others build theirs async architecture. Great source of patterns and solutions is AWS Solutions Reference Architectures and cdkpatterns.com.

Understand Lambda execution to improve speed

When you are Node.js developer, cold starts (increased invocation latency) might not be your main issue. If Lambda does a lot of work eg. soon after invocation (making connection to DB, retrieving data from SSM or so), you might want to improve that speed — especially if that could improve UX.

Good idea is to cache some data in memory, so it can be used in another invocation within the same execution environment (runtime) later. You can play with Middy middleware (cache, ssm packages) or memoize some pure functions eg. with Memoizee.

Think about provisioned concurrency (requested number of execution environments will be always prepared to respond to your function’s invocations), but be careful — it might be too much costly for some cases. If your traffic is very dynamic, you can manage your provision concurrency within Application Auto Scaling.

Learn about Lambda execution environment lifecycle. It might be good idea to run some logic in Init phase — out of the handler, so it can be run after starting runtime, eg. by provisioned concurrency, so before invocation. You can put into Init phase synchronous code, up to 10 seconds to compute. Async code in Node.js (such us loading SSM by SDK) might not be finished in Init phase (will be frozen and resume in first invocation). If you would like to finish loading data (eg. from SSM or DB) in Init phase, try Python.

Problem of async code in Node.js is not limited do Init phase only. If you run async code in Lambda, make sure that it’s finished before you will return response / resolve handler. The best way is to always keep async logic in async-await promises instead of callbacks. Callback pattern (such as using setTimeout) can lead to execution leaks (problem, when executing code runs in a different invocation than the original execution context).

Use Lambda layers

You might be in situation, that one service share a lot of code with others. In Node.js you can have some common node_modules and it might not be good idea to include all of them in each bundle, or to deploy to each Lambda container separately. Think about deploy common code as AWS Lambda layer. It could be useful also if some part of your Lambdas code is heavy, and you would like to avoid deploying it each time, when you change something.

DynamoDB — performance and throttling

Learn about difference between DynamoDB and relational DB

Proper understanding NoSQL database is key to write performant queries. DynamoDB has been created to be fast and scalable. For someone who came from SQL world into DynamoDB, it might me hard to change the way of thinking about data schemas. I recommend read Alex DeBrie’s DynamoDB guide (dynamodbguide.com), blog or buy “The DynamoDB Book” (dynamodbbook.com), which gives you deep knowledge about that database plus some great pattern to work with DynamoDB. Good summary of Dynamo DB you can find in this article.

Think about access patterns

DynamoDB is fast, because you access data directly from partition (you always need to know partition key and optionally sort key). When you query for more than one item, it’s important to have proper keys design. One key can be composition of different data. It’s common practice to give simple name to keys (as PK, SK) and keep different types of data in the same table (single table design). You can different that data types by adding prefix to your key followed by hash, such as TYPE#123. Because data in DynamoDB are organized in B-trees, you should also think about, how you would like to sort / filter it, and then design your sort key accordingly. If you would like to access your data in couple different ways, consider using GSI or LSI (global / local secondary index).

Keep in mind DynamoDB limits

DynamoDB has some limits, which you should always keep in mind:

• max size for single item is 400KB
• max result size for Scan/Query operation is 1MB
• batch get: max 25 items, 16MB; batch put: max 100 items, 16MB
• 3000 RCU (Read Capacity Units) and 1000 WCU (Write Capacity Units) throughput limit per partition
• 40 000 RCU and WCU default limit per table, 80 000 per account
• max 10GB per collection with the same partition key (doesn’t include global secondary indexes)

Distribute traffic

If your application has big traffic or operate on large payloads, you might hit throughput limits. That’s why it’s important to have proper design for your partition keys, to avoid situation, when one partition has much more traffic than others (“hot” partition key). If you struggle problems with proper design, you can try distribute writes across more partitions by sharding.

Handle errors

DynamoDB should be reliable, especially if you’re using Global Tables, but don’t forget about handling errors — log them, think about retry logic and set DLQ for your Lambda.

If you’re using AWS SDK, you will have retry and exponential backoff algorithms set by default. It is useful, as SDK will retry request in case of limits /throttling issues or 5xx errors.

Situation is a bit more complicated, when you are using batch methods (BatchGetItem, BatchWriteItem). As some operations can failed (eg. because of throttling issues) and some succeed in the same request - you will receive 200 response and SDK won’t retry that request. You will need to write your own retry / backoff mechanism to handle UnprocessedKeys / UnprocessedItems if there any in response.

Read more about error handling in DynamoDB.

Others

Do not forget about monitoring

If you are Node.js developer, you might be interested in pino logger. It has a lot of helpful functionalities, such as redaction (if you would like to avoid showing sensitive data in your logs), levels (eg. to do not show debug logs on prod) or child loggers — to add common data for each logs.

Proper logging (eg. payloads, ids, states, errors) will help you debug your code (eg. building queries in CloudWatch Logs Insights). This is especially important in event driven application, when you cannot debug whole workflow within one trace id.

You should also consider create alarms for metrics, which are important to you. For example creating CloudWatch Anomaly Detection for lambda invocations sounds interesting.

Do not forget about creating charts, diagrams and others statistics. In CloudWatch you can create dashboards, but think about third party solutions. There are a lot of services, which work well with AWS and make your debugging, tracking performance much easier — eg. New Relic, Datadog or Sumo Logic.

Do not overcomplicate your stack

Your services can grow quickly. You might think to put all functions / resources definitions into one CloudFormation stack (eg. in single serverless.yaml /Serverless framework). As this sounds good for few functions (which together delivers single feature), in future it could make you some troubles.

For a long time CloudFormation had limit of 200 resources in stack and because one function needs couple of resources (Lambda, versions, roles etc.), there was high risk, that you will hit that limit quickly. Especially if you keep common resources (such as S3, DynamoDB, SNS, SQS) in the same stack. Now that limit has been increased to 500, but still —you should avoid keeping so many resources in one stack.

If you hit that limit, moving resources from one stack to another can be painful — you will need to guarantee, that during operation all yours resources will be reliable. AWS supports importing resources from one stack to another, but it’s not easy to set, especially if you use framework, such as Serverless, to generate and deploy CloudFormation stack.

In summary:

• think about resources limit at the beginning, to avoid refactoring problems;
• splitting architecture into multiple stacks might improve you build / deploy process;
• keep you global / common resources (database, queues, topics) out of stacks, which you often deploy;
• if you really need to keep all resources in single CoudFormation stack, think about nested stacks.

Choose the best framework for you

If you are new in Serverless and want to create something simple, think about AWS Amplify.

For larger project it’s worth to try Serverless framework, which improve building and deploying Lambda services (and surrounding resources). It has large community, and a lot of plugins (eg. Webpack or Offline).

Serverless framework supports different providers, but you might think about something dedicated only for AWS — try Serverless Application Model.

You should also consider Terraform, which is great framework for managing persistent shared infrastructure. It’s common practice to use Terraform for cloud infrastructure and SAM or Serverless for deploying applications.

Listen podcasts, read newsletters, prepare for certificate

My favorite resource is Off-by-none newsletter + podcasts: AWS Podcast, Serverless Chats, Real World Serverless.

Very useful informations can be found on AWS YouTube channel, AWS Solutions Reference Architectures and cdkpatterns.com.

Some of Twitter accounts, which are worth to follow: @jeremy_daly, @sheenbrisals, @danilop, @dabit3, @theburningmonk, @jbesw, @houlihan_rick, @jeffbarr, @alexbdebrie, @awsgeek.

For beginner great resource is serverless-stack.com.

If you would like to be AWS architect, you can prepare for certificate. I can recommend Ultimate AWS Certified Solutions Architect Associate course.

Exploring serverless world has opened my eyes on problems / areas, which I still need to learn. This was just small list of tips, but hope you found something useful.

Engineers spend a lot of time debugging and looking for performances bottlenecks. Memory leaks, split brains issues… You need to know, how to write good code but also predict which parts of service will need more power and eventually move them into separate microservice or create serverless function (eg. AWS Lambda).

PDF renderer engine is good case for using AWS Lambda, because it uses a lot of memory in comparison to the rest of my application. I want to share with you implementation of PDF news app which I have created based on AWS: EC2, S3, DynamoDB, Lambda.

Concept

Application renders news in PDF for chosen country and category. As a source of news I use newsapi.org REST endpoint. UI is simple React app and server is created by Express.js. Source code: https://github.com/machnicki/aws-pdf/tree/version/1

Workflow

1. Before Express API will call newapi.org, first it will look for proper document in S3 . If there is PDF already generated, API will return it instead of create fresh one— simple 1-hour cache mechanism. App creates only one PDF for particular combination of time (date and hour), category and country. ${country}-${category}-${date}.pdf is unique key for document and file name.
2. Express API calls newapi.org with proper category and country. Response is in JSON format.
3. After receive news in JSON, Express API generates HTML document based on that JSON data and HTML template (template is kept in DynamoDB). Templating is done via handlebars (https://handlebarsjs.com/). Generated HTML document is saved into DynamoDB.
4. Express API invokes Lambda function, passing file name (unique key for HTML document) and expecting url for generated PDF as response.
5. Lambda function gets proper documents HTML from DynamoDB.
6. Lambda function generates PDF (I use html-pdf node module: https://github.com/marcbachmann/node-html-pdf) and saves it into S3.
7. As response Lambda function returns url to generated PDF document.

React SPA

For UI I have used create-react-app with react-bootstrap. All frontend code you can find here: https://github.com/machnicki/aws-pdf/tree/version/1/src.

For purposes of this article, important part is, how javascript use HTML form data, send them to API and open generated PDF in new window: https://github.com/machnicki/aws-pdf/blob/version/1/src/App.js#L33-L53

Express backend

I decided to work with newest ES syntax with experimental ES modules (node version 11). Thats why I need to use — experimental-modules flag to run my nodejs server: https://github.com/machnicki/aws-pdf/blob/version/1/package.json#L20.

I have defined couple simple API endpoint, most important is POST /api which triggers generating PDF based on UI payload: https://github.com/machnicki/aws-pdf/blob/version/1/api/index.mjs#L21.

Other endpoints are not part of this article, they are used to edit template, download list of documents or particular document. You can explore it yourself, enjoy!

Main GET / route is used to serve UI (index.html). Depends on environment, this is webpack dev server or static build.

DynamoDB vs. DocumentDB vs. S3 store

In my application I need to store documents HTML code to consume it by Lambda function. It’s massive string so I cannot just pass it as payload. I have decided to use NoSQL database. DynamoDB was simple enough, but I have considered another NoSQL database from Amazon — DocumentDB, which gives me MongoDB and possibility to save larger data (16MB for document, where MongoDB max size of item is 400KB — for large HTML code it could be not enough). At this moment I stick with DynamoDB.

Documents are kept in simple pair: id (file name) and html (base64 HTML). Template used for generating document is specific document with name “template”.

All code which is run do create DB structure you can find here: https://github.com/machnicki/aws-pdf/blob/version/1/api/db/createTable.mjs

In future I will probably resign from DynamoDB and keep all data directly in S3. S3 will inform Lambda to create new PDF if S3 will receive HTML document. I consider S3 Select or Amazon Athena tools.

EC2, DynamoDB, Lambda via LocalStack

Create HTML document and save in DB

My POST /api endpoint gets country and category as parameters and calls newapi.org for news in JSON format. Based on that data and HTML template (which was crated in previous chapter) I create HTML document (using handlebars node module) and save that document into DynamoDB. Next I call Lambda function, passing file name (id of HTML document in DynamoDB) and expect url of PDF document (PDF is generated and save into S3 by Lambda — explained later).

Source code: https://github.com/machnicki/aws-pdf/blob/version/1/api/pdf.mjs and https://github.com/machnicki/aws-pdf/blob/abf5f072f04d5bb680708186fbf237f54d1fb225/api/pdf-generator.mjs (invoking AWS Lambda)

LocalStack

I was looking for solution, where I could write all my serverless code locally before I will deploy it into AWS.

LocalStack gives me local version of S3, DynamoDB and Lambda.

GitHub - localstack/localstack: 💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline

To work with all services locally (so with Lambda as well) you need to run LocalStack via docker (as I have described in README: https://github.com/machnicki/aws-pdf/tree/version/1#aws-lambda-pdf-news-generator).

All services are run or different ports and each Lambda function on different containers.

In config file you need to use LOCALSTACK_HOSTNAME env variable instead of localhost to allow those containers talk each other.

Check my config file https://github.com/machnicki/aws-pdf/blob/version/1/api/config.mjs which returns different values depends on environment. For production (AWS) I don’t need to specify endpoint parameters for each service. Additionally for Lambda environment I don’t need so specify accessKeyId and secretAccessKey (Lambda permissions are configured in AWS console).

Lambda

My lambda (https://github.com/machnicki/aws-pdf/blob/version/1/lambda/pdf.mjs) apart of source code needs also something which will convert HTML into PDF. I have decided to use html-pdf node module, which is based on phantomjs.

Lambda function as a parameter retrieve file name, load HTML document from DynamoDB, generate PDF, save it into S3 and returns url to that PDF:

Generator

For most cases Serverless framework is easy and quick way to develop Lambda functions, but in my code I rely only on aws-sdk node module.

This is what I’ve needed:

• write functions in Node.js 11 with experimental ES modules enabled,
• deploy large phantomjs binary file (for generating PDFs),
• depends on environment, deploy function to AWS or to LocalStack.

Webpack bundler

AWS Lambda supports only 6 and 8 versions of Node.js.

I have decided to use Webpack to create one bundle file which will be readable by AWS Lambda.

Phantomjs binary

My AWS Lambda creator (https://github.com/machnicki/aws-pdf/blob/version/1/lambda/create.mjs) can create zip file based source file of Lambda function with additional files (like phantomjs binary).

Lambda package size can be maximum 50MB. For larger zip file then 10MB it’s recommended to upload them first to S3 and then point Lambda to that file.

I have discovered that its better to generate zip file by using command line instead of node module node-native-zip, which works fine for simple functions, not necessary for large files, like phantomjs binary (50MB and more).

If you will struggle issues with zip generator in my code, try use another one. Command line is probably good choice.

At the end your zip file which you will deploy to AWS Lambda should include 3 files:

• PDFGenerator.js (commonjs bundle — main function)
• phantomjs (binary — you need to download linux version to run it in AWS Lambda environment, eg. form https://bitbucket.org/ariya/phantomjs/downloads/)
• scripts/pdf_a4_portrait.js (needed by html-pdf node module to generate A4 format)

In README (https://github.com/machnicki/aws-pdf/blob/version/1/README.md) you will find necessary informations, how to run application locally.

Hope you have enjoyed this article. In near future I will try to use other AWS tools and refactor current code:

• Make sure that only PDF creator has access to PDF (authentication)
• Move rest of the app from EC2 into Lambda
• Replace DynamoDB (as a store for html code) by S3 events
• Performance tests and improvements