In AWS, configuration and provisioning of any resource should be done either through CLI or the console. So when using these interfaces, you have to follow multiple steps to configure resources/architecture. For fewer resources, it won’t be hard to configure, but in the case of large scale deployments, it will be tough to handle the configurations. In this case, CloudFormation comes into the picture. Cloud formation is an AWS service to configure resources using a template. The template is used to define all required configurations to setup architecture in one go.

What is Template?

It is nothing but a JSON/YAML file. Attributes of the template should be defined as per CloudFormation documentation. CloudFormation documentation clearly explains how to configure resources and attributes in templates. Here I’m mainly discussing the important attributes of the Cloud Formation template. Please refer to this link for CloudFormation documentation.

Main Attributes of Cloudformation Template :

• Parameters: this attribute is called as inputs to the template, parameters will be referred in many places inside the template.
• Resources: this attribute is very crucial, this is used to configure AWS resources, without this, we can’t configure any resource in the Cloudformation template.
• Outputs: Outputs are used to get the values after the execution of the template.
• Mappings: Mappings are used as pre-define a JSON body which matches a key to a corresponding set of named values.
• Conditions: these are the statements used to perform certain actions when the statement is true.

I gave a brief explanation of the main attributes in the CloudFormation template. Now I am going to explain the functionalities of the template to create an AWS ec2 instance.

While executing the above template, firstly it asks values for parameters. We have only one parameter and that is “EnvType” as I mentioned in allowed values, so we should select either “prod” or “test”.

At Conditions attribute, based on given EnvType we are setting CreateProdResources value, if EnvType is “prod” then CreateProdResources value will be “true” here otherwise the value will be “false”.

If we check MountPoint and NewVolume Resources, they are depending value of CreateProdResources.

• If CreateProdResources is true then MountPoint and NewVolume will be configured.
• If CreateProdResources is false then MountPoint and NewVolume will not be configured.

The above template has three resources, namely, Ec2Instance, MountPoint, and NewVolume.

If we see elements in Ec2Instance resource, Type is used to define the type of AWS resource, so here we are creating one Ec2 instance so Type is AWS::EC2:: Instance.

Note: cloud formation defined a specific resource type for each resource.

Note: AWS:: Region returns cloud formation template running region.Supppose template is running in us-east-1 region then AWS:: Region is us-east-1.

If we see ImageId property, it is fetching the value from the RegionMap object in the Mappings attribute. It checks keys in RegionMap by AWS:: Region value. So ImageId gets the values from RegionMap object in Mapping with associated AMIs to the regions.

If we see MountPoint, it is referring Ec2Instace for getting the value of InstanceId and referring NewVolume for getting volume.

Also, if we see a new volume, it is referring Ec2Instace for getting value AvailabilityZone.

Note: If one resource is depending on another one in the template, in the execution order of template, independent resources run first then dependent resources run.

So first Ec2Instance will be configured, after NewVolume and MountPoint will be configured.

Outputs give values after all resources are configured in the template, the volume gets the value after NewVolume resource creation and if CreateProdResources is true.

Conclusion

In the template we used several intrinsic functions to fetch values from other elements, they are Fn:: GetAtt, Fn:: Equals, Fn:: FindInMap. These are Cloud Formation predefined functions that we can use as per our requirement. Please read this doc for more functions.

Here, I explained only the functionalities inside the template, but I didn’t show how to execute through the AWS console, so please refer to this link to understand the execution steps of the cloud formation template.

This Blog has been moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

Fundamentals of AWS CloudFormation was originally published in Tensult Blogs on Medium, where people are continuing the conversation by highlighting and responding to this story.

This Blog has been moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

I have implemented this setup for monitoring logs of multiple EC2 instances in an AWS Account. Generally, every instance will generate several logs during runtime, it might be system logs, application logs, etc. so how can we monitor all these logs at one place? I couldn’t find any service for the centralized monitoring for instance logs in AWS, so I have implemented one customized solution for this by using Elastic Stack and AWS services ( Lambda and CloudWatch).

Note : To make it simple and understandable, I took apache2 server logs of a single instance but the same process can be done on multiple instances with different logs.

Prerequisites :

1. FileBeat is the main application for shipping and filtering the logs, so we need to install FileBeat on the instance. To download and install follow the steps here.
2. We use Elasticsearch for storing filtered logs data and here I use AWS Elasticsearch cluster endpoint. To configure AWS Elasticsearch cluster follow the below steps here.
3. AWS Lambda is used for running the script to fetch error logs data from the Elasticsearch index.

FileBeat Configuration :

FileBeat is one of the core applications in Elastic Stack and it is used for shipping logs to other Elastic Stack services like Elastic Search and Logstash, etc. As per my implementation, logs should be shipped to Elastic Search. For that, Filebeat configuration should be modified. filebeat.yml is the main configuration file of FileBeat. Refer the below image for the location.

Once you have opened the file follow the below steps:

Step 1: Fetch logs from the system: We need to set paths from where FileBeat should fetch the logs. In the below image we are setting up the path of apache2 server, so FileBeat will get apache2 logs here.

Step 2: Filter fetched logs: FileBeat can ignore unwanted event messages from logs. Processors perform certain actions as per commands mentioned under them, here we are saying that ignore messages which have “HTTP/1.0\” 200" sentence. Please refer to the below image:

Step 3: Send logs to Elastic Search: For sending logs to Elasticsearch we need to set up below configurations at Elasticsearch output section in filebeat.yml

Once all configurations are done, we need to start the FileBeat application, use the below command for the same.

~/filebeat-6.6.2-linux-x86_64$ ./filebeat -e -v -c filebeat.yml

Now it will start ingesting apache2 server logs into Elasticsearch cluster index.

Now that the FileBeats configurations are done, we need to configure Lambda and CloudWatch for generating time-based CloudWatch metrics for error logs.

Lambda Configuration :

Lambda is a serverless computing platform to run applications and it is used here to execute the below script. Below script is written for fetching log data based on queries from Elasticsearch and to set custom metric data in CloudWatch. Here I’m fetching error logs data by checking “response_code” and it is written in javascript.

Whenever Lambda executes the script, Elasticsearch will return error log data and it will generate custom metric data in CloudWatch.

Note : Attach a role to lambda with below policy to access Elasticsearch

CloudWatch Configuration :

Lambda will be executing the script every five minutes. To execute the function we have to configure a CloudWatch rule to trigger the Lambda function. Please refer to the image below:

The script will be continuously executed and it will generate CloudWatch metric data as mentioned in the below image. By setting up alarm on this metric we can get alerts when it reaches a threshold limit so that we can get notifications for error logs.

Conclusion

This implementation is done in a single AWS account, using one Elasticsearch cluster and we can ship only the filtered data thereby reducing unnecessary storage space in Elasticsearch. So this is how we ship log data to Elasticsearch using FileBeat and Lambda. Let me know your thoughts in the comments section also try to implement this using multiple accounts.

Centralized log monitoring using Elastic Stack was originally published in Tensult Blogs on Medium, where people are continuing the conversation by highlighting and responding to this story.

This blog explains step by step process of Git-flow methodology to create versioning in Git. There are so many methods to deploy production code from Git. Generally, one particular branch will be fixed for production deployments, either master branch or another branch. Git-flow also uses the master branch for deployments, but it maintains tags for every production release, so we can easily switch from one release to another by tags.

Installation

Git-flow is a small plug-in for Git. It is available on multiple operating systems. For OSX, we can install using CLI command brew install git-flow. For Window OS use this link to download.

Git-flow has specific branches, each branch has its own role. So I will explain the purpose of each branch with proper commands. Let’s start the process now, just execute below command after successful installation of the Git-flow plug-in.

$ git flow init
Initialized empty Git repository in ~/project/.git/
No branches exist yet. Base branches must be created now.
Branch name for production releases: [master]
Branch name for "next release" development: [develop]

How to name your supporting branch prefixes?
Feature branches? [feature/]
Release branches? [release/]
Hotfix branches? [hotfix/]
Support branches? [support/]
Version tag prefix? []

$ git branch
* develop
 master

Master and Develop branches :

Git-flow sets master branch for production and develop branch for next release, you can observe that in the above snippet, the master branch is used here for production release history and develop branch for integrating with feature branches.

Feature branches :

Feature branches are for implementation, suppose if we want to work on a particular feature in the project, then we should create a feature branch.

Note : Feature branches should be created from develop branch.

For creating feature branch use below command.

Starting a feature branch :

$git flow feature start featurebranch_01

Switched to a new branch 'feature/featurebranch_01'

Summary of actions:
- A new branch 'feature/featurebranch_01' was created, based on 'develop'
- You are now on branch 'feature/featurebranch_01'

Feature branch is featurebranch_01. Now, you can work on the feature branch ( featurebranch_01 ). Once work is completed on the feature branch, then we should merge that feature branch into develop branch. Follow below steps to do that.

Finishing a feature branch :

$git flow feature finish featurebranch_01

Switched to branch ‘develop’
Updating 8705e32..6b59eca
Fast-forward
 xxxxxxx.js | 1 +
 xxxx.js | 2 ++
 2 files changed, 3 insertions(+)
 create mode 100644 xxxxxxx.js
Deleted branch feature/featurebranch_01 (was 6b59eca).

Summary of actions:
- The feature branch ‘feature/featurebranch_01’ was merged into ‘develop’
- Feature branch ‘feature/featurebranch_01’ has been removed
- You are now on branch ‘develop’

If you observe the above summary of actions, it has automatically performed three actions. Same way, we can merge as many feature branches to develop as we wish.

Release branches :

Once develop branch has enough features for a release, then we need to create a release branch.

Note: Release branches should be created from develop branch.

Once release branch is created that means new project cycle is started, so every new feature will be considered for the next release, not for current release, so we shouldn’t merge any feature directly to the release branch, we should only add bug fixes and release notes to release branch.

Starting a release branch :

$ git flow release start 0.1.0

Switched to a new branch ‘release/0.1.0’

Release branch is 0.1.0. Now it is contained with all features came from develop branch.

Finishing a release branch :

$ git flow release finish ‘0.1.0’

Deleted branch release/0.1.0 (was 6b59eca).

Summary of actions:

- Latest objects have been fetched from ‘origin’

- Release branch has been merged into ‘master’

- The release was tagged ‘0.1.0’

- Release branch has been back-merged into ‘develop’

- Release branch ‘release/0.1.0’ has been deleted

If you observe the above summary of actions, the release branch is merged with master and develop branches. the release branch is tagged with “0.1.0” and it is removed.

This tag “0.1.0” contains the whole package of release branch “0.1.0”. the same way, every release will be stored into tags ex: 0.1.1, 0.1.2 etc… So we can easily track releases using tags.

Hotfix Branches :

These hotfix branches are used to fix bugs and patches of production releases. Once fixes are completed, it should be merged into both master and develop (or the current release branch), and the master should be tagged with an updated version number.

Starting a hotfix branch :

$ git flow hotfix start hotfixbranch_01

Switched to a new branch 'hotfix/hotfixbranch_01'

Summary of actions:
- A new branch 'hotfix/hotfixbranch_01' was created, based on 'master'
- You are now on branch 'hotfix/hotfixbranch_01'

Follow-up actions:
- Bump the version number now!
- Start committing your hot fixes
- When done, run:

Finishing a hotfix branch :

$ git flow hotfix finish hotfixbranch_01
  Switched to branch 'develop'
  Merge made by the 'recursive' strategy.
  xxxx.js | 1 -
  1 file changed, 1 deletion(-)
  Deleted branch hotfix/hotfixbranch_01 (was e9308a9).

Summary of actions:
- Latest objects have been fetched from 'origin'
- Hotfix branch has been merged into 'master'
- The hotfix was tagged 'hotfixbranch_01'
- Hotfix branch has been back-merged into 'develop'
- Hotfix branch 'hotfix/hotfixbranch_01' has been deleted

Note: To check out into tags, use this command.

$git checkout tags/0.1.0

Finally, we should push the master branch and tags to your repository.

$git push 
$git push --tags

Advantages of Git-Flow:

• Managing of branching system is very easy to understand and doable.
• All Commands are customized, so we can easily remember commands for a long time.
• If we follow the Procedure properly then there is no chance of getting conflicts between releases.
• Maintaining multiple branches in git is a little complicated, which can be maintained easily using Git-Flow.

Conclusion:

Git-Flow has removed some complexities in git and made the process very simple and understandable, so even new guys can understand the flow easily. It is customized for the developmentation environment, so it helps developers/programmers for understanding git easily. Tagging concept of Git-Flow is giving an easy way to track the production releases.

Version controlling using Git-flow tags. was originally published in Tensult Blogs on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this blog I will be explaining the advanced concepts of DynamoDB. If you are not familiar with the service, you can visit my DynamoDB core concepts blog here. I hope this blog will be useful to enhance your knowledge on DynamoDB and which would further add value to your application design processes.

Secondary Indices

What are indices in DynamoDB? DynamoDB provides fast access to items in a table by specifying Partition Key values. But if we need to query by attributes other than keys of the table, then secondary indices are useful to make such queries, secondary indices are two types.

• Local Secondary indices.
• Global Secondary indices.

Why do we need the local secondary index?

If we need to sort items of the table by any attribute other than tables’s sort key, then we can use local secondary index. While creating local secondary index, we can select a new attribute of the item as sort key but Partition Key remains same as the table.

We can only create local secondary indices at the time of table creation and once the table is created these indices becomes unmodifiable.

Why do we need the global secondary index?

We know that to query a DynamoDB table, we need to provide table Partition Key as a parameter but what if we want to query by some other attribute? This is when we need to use the Global secondary indices. Unlike local secondary index, we can select both Partition and sort keys in global secondary index but sort key is optional. So when we create a global secondary index, we need to specify the new Partition key other than the Partition key of the table and sort key can be same or different as the table’s sort key.

When we create a global secondary index, we also need to specify the attributes that will be projected into the index and three different options are available here :

KEYS_ONLY : Each item in the index consists the source table key values and index key values.
INCLUDE : In addition to the attributes described in KEYS_ONLY, the secondary index will include other non-key attributes that you specify.
ALL : The secondary index includes all the attributes from the source table because all of the table data is duplicated in the index.

In reality when you create global secondary key, DynamoDB creates a separate table and replicates all the table operations like insert, update and deletion asynchronously.

We can have upto 5 local and 5 global secondary indices but as local secondary indices have to be created while creating the table so we need to analyse how our query patterns will be before even creating the table.

Pagination

Pagination is a process of splitting results into separate pages, but DynamoDB handles pagination in slightly different way. DynamoDB processes query requests in two phases. First phase is the Query phase and in this phase, DynamoDB fetches records from the table using partition key and optional sort key and the second phase is the Filter phase where DynamoDB applies other attribute filters on the data fetched in the first phase.

DynamoDB provides a pagination parameter called limit and when we pass this parameter in query request, it first fetches items matched with key parameters up to a limit value in query phase and then applies attribute filters in filter phase hence the final items returned by DynamoDB after both phases will be less than or equal to the limit value. In Scan operation, during the query phase, DynamoDB directly fetches items up to the limit value and there is no matching of keys in this operation.

The maximum size of the items fetched in the query phase is limited to 1MB so even if we pass a very large limit value, DynamoDB returns only items that fits into 1MB limit.

Conditions

We can put conditions while creating or updating items. For example: put item, if it doesn’t already exist or update item if version attribute matches to the given value. To know more about conditions go to the AWS official documentation here.

What is hot partition in DynamoDB?

DynamoDB makes hardware partitions based on capacity units of the table. Capacity units of the table are distributed equally among all partitions in DynamoDB. If any partition has consumed more capacity units than provisioned capacity units, then that partition is called as hot partition.
How to avoid hot partitions? The main reason of hot partitions is partition key of the table, when a partition key value is getting more requests then obviously the item of that partition key consumes more capacity units, so selecting proper Partition key is important in DynamoDB to avoid hot partitions. Partition key should be unique and random.

Encryption At Rest

DynamoDB has provided encryption at rest feature for making table data more secure from an unauthorised access. DynamoDB encryption is a server side encryption and it is transparent to the user so we don’t have to change the application’s code. Once encryption is enabled on the table, all the data related to that table will be encrypted : the data in the table, local secondary indices, and global secondary indices.

Encryption will be done using AES-256 encryption. Encryption automatically integrates with AWS Key Management Service for managing the single service default key that is used to encrypt your tables. If a service default key doesn’t exist when you create your encrypted table then DynamoDB automatically creates a new AWS KMS key in your account. This key will be used for the tables that are created in the future.

Encryption at rest can be enabled only when the table is created. We can’t enable it after the table is created, also once encryption at rest is enabled on the table we can’t disable it back again.

TTL

Time To Live ( TTL ) is a feature that enables you to clean up items of the table automatically after a specific time. To enable TTL on the table, first we need to choose the name of the TTL attribute then we can use this TTL attribute to specify a timestamp at which point items should be deleted by DynamoDB from your table and DynamoDB never takes any additional charges for deleted items using TTL.

Note : DynamoDB deletes expired items within 48 hours after expiration, it depends upon the nature of the workload and the size of the table.

Backup

DynamoDB provides automated on-demand backup, restore, and point-in-time recovery. We have already written a blog on DynamoDB backup, and if you would like to read it, please click here.

Access control

You might have valid AWS credentials to access DynamoDB resources(tables, indexes, and streams), but until you have appropriate permissions, you cannot access DynamoDB resources. Click here to know more about AWS IAM policies.

In DynamoDB we can constrain users to certain actions while granting permission using IAM policy like allowing users to read certain items and attributes in a table or a secondary index and allowing users to write on certain attributes in a table, based upon the identity of that user.

We can achieve above scenarios using IAM policy condition element, by adding a condition element to an IAM policy, you can allow or deny access to items and attributes in DynamoDB tables and indexes. We can then apply this policy to IAM users, groups, or roles to restrict certain services or users. To know more about Access control go to an official documentation here.

Limits

Read & Write Capacity Unit Sizes :

One read capacity = One read per second (items size upto 4KB )

One write capacity = One write per second (items size upto 1KB )

Throughput limits : Except US East (N. Virginia)region, rest of the regions have the same limits.

US East (N. Virginia) Region:

• Per table — 40,000 read capacity units and 40,000 write capacity units
• Per account — 80,000 read capacity units and 80,000 write capacity units

All Other Regions:

• Per table — 10,000 read capacity units and 10,000 write capacity units
• Per account — 20,000 read capacity units and 20,000 write capacity units

Note : The throughput limit is at an AWS account level. All the account’s available throughput can be applied to a single table or across multiple tables.

Table Size : There is no limitation on table size in terms of the number of items and the number of bytes. Default limit on the Number of tables for AWS Account is 256/region. It is a soft limit so it can be increased by raising Service Limit Increase support case with AWS.

Item Size : item size should not exceed 400KB

Partition Key Size : All the items with same Partition Key goes to the same partition and the maximum partition size of DynamoDB is 10GB hence we can only store a maximum of 10GB data with a single Partition Key.

Advanced features of DynamoDB was originally published in Tensult Blogs on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is DynamoDB? DynamoDB is a managed NoSQL database provided by AWS, and it is a highly scalable and reliable database. We can scale from 10 to 1000 transactions per second (tps) in couple of seconds. Since it is a managed service, we don’t need to worry about the underlying hardware, servers or operating system. Data stored in the DynamoDB is redundantly copied across multiple Availability Zones so by default it provides protection for data loss due to underlying hardware failures.

DynamoDB organises data as tables and each table contains several items (rows) and each item has Keys and Attributes (columns). The tables in the DynamoDB are non relational and non schema based. This means table joins are not supported at DB level. For most use cases, we don’t need table joins and those fit really well with DynamoDB.

Now let us discuss DynamoDB terminology.

Tables

When we create the table we specify Partition Key and an optional Sort Key, we can’t change these later but rest of the attributes (columns) of item (row) can change. Also each item can have different set of attributes.

Example

Let us say we want to store articles from various authors in DynamoDB:

Table Keys
Partition Key : authorId
Sort Key : publicationDate

Item 1 has mainImage but not mainVideo attribute, where as Item 2 has mainVideo but not mainImage attribute. This is possible in DynamoDB as it is non-schema based.

Item1 and Item3 have same partition key abc123 and with different sort keys, which is possible in dynamoDB.

There is no limit on how many items can be stored in a table.

Item

A DynamoDB item is nothing but a row in the table. We can change any attribute of an item except its keys: partition key or sort key, these keys are an identification for an item; if we have to change these keys, then the only option is to delete an item and create it again.

Data Types

DynamoDB supports different data types for attributes of an item, they can be mainly categorised into the following:

• Scalar Types : Number, String, Binary, Boolean and Null.
• Document Types : List and Map
• Set Types : Number Set, String Set, and Binary Set.

Partition Key

This key is mandatory for the DynamoDB table and item. DynamoDB partitions the items using this key, that’s why this key is also called as the partition key and some times is also referred as a Hash Key.

Sort key

This key can be used in conjunction with the Partition key but it is not mandatory. This is useful while querying the data relating a Partition key. We can use several different filter functions on the sort key such as begins with, between etc. Some times it is also referred to as a Range Key.

Primary Key

Primary key is just a combination of both Partition key and Sort Key.

Batch APIs

BatchGetItem : This can be used to fetch items from different tables using Partition Key and Sort Key. In a single BatchGetItem call, we can fetch up to 16MB data or 100 items.

BatchGetItem can be performed only on tables not on secondary indexes.

BatchWriteItem: This can be used to delete or put items on one or more tables in DynamoDB in one call. We can write up to 16 MB data, which can be 25 put and delete requests.

BatchWriteItem cannot update items, for that use UpdateItem API call

Query

To query table we must pass partition key so selecting proper partition key for the table is important. Query operation will return all items that are matched with partition key of the table. Sort Key is further useful to filter and sort items but it is optional.

Scan

Scan operation does’t require Partition Key or Sort Key to fetch results from the table. As the name suggests, it scans an entire table to filter results based on attribute values passed as filters.

Pagination

DynamoDB Query/Scan results return maximum of 1MB response size so if our request matches an items of size more than 1MB, it gets automatically divided into pages. In such cases DynamoDB returns a special response parameter “LastEvaluatedKey” and this can be used to fetch next page results. Please note we need to pass the value of “LastEvaluatedKey” as “ExclusiveStartKey” in the next request to DynamoDB.

In some cases we might want to fix page size to number such as 10 or 20 results per page. In those cases we can use the “Limit” parameter. Please note if the results matching to the “Limit” is more than 1MB then DynamoDB only returns subset of the results which fits to 1MB limit.

Sorting

When we use Query/Scan operation on a DynamoDB table, then by default the results are sorted based on Sort Key value of the table. Incase we want that results in reverse order then we need to pass “ScanIndexForward” as “false” in query/scan request parameters. If the data type of Sort key is a number, then the results will be in a numeric order, otherwise, results will be in UTF-8 bytes. By default sort order is ascending. To get results in a descending order, pass “ScanIndexForward” as “false”.

How to call DynamoDB APIs programmatically ?

AWS SDK for DynamoDB is supported for several languages, which can be used to interact with DynamoDB API seamlessly. We can also use DynamoDB using AWS CLI.

Pricing

Last but not least… let us see how is DyanamoDB is priced.

Free Tier :

Throughput limit : 200 million requests per month ( 25 read and 25 write capacity units)

Stream limit : 2.5 million read requests per month.

Storage : 25GB of indexed data storage.

Note : Above free tier does not end after 12 months.

For more information about pricing click here

What next?

Now that we have learnt basics of DynamoDB, I will be discussing more concepts related to DynamoDB in my future blogs. So if you want me to particularly focus on any topic of DynamoDB, then please leave a comment on this blog.

Core concepts of Amazon DynamoDB was originally published in Tensult Blogs on Medium, where people are continuing the conversation by highlighting and responding to this story.