Our setup

We run the Docker registry on EC2 instances with AWS S3 as a storage backend for images and AWS CloudFront as CDN for caching images. Docker registry container behind an Nginx proxy for authentication. We use the way recommended in the documentation.

Removing CloudFront middleware

To save data transfer costs from outside the VPC, we disabled CloudFront middleware from the configuration. Removing CloudFront from our configuration impacted our Docker pull times in the Oregon region, but there was no impact in pull times for the Singapore region since the S3 bucket storing all the Docker images is in the Singapore region as well.

We were expecting the data transfer cost to go down but it didn’t create any significant impact since the data transfer happening between VPC & S3 was still over the internet. We decided to set up an S3 VPC endpoint for our VPCs to make data transfer between our VPC & S3 private. After this change, our data transfer costs for the Singapore region reduced quite a bit. This was our first win!

We were still getting charged for data transfer between S3 & Oregon region because the S3 bucket storing Docker images bucket was in the Singapore region and VPC Endpoints for services work for VPCs and AWS services within the same region.

Pro-tip: If you want to use CloudFront in one region but want to use s3 for some regions, you can use ipfilteredby & awsregion to bypass CloudFront and pull from s3 directly. You can check the documentation here for more information.

Setting up a Docker registry as a pull-through cache

After going through the Docker registry’s documentation, we came across how to setup Docker registry as a pull-through cache. This was a simple configuration change. If the image earlier was at my-docker-registry.com/my-image, I can also pull the same image through my-docker-registry-cache.com/my-image.

Docker introduced rate-limiting on Docker pulls from their registry in November 2020. We haven’t tried this out, but a private Docker registry as a pull-through cache can be used to avoid getting rate limited as well as improve image pull times. You need not be concerned about pulling stale images since it checks for stale images by making a HEAD request upstream. And HEAD requests are not counted by Docker for rate limiting. You need to set https://registry-1.docker.io as remoteurl in the configuration for this.

We used a configuration like this:

Once we had set up the pull-through cache, the next challenge that we faced was how do we change it for applications. Making changes to every codebase and handling a different image path for production and development environments was difficult. We explored two approaches:

1. Geolocation based DNS records
2. Mutating webhook in Kubernetes

Geolocation based DNS records

How this approach works is that it will return different responses depending on your location. This approach seemed great but had two issues:

1. Not every DNS provider supports geolocation based DNS records (e.g. — AWS Route53 supports this but not Cloudflare which is our primary DNS provider)
2. We could pull images from the Oregon region, but it failed when we tried to push images.

We had some workflows which were building images in the Oregon region and pushing them to our private registry. So we couldn’t go ahead with this approach since it would break existing workflows.

Mutating webhook in Kubernetes

The majority of the pulls in the Oregon region for us were from our Kubernetes cluster. So, we deployed a mutating admission webhook, which replaces my-docker-registry.com with my-docker-registry-cache.com in Docker image URLs.

Conclusion

We deployed this change around May 20. The previous drop was because of image caching on Kubernetes nodes. This setup improved our Docker image pull time and reduced our data transfer costs from our Docker setup by more than 50%. We explored migrating to AWS ECR, but it didn’t support cross-region replication last year when we did this. Since AWS ECR supports cross-region replication now, it might be a good idea to explore that as well now.

Reducing data transfer costs with a Docker registry cache was originally published in Lambda by Blinkit on Medium, where people are continuing the conversation by highlighting and responding to this story.

We have been extremely focused on growth before COVID started. AWS is central to how we ship fast that helps fuel that growth. Some of our early mistakes resulted in wastage resulting in high costs over time. COVID-19 pushed us to bring more efficiency in our business operations to provide more savings to our customers during these tough times. As a part of that initiative, we started looking at ways to reduce our infrastructure cost.

In this post, we will try to share tools, our key learnings and strategies to attack the problem of high infrastructure costs.

Identifying Wastage and Optimization Opportunities

To reduce costs, we needed to look at our biggest contributors to cost. AWS Cost Explorer is a good place to start for this. If you have tagged your resources properly (we tag by service, team, division, etc. There is a nice guideline for this by AWS here), Cost Explorer can show you how much a team, application or microservice costs.

Service-wise and application-wise cost attribution and reports is a good place to start. But as you start to identify the reasons for costs under different heads, you will need to get a more granular view of why a particular service or application is costing you so much. To get data at a granular level such as operation or API level, Cost & Usage Reports (CUR) are helpful. To setup CUR Reports, you will have to set up a pipeline to make the CUR data accessible for queries and then query the reports using Athena.

Major Contributors to Cost

Depending on your business, you will have different things that will be your major cost contributors. However, there are some AWS services that are common amongst most businesses — EC2, RDS, S3, and Data Transfer. This applies to us as well.

Major contributors to cost for us are:

• Data Transfer
• EC2
• S3
• Reserved Instances

Let’s look at each of these in detail.

Data Transfer Costs

Data transfer costs are the ones that get ignored most easily because it is hard to account for data transfer over the network as compared to compute usage. Not to forget that when you are building up your business, the network usage is very hard to estimate. The cost of network early on is usually so small that it doesn’t warrant so much attention. However, some higher level decisions such as avoiding usage of public network routes for accessing internal services or reducing data transfer due to infrastructural components (such as monitoring) can go a long way in saving money. That is as much as you can do. As you scale up, the cost of the network starts to show up.

Data transfer costs show up in EC2-Other service in AWS Cost Explorer. AWS charges you for different kinds of data transfer:

1. Data transfer from outside the VPC — listed as NAT Gateway, PublicIP-In, PublicIP-Out, VPCPeering-In, VPCPeering-Out in AWS Cost Explorer
2. Cross-AZ data transfer — Listed as InterZone-In, InterZone-Out in AWS Cost Explorer

Data transfer from outside the VPC (Internet)

Our major costs here were NAT Gateway costs. NAT Gateway is used to enable instances in a private subnet to connect to the internet. AWS Cost Explorer doesn’t provide much information about what data transfer you are being charged for. This documentation by AWS recommends these solutions to reduce data transfer costs:

1. Setup NAT Gateway per AZ to reduce cross-AZ (or inter-AZ) data transfer.
2. If you are using DynamoDB and S3 in the same region, setup gateway VPC endpoint.
3. If the majority of the traffic is to AWS services supporting VPC endpoints, create one interface VPC endpoint.
4. If the majority of traffic needs to go to the internet, set up an internet gateway.

Pro-tip: You can also set up a public instance and use that instead of a NAT gateway if you don’t want to pay for NAT Gateway running hours. This can be risky if the instance goes down, needs maintenance or your network bandwidth requirements cannot be satisfied by the network bandwidth cap of EC2, so you can probably try this in your non-prod environments. We didn’t try this but learned from people we know that some companies do this as well.

We already had a NAT Gateway per AZ so we don’t get charged for cross-AZ data transfer.

VPC endpoint is a hidden gem. Data transfer cost due to S3 is often a big contributor. As soon as you roll out the VPC endpoint, you will see a sudden drop in data transfer cost. Before we set up a VPC endpoint, we needed to know where we were making requests to get a better understanding of our costs and usage. If S3 or other AWS services show up in your analysis, you should look at deploying VPC endpoints immediately.

VPC endpoint is a hidden gem. Data transfer cost due to S3 is often a big contributor. As soon as you roll out the VPC endpoint, you will see a sudden drop in data transfer cost.

Reducing NAT Gateway charges further would require building a better understanding of the reasons for high NAT Gateway charges. To further drill down into where all we were being charged for NAT Gateway data transfer, we used VPC Flow Logs on a log group and AWS Insights to find out our biggest contributors. You can find some sample queries here and here. You can also dump VPC Flow Logs to an S3 bucket and later query them using AWS Athena to do custom analysis if your infrastructure and business setup requires you to find deeper answers about your network usage. For details on how to set that up, follow this post by AWS. If you need to identify AWS service from IPs, you can check AWS IP Ranges here.

After analyzing our VPC Flow Logs, we found out that most of our NAT Gateway cost is from:

1. Our applications sending requests to other applications via public load balancers
2. Cost of of data transfer between our own EC2 and S3

Application sending requests to our own public load balancer

Identifying which applications are making requests to public load balancers can get a little tricky because access logs of your receiving load balancer will show your NAT gateway’s public IP. We had to perform the painful process of talking to all service owners who had public load balancers and set up an additional private load balancer, and then have all the applications migrate to using the private load balancer. In some cases, we had to set up two load balancers — one for public travel and another for internal traffic, because our NAT Gateway data transfer costs were higher than the cost of an additional load balancer.

Cost of of data transfer between our own EC2 and S3

We created a gateway VPC endpoint to make traffic to S3 private. After making this change, our S3 data transfer costs went significantly down, but the NAT Gateway costs due to S3 were still high. This was because we were transferring data from buckets in a different region. We had to migrate some buckets to the same region from where these buckets were being mostly accessed.

One of the big S3 data transfer expenses was caused by our internal Docker registry. After all these changes, we couldn’t reduce the cost of data transfer because our stage and CI environment is in a different region. Private endpoint was effective only for the environment where the registry’s region was the same as the environment’s region (production in our case). So we set up a docker registry with a proxy in the same region as our stage to cache docker images. We will be publishing details about how we achieved this in a separate blog post soon.

Cross-AZ Data Transfer

We had significant Cross-AZ data transfer in our production as well as stage environments. There are a few easy things you can do here — like enabling compression, reducing unnecessary data transfer like debug logs, etc. If you have done all the configuration and tuning, the next thing to look at is trying to reduce the data transfer footprint of your services. This is not easy to solve so we decided to take it up as a long term improvement. Depending on your risk appetite, you can also look at running your entire infrastructure out of a single AZ. We migrated our stage infrastructure and migrated large portions of production infrastructure to a single AZ bringing down the inter AZ cost to as less as we could. This was an interesting migration. You can read more about the migration here.

EC2 Costs

Decreasing EC2 costs depends a lot on the infrastructure you are running. There were multiple steps we took to reduce our EC2 costs.

Resource Optimization

The first step is to see which EC2 instances are costing you the most and then work your way from there. Once again our tagging strategy helped us a lot in determining this using AWS Cost Explorer. We made a list where the instance type could be changed or if an auto-scaling group’s minimum size could be reduced. For Kubernetes, we needed something similar to Cost Explorer. After some exploration, we came across kube-resource-reports. This provides the current view of costs and usage by applications, namespaces and pods. We used this to identify resources that were requesting lots of resources but not utilizing them so that we could right-sized them.

Utilization of Reserved Instances (RIs)

We had bought RIs a couple of months before COVID-19 hit us. After changing instance types and reducing auto-scaling groups’ min size, our RI Utilization went down. This was pure wastage. But fortunately, we bought Convertible RIs so we could convert them to other instance types to make sure that our RI Utilization is 100%. If you are looking to buy EC2 RIs, we would suggest looking at Savings Plan launched last year which is an offering similar to RIs but you don’t have to worry about converting RIs or purchasing them separately for every regions. One important caveat with Savings Plan is that it’s not exactly capacity reservation. It is purely a discounting construct.

Spot Instances

Spot instances are a great way to save costs. They can be as cheap as 30% of the on-demand cost. Majority of our infrastructure was running on on-demand instances. We had explored spot instances earlier but we had some concerns:

1. If the spot instance gets terminated due to price issues, how do we ensure application remains available?

Spot instances are quite easy to manage with AWS’s autoscaling groups if you are using a launch template. In a launch template, you can specify different allocation strategies to ensure high availability of your application. These strategies would ensure that if an instance type is not available, another instance type depending on your allocation strategy would be launched.

2. Graceful shutdown of an application when the instance is about to terminate so that error rate doesn’t increase.

You can get to know if a spot instance is about to terminate before 2 minutes by making a simple curl request periodically (read more here). This was a little tough to solve for existing applications running on EC2 instances because that would have required changes in all of our ASGs. Another option was to use a managed service like SpotInst to do the work for you.

For Kubernetes (which is most of our production), it turns out that it’s a lot easier to do spot instances than EC2. AWS and the DevOps community has built awesome tools that make it super easy. You can use aws-node-termination-handler or kube-spot-termination-notice-handler.

It is a good idea to set up Pod Disruption Budgets for your deployments to ensure that there is least possible disruption. Today, we just change the percentage of spot instances in our node group’s ASG as and when we need to tune our spot spread.

This approach has helped us run all of our stage Kubernetes workloads and a significant portion of our production Kubernetes workloads on spot instances. In production, 99% of our workload is covered under spot, reserved instances and savings plan. Only 1% workload runs on-demand, mostly because of the difficulty of managing spot coverage to avoid RI or savings plan wastage because of our varying workloads.

Nightly shutdown of stage infrastructure

All of our developers are in the same time zone, so our stage environment is not in use at night. We were already stopping EC2 instances in the stage environment at night. On Kubernetes, we have an internal tool that helps every developer create their own namespace and set up a personal dev environment. We set replicas for all deployments and stateful sets to 0 at night. And if there hasn’t been any change in the namespace for 5 days, we delete them. This saves at least 35% of our stage costs. Also, we get a nice side effect of ensuring that our stage environments are repeatable and re-creatable through code and automation.

Nightly scale down in production

Since our operations are based only in India, we don’t receive significant traffic at night. We decrease the number of instances that we run for a service at off hours. We were already using scheduled actions on Autoscaling Groups on EC2. On Kubernetes, we needed something similar for HPA for our services running in our Kubernetes cluster. kube-schedule-scaler is capable of doing this. We use a modified fork of kube-schedule-scaler. Developers can specify the scale down schedule by just putting a simple annotation on HPA objects. When the services scale down, the cluster autoscaler takes care of reducing the size of node groups. You can also explore kube-downscaler for a similar functionality.

Here is an example of what an HPA with a scheduled action looks like:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    kube-schedule-scaler/schedule-actions: '[
      {"schedule": "0 19 * * *", "minReplicas": "7"},
      {"schedule": "30 1 * * *", "minReplicas": "20"}
    ]'
  name: my-hpa
  namespace: my-namespace
spec:
  maxReplicas: 100
  minReplicas: 20
...

Resource optimisation for Kubernetes dev cluster

In Kubernetes, you can set requests and limits for a pod. In production, the difference between limits and requests should be small else you might see performance issues (read more on this here). But in the case of the stage environment, the workload is generally not CPU intensive. So, it doesn’t make sense to keep resource requests very high. It’s a little tough to do this for all your services, so we created a Kubernetes controller which, depending on namespace labels, decreases the requested CPU of containers but keeps the limits at the same value to allow some peaks for applications.

Reducing requests to minimum while keeping limits high enough helped us reduce number of cores by 70%.

We faced a few challenges around resource management — you can read about that here.

Reducing EBS cost

We use a mix of magnetic and GP2 volumes. We try to use magnetic volumes wherever we know that we won’t need much IO. Most of our workload using EBS volume is stateless and were used as root volumes in EC2. We didn’t have a good policy around EBS size selection while launching new EC2 infrastructure which led to a lot of wastage of EBS (most instances using as high as 20% of provisioned storage). The solution here is simple but needs a lot of manual work (some of it can be scripted out as well):

1. Recreate all AMIs and update the launch template to provision less only necessary EBS capacity.
2. Fix EBS provisioning policies and set good alerts on usage to solve this in the long term.

Stage ELB costs

We were using ELB to expose the services and provide an endpoint for local testing. But this resulted in extra ELB costs and it was a big inefficiency in our setup. We decided to replace all ELB with Nginx ingress to expose services. We wrote an in-house Kubernetes controller to create a NodePort service backed by Nginx ingress for every request to create an ELB based service. This did not require any change in our service manifests and drastically reduced our ELB costs.

S3 Costs

S3 is quite cheap for data storage but without proper handle on how you are using S3, the costs can keep increasing steadily with time if you haven’t configured your lifecycle policies for your buckets. We have hundreds of buckets. To get the cost attribution report for S3, we used AWS Athena to query our CUR reports and to filter out the buckets for which we were getting charged the most for. After identifying the costliest buckets, we found a few common problems:

1. No lifecycle policy. We don’t retire/purge any data which adds up over months and years.
2. If storing backups, backups were not compressed.
3. Logs are dumped but not archived.

You can set up a lifecycle policy on your bucket to change the storage type of your files or remove them after some time. Read this post by AWS for more details.

We’d like to especially call out the intelligent tiering option which can help reduce storage costs by a great deal.

Reserved Instances

AWS offers Reserved Instances (RIs) for many of its services (like RDS, Elasticache, Redshift, etc.). If you know that a resource is going to be there for a year or more and wouldn’t require changes in its instance type, it would be better to buy a RI for that instance. AWS also supports converting RIs before expiry of purchase contract for some of their services. Explore those options before making your purchase.

Buying a lot of RIs can also be problematic if your RI Utilisation decreases because that would be pure wastage. You can use AWS Cost Explorer to track this. Some 3rd party tools can also be used for setting up alerts.

Here is what we learnt

By undertaking these initiatives, we reduced our costs by almost 40%. The pandemic made us push our limits to optimise cost. It pushed us to put better governance of infrastructure and engineer solutions that fundamentally changed how we manage infrastructure.

Here are a few lessons that we want to summarise:

1. Resource Tagging — having resource tagging helped us dive deep into our AWS bills and squeeze out wastage as much as possible. We recommend every team should figure this out early in their journey to avoid this debt
2. Partner With The Finance Function — your company’s finance function can help optimize costs a lot by asking the right questions and providing a business framework to look at cloud costs. Use their expertise to your advantage to model your infrastructure costs better.
3. Regular Cost Review — most teams do once in a blue moon cost optimization activity. That’s fine in the initial stages of your company. In later stages as the business grows, being able to regularly review costs becomes important to prevent slippage and keep costs under check. We do this as a monthly activity jointly with our finance team.
4. Monitor Data Transfer Cost — incorrect decisions can creep in and data transfer costs can keep increasing if you don’t have a handle on it.
5. Set Good Governance Policies — If you put too many restrictions and have your “devops engineers” manage everything, then it’s nothing more than more expensive on-premise infrastructure. The cloud must enable your team to move fast but at the same time how do you control costs when you open it up to your engineering teams? We give a lot of freedom to our developers to use AWS. We missed covering some important aspects of governance such as EBS storage, ELB fixed costs, S3 storage, data transfer, etc. Thinking about governance of some of the most commonly used AWS services will go a long way in controlling cost and keeping a good security posture.
6. Kubernetes Governance — Kubernetes is hard to introduce and migrate to but it’s even harder to govern because all the angles of governance are not fully covered by available solutions today. You can find some open-source solutions or you can build some using APIs. Do not overlook it if you are heavily dependent on it or plan to make it central to your infrastructure strategy. We are also exploring a new tagging strategy for kubernetes resources and kubecost to monitor costs.

There are some initiatives that need their own detailed blog posts. We hope to write about them soon and share with you. We hope you found our experience of cost optimization useful.

Vishesh Jindal is an engineer on the Infrastructure Engineering team at Grofers.

Thanks for reading Lambda.

Say hello on Twitter or follow us on LinkedIn.

We’re hiring!

We are hiring across various roles! If you are interested in exploring working at Grofers, we’d love to hear from you. You can either apply on LinkedIn or directly reach out to the author on Twitter or LinkedIn.

How COVID-19 pushed us to reduce our AWS cost by half was originally published in Lambda by Blinkit on Medium, where people are continuing the conversation by highlighting and responding to this story.

Auto-Scaling Groups (ASG) in AWS EC2 provide a really easy way to scale your services horizontally based on different kinds of alarms. It launches instances using existing AMIs baked by you for your services.

While ASGs are easy to understand at a high level, they have their own challenges — a fairly difficult one being how to bring the latest code, packages and configuration into the AMI. You can automate building AMIs but it quickly becomes challenging to build AMIs for code bases in a micro-services architecture that can potentially be updated every hour.

How did we solve it

If you have been following our blog, you would know how much we love Ansible. Our entire engineering team uses Ansible for setting up and deploying services. So our natural inclination was towards using Ansible for simplifying our auto-scaling setup as well.

Any EC2 instance in our infrastructure goes through at least two separate runs of Ansible:

1. First playbook for basic setup of the instance (installing basic common tools like consul agent, rsyslog, collectd, etc.)
2. The second one is used to setup the service built or being setup by development teams.

Having these two run on a new EC2 instance would make sure that the service is ready for use.

When a new instance is added in an ASG, we want the above two things to happen, but slightly differently to start the services essential for business as fast as possible.

Introducing Bootstrap

Bootstrap is our internal tooling that developers use to prepare easily maintainable AMIs for their services which helps them bring their services up as fast as possible in the latest desired state.

Written in Bash, it is basically a wrapper on top of Ansible that gives the flexibility to configure how to execute playbooks using a simple configuration. When AMI is baked with bootstrap configuration for a service, bootstrap will run Ansible on startup to bring up that service in the desired state. This way you have to rebake your AMI only when you want to change your bootstrap configuration. There is no need to rebake AMIs when code or ansible playbooks change.

A bootstrap configuration file includes information like ansible repository, branch, playbook name, and some other stuff required for us to run execute the ansible playbook. Using this information, the playbook is executed locally on the instance. We have our own ansible module which places the file with the required values. All that needs to be done to setup an AMI is:

- name: Add bootstrap file for consumer web
  asg_bootstrap:
    ansible_git_repo: git@github.com:grofers/ansible-repo.git
    ansible_git_branch: master
    ansible_playbook: my_playbook.yml
    ansible_roles_requirements: roles.yml
    ansible_playbook_args: "-t tag1,tag2"
    file_name: 00_setup_abc
    state: present

1. Create a task in your ansible playbook to create bootstrap’s configuration file using our custom ansible module called asg_bootstrap. The task would look something like above.
2. Run ansible playbook on the new instance and create the AMI out of it.

How does it work

When an instance starts, there are a bunch of tasks that need to be performed like tagging of the instance, configuring rsyslog, collectd, etc. But during traffic peaks, we can’t wait for services like collectd and rsyslog agent to come up before our business critical backend services.

To solve this, bootstrap allows you to set priority of tasks so that you can choose any critical setup necessary to be done before business facing service can be started. We have a directory structure like this for the configuration files

./
|── bootstrap.d/
|   ├── prerun/
|   |   ├── 00_tagging
|   |   ├── 01_set_hostname
|   ├── 00_setup_abc
|   ├── 01_setup_xyz
|   └── postrun/
|   |   ├── 00_setup_common

Configuration files get their name from the file_name argument of the asg_bootrap module as shown in the code snippet above. Bootstrap picks up configuration files in bootstrap.d/prerun directory first, then in bootstrap.d/ directory at the root level and finally in bootstrap.d/postrun directory. We prefix the configuration files with numbers so that we can easily control the order of execution. The order of execution for the configuration files in the above example will be like this:

1. prerun/00_tagging — Responsible for setting the Name tag for that instance
2. prerun/01_set_hostname — Responsible for setting FQDN on the basis of the Name tag
3. 00_setup_abc — Responsible for setting FQDN on the basisof the Name tag
4. 01_setup_abc — Responsible for setting up business facing critical service XYZ
5. postrun/00_setup_common — Responsible for setting up common tooling like rsyslog, collectd, etc.

Challenges & fixes

Setting dynamic Name tags for identifying instances

A single service can have a lot of instances and every instance must be uniquely identifiable, meaning it should have a unique Name tag. ASG doesn’t have support for dynamic Name tags for new instances — every new instance started in an ASG would always get the same Name tag. Our 00_tagging script solves this by querying a the instance’s key in Consul to get the last index number for that service, increments the index number, update the key (using CAS operation) with the new index number in Consul and retag the instance with the appropriate Name tag.

Missing Tags

Name tags are required by our Ansible’s dynamic inventory script to function properly. We started seeing intermittent failures because AWS’s autoscaling sometimes takes time to tag the instance and is not deterministic. So at the time of Ansible running on instance start, it is possible that the name tag is not present on the instance. To fix this, if the instance doesn’t have a name tag, we check which ASG the instance is attached to, directly get the Name tag from there and forcefully apply it.

AWS API Limit Exceeded

We have our own dynamic inventory script which gets the instances from AWS and creates groups on the basis of their tags. This resulted in us hitting AWS API’s limit resulting into failures. To reduce the number of AWS API calls, bootstrap script sets some environment variables (like ip address, name tag, etc.) which are used by ansible instead of dynamic inventory script to create groups at its end.

Changes in bootstrap script

It is a little troublesome if there are changes in bootstrap itself. We have to bake a new AMIs for every service with a new bootstrap script. One of the ways this can be fixed is to keep the script at a remote location which could be fetched and executed when the instance comes up. This problem is still unsolved for us but having iterated on bootstrap enough number of times has got us to a point where this is not a very big problem for us.

Conclusion

Though we faced some challenges in the beginning, this setup has helped us achieve auto-scaling without the overheads of having an AMI build pipeline and in giving developers the right kind of tooling that enables them to quickly implement auto-scaling with a lot of control, improving the overall DevOps culture in our team.

If you like to solve similar problems at scale, we are always looking for new talent. Check for open positions here.

Read our other blog posts in this series:

1. Ansible at Grofers (Part 1) — Introduction
2. Ansible at Grofers (Part 2) — Managing PostgreSQL
3. Ansible at Grofers (Part 3) — Managing EC2 Instances

Ansible at Grofers (Part 4) — Manageable auto-scaling with Ansible was originally published in Lambda by Blinkit on Medium, where people are continuing the conversation by highlighting and responding to this story.

After a Friday night party, we found out that someone had accidentally put AWS Access Keys in a public repository (thanks to AWS Security Team for reporting this). Keys didn’t have much access and were of a test environment, but this could have happened with any other key pair with higher privileges. We immediately deactivated them and got the keys rotated everywhere they were being used.

This was not the first time. Things like these have happened in the past with us. Asking friends in the industry reveals that this is a common issue with many engineering teams. Analyzing past incidents like these we learned that whenever this has happened at Grofers, they usually happened because we somehow missed the implications of something we did while trying to build a cool experience for our users. And this can happen to anyone who ends up mishandling “secrets” secret because of a mistake or lack of awareness.

We understood that a big part of the problem is education and started thinking how can we educate our team better but even with all best efforts put in place, things like these can still happen. We realized that education alone is not the solution and we need to put better guards in place — like ones that the AWS Security Team has put for its customers.

How we secured it

We started trying to figure out a way to make AWS keys more secure. In IAM docs (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html#policy-vars-infotouse), we came across sourceIP and ec2:SourceInstanceARN conditions.

We have two kinds of users on AWS:

1. Application users — applications using AWS APIs
2. Human users — members of our engineering and product teams using AWS CLI

The only thing we could use was aws:sourceIP. It would be okay for the developers when they are in office, but that would stop them from using keys from their home or anywhere else. We want our developers to have the flexibility of working from anywhere.

We already have a VPN setup to access all of our infrastructure. So we thought why not route all requests to AWS APIs through our VPN itself. This will make the requests AWS APIs with our VPN instance’s IP as the source. Our VPN server has support to push routes to clients when a connection is established. So, we route all requests falling in AWS IP range (https://ip-ranges.amazonaws.com/ip-ranges.json) through our VPN. Now, every request that AWS receives comes through our VPN infrastructure and has the aws:sourceIP of our VPN servers. So, we could now use a policy like this:

"Condition": {
  "IpAddress": {
    "aws:SourceIp": [
      "203.0.113.0/32",
      "23.0.123.0/32"
    ]
  }
}

This restricts usage of AWS Access Key from our VPN only.

We have more than 100 IAM users and more than 300 IAM policies. Some policies are in use by both human users and application users. It’s cumbersome to create new policies and update all the policies to add aws:sourceIP conditions.

According to the documentation here, adding an explicit deny will deny all the requests if it matches that condition.

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Deny",
    "Action": "*",
    "Resource": "*",
    "Condition": {"NotIpAddress": {"aws:SourceIp": [
      "192.0.2.0/24",
      "203.0.113.0/24"
    ]}}
  }
}

Ref: AWS IAM Docs

So we created the above policy and mapped it to IAM group which contained all the human users.

As for application users, we needed a similar thing. Since most of our instances are private, the sourceIP was the internet gateway’s public IP. So we used these IPs for the application users.

We created two IAM Groups, human-users and application-users. And added all the IAM users to these groups according to their type.

Conclusion

Now AWS APIs can be accessed over VPN by human users and from private EC2 instances by application users. This prevents the use of application user’s credentials by human users and vice versa and makes sure that no IAM user’s credentials can be used by anyone outside our private network.

If you are interested in solving problems like this, we are hiring for all roles. Come join us.

Securing AWS Access Keys was originally published in Lambda by Blinkit on Medium, where people are continuing the conversation by highlighting and responding to this story.