As a pragmatic Engineering leader, I prioritized completion over perfection and waited for the opportune time to start optimizing. While optimizing the application for performance and scale, I also decided to optimize for AWS cost.

Using the techniques listed in this article, I reduced AWS bill for these two products by 70%.

TLDR; Measure (your spending), Prioritize (the changes that deliver biggest bang for the buck), Execute( rinse and repeat) and Optimize (your architecture to further squeeze the juice) .

“You cannot improve what you cannot measure”

What should you measure?

Before you start measuring the impact of using an AWS service, you need to identify and map the AWS services that you are using to the cost centers that they should be accounted against. AWS natively provides tagging as a way to add metadata to your AWS resources.

A common tagging scheme is shown below:

Product Code — The product that is utilizing an AWS resource e.g. Checkout , Accounts, Catalog etc.

Environment — The environment that this service is being used in. Helps you keep costs down for non-prod environments. e.g. prod, preprod, qa, dev et.

Cost Center — The Cost center that is paying for the resources. Not so useful in smaller orgs but might become useful in larger companies with multiple cost centers each responsible for their own spending.

Now that you have a way to identify services and map them to internal products, environments, and cost centers, let’s start measuring the financial impact of using those services.

There are two important metrics to consider when trying to optimize for cost: utilization and cost.

Utilization tells you the efficiency at which you are running the service. If you’ve provisioned a c5.xlarge EC2 instance but running it at 10% CPU utilization then you may have over provisioned your instance type. Mind you, this is a simplistic example. You should look at your CPU, memory and network throughput needs to choose an EC2 instance type.

Cost tells you how much you pay for that over provisioned EC2 instance. AWS has the information available in a variety of places but estimating the actual spending is near impossible if you just consider the reference price points on AWS. You need a dashboard that shows you the actual spending you accrued over a period of time.

While you may be spending $20 a month on Cloudwatch, it’s the $2000 EC2 bill that needs more attention.

How should you measure cost?

AWS provides a few tools to measure the cost of running their services. Based on the size of your organization and the way the AWS accounts are set up, you may not have access to the AWS billing or Cost Visibility tools.

AWS Trusted Advisor gives you the visibility into unused and idle instances.

Cost and Usage Report is a more detailed report by service, time and by tags that you’ve defined. The data is available in an S3 bucket and is a great resource to build your own dashboards on.

At Ticketmaster we have restricted access to the billing account and could not utilize the standard AWS tools. The company did invest in an Enterprise Cloud cost optimization tool called Cloudability that was pivotal in the analysis and the subsequent optimization.

Here’s the blurred out Dashboard for one of the products.

I promise I am coming to the actual cost optimization parts.

Prioritize

• Unused Instances

The biggest advantage of a cloud provider is the ease of spinning up services that you need. The buffet of services that AWS offers can be tempting and engineers love using the latest and greatest. And sometimes they forget to clean up after their little experiment.

We found a 14% savings from deleting unused or underutilized instances.

These included RDS instances with zero connections, Elasticache Redis replicas with no clients and EC2 instances of deprecated environments.

EBS Volumes continue to accrue cost when the EC2 instances are in the stopped state. Go ahead and terminate the EC2 instances that you are done using.

• Right Sizing

AWS provides a plethora of options for every service or instance type letting you choose from memory, vCPU and Network throughput.

Reducing a cache.r4.xlarge to cache.r4.large is a 50% savings in Redis usage costs.

You’ll be surprised at what a t3a.medium can run.

Actionable Insight: Make AWS work for your money. You could throw a small office party with the money saved on unused and underutilized resources .

Do you need to pay the AWS premium for open source solutions?

AWS ElasticSearchService costs about 30% more than running it on your own EC2 instances. Redis, Kafka, Kubernetes are other open source tools that AWS provides as a managed solution.

The managed solutions are a great way to get started and for most companies to use in production long term. At Ticketmaster’s volume of traffic, the AWS services got really expensive really fast.

ElasticSearch was core to running the API and the team was highly skilled in operating the cluster. We were confident that we can manage our own cluster with a little bit of work to achieve the flexibility offered by ElasticSearchService.

20% Savings unlocked. Ka-ching!

We made additional architectural changes that may or may not apply to your specific use cases:

1. Review your mappings and assess if you need to index every single field in the document.
2. Look for write time optimization to reduce query time load.
3. Precompute and make the data available in S3 or Kinesis to the non-real time users of your data.

Actionable Insight: Take a hard look at the managed AWS services in your stack and determine if you can run it yourself.

Are you really Elastic?

When we evaluated our traffic patterns, most of the traffic was between 6 and 10 a.m when we released new inventory and between 5 and 8 p.m. when sporting events or shows were playing off. The rest of the day was about 40% lower traffic and nights being 70–80% lower than the day.

We ran load tests modeling different times of the day, days of the week, and month and arrived a baseline number to run these two applications.

The baseline capacity was moved to Reserved instances. For the traffic variations through the day, we created a scheduled scale up and scale down. Unfortunately, AWS autoscaling doesn’t scale fast enough for the traffic spikes we experience during the morning rush and we had to opt for pre-warming the clusters.

We moved EC2, RDS and ElastiCache to Reserved Instances and saved 70% on those specific instances over a period of a year.

Insight: Move to Reserved Instances if you know that you are going to run at a committed capacity for a prolonged duration.

Optimize: Take another look at your architecture.

Now that the obvious savings were achieved, we wanted to squeeze the lemon for a bit more juice. The next step was evaluating our architecture and identifying opportunities to eliminate components, reduce query time computation and prevent more requests from reaching our servers. After all, the most cost effective infrastructure is not infrastructure at all.

The aspirational goal that I set for the team was to serve the entire application from S3. At the time of this post, the team is finding creative ways to get to the goal.

The web application we served was localized to a market with small modules personalized to the user. This gave us the opportunity to leverage cache both at the CDN and the application tiers.

The web application was built on next.js, a node.js backend for frontend (we named it the adapter) and small backend on Apollo’s GraphQL server.

By using GraphQL we were able to make fewer network calls between the client and server. We also implemented a dynamic query caching algorithm on GraphQL that allowed us to cache graphql query responses.

Over time, we migrated all the REST calls to GraphQL calls, implemented dynamic query caching on the edge CDN and reduced the number of network calls.

On the frontend, we moved the static assets from the node.js frontend application to S3. This saved us both Load Balancer bandwidth and EC2 execution times.

Cache

We used Elasticache Redis as a distributed cache to reduce the number of requests to the upstream services. Because Redis needs to contain all the data in memory and we were running at a high throughput, we started out with cache.r4.xlarge instance Type.

As we leveraged edge caching of the GraphQL queries, we required fewer objects to be cached in Redis. Scaling Redis instance type to cache.r4.large (50% cheaper) and removed replicas( 66% reduction in cost) to further reduce cost.

Actionable Insight: Elasticache is expensive to run, when possible move your caches to Proxy or CDN caching. Use GraphQL to reduce network calls. Identify assets that can be offloaded to a CDN or S3. The cheapest server is having no server at all.

Once you start measuring the spending, the most obvious cost savings will be apparent. After going through the easy pickings, you will need to take a deeper look at your architecture and traffic patterns to further optimize your AWS usage.

Follow me on Twitter @muddassar_shk for more on AWS, Application Architecture, and Engineering Leadership.

Ads drive the internet economy.

The Internet remembers everything for ever.

This article will show you a few steps to browse the internet safely, protect your privacy, and protect your data from snooping by ad companies, your coffee shop and your ISP.

1. Change your DNS server

DNS servers help translate a domain into an IP address that your computer, the internet and the servers understand. Typically the ISP either acts as a the DNS or intercepts your requests as it forwards your request to a DNS server.

This data is aggregated by the ISP and sold as demographic dataset to advertisers. Google and Cloudflare offer a DNS alternative to your ISP. You can go to your Windows or Mac settings to configure the DNS server for that device. You can go to your router configuration to change the DNS setting for all devices that connect to your home router.

2. Start using a VPN

Using a VPN resolves all the problems that the DNS solves but for every request instead of just the domain resolution. A VPN( Virtual Private Network) creates a tunnel between your computer and the broader internet. All of your traffic to the internet is through this tunnel.

There are two components to a VPN — a client and a server. The client is a piece of software installed on your computer or phone. The client encrypts your information before it leaves your computer. The server is a machine on the internet that manages your connection, connects you to the address you requested and also acts as a “client” to the requested website. Any website you requested will get the IP of the VPN server you connected to as the client IP. This cloaks your IP and the website is unable to create a profile associated with your home IP.

Choosing a VPN provider is equally important. Just like an ISP, a VPN provider knows all the addresses you are requesting and some VPN providers collect this data to be sold to data aggregators. As the adage goes, if a product is free to use that means you are the product. Look for a VPN provider that does not save data logs.

3. Change your Browser

The most popular browser, Chrome, is owned by the biggest ad company in the world, Google. When you login to your Chrome profile to make it easier to port your bookmarks over to a new device, you are also making it easier for Google to associate all traffic from that browser to your profile.

The Incognito mode in your Chrome browser is the biggest intentional misnomer in the history of naming tech features. The Incognito mode is anything but that. The only value add it provides is deleting your local history. Servers can still identify you by IP or device fingerprinting.

These browsers provide

Safari — Safari is refreshingly different from the other big Tech browser builders like Google and Microsoft. Safari comes with anti-fingerprinting, a private browsing mode that is more restrictive than typical browsers. Safari also upgrade all your connections to HTTPS. Safari also uses machine learning to prevent ad trackers from tracking you across different domains.

Apple still collects your browsing information though. If you are a Mac user and don’t want to download another browser this is your choice.

Brave — The browser that was built with privacy as a core principle. Brave has a built in ad-blocker, anti-fingerprinting controls, anti-tracking controls and a default search Engine as DuckDuckGo.

Blocking all the ads and the pixel trackers also improves your browsing experience by making the pages load faster. The browser is built on the open source browser called Chromium that makes the transition from Chrome extremely easy.

Brave automatically upgrades all requests to HTTPS, blocks extensions that are unsafe, and protects users from malware, and phishing sites.

Brave pays you to view browse and provides the ability to reward content creators from websites that you visit most often.

Tor Project — This is probably the safest of all browsers. Tor Project uses TOR protocol to circumvent almost anyone from identifying the originating request. It considerably slows down your browsing experience because of the multiple hops your requests takes before reaching the destination server and the same traversal on the way back.

You can use Brave with a “Private Browsing with Tor” option to use Tor to navigate to a site. Skip this browser unless you are in a country with repressive laws and trying to hide your browsing history from the government.

Firefox — Firefox started out from the embers of the original Netscape Navigator. Firefox has seen its popularity rise and fall over the years. The latest iteration of Firefox is fast and one of the most secure browsers out there.

Firefox comes with Ad-blocking, blocks third party trackers and comes with a private VPN network to boost.

4. Change your Search Engine —

If you’ve not caught the drift yet, Google makes money off of your browsing data. Using Google products is inevitable if you spend more than an hour online everyday. You can limit your exposure to Google’s all seeing eyes by switching your search engine.

Duck Duck Go and Startpage are two privacy focused search engines that are decent for most searches. Both search engines do not save your searches, do not log your IP and do not sell your search history to advertisers. Both the companies serve ads but those ads are based on the keyword instead of the user’s browsing history.

DuckDuckGo build a brand new search engine and is good for most searches. Startpage used a clever hack and they strip out all the user data before sending it to Google. You are getting Google search results without sending any user identifying information to Google. A win-win preposition.

These tips should help you evade the ad trackers. They will prevent the awkward moment when an ad for a product that you were searching for after work starts appearing on your browser while you are sharing your screen on Zoom. It will obfuscate enough data from the ad companies that your profile may not be marketable to them.

Some common terms used in this article

IP : Short for IP address or Internet Protocol Address. There are two common forms of IP addresses — IPv4 and IPv6. This address identifies a computer or device on a network.

ISP: Internet Service Provider. This is the company that provides your internet connection. AT&T, Verizon and Spectrum are some examples of ISP.