<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[CloudForecast - Medium]]></title>
        <description><![CDATA[Surprised by your Monthly AWS Cost? CloudForecast helps companies monitor and eliminate wasted AWS cost without spending significant engineering time &amp; resources. - Medium]]></description>
        <link>https://medium.com/cloudforecast?source=rss----615927e38fe5---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>CloudForecast - Medium</title>
            <link>https://medium.com/cloudforecast?source=rss----615927e38fe5---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:40:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/cloudforecast" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kubernetes Cost Management and Analysis Guide]]></title>
            <link>https://medium.com/cloudforecast/kubernetes-cost-management-and-analysis-guide-53d3d3d7de2c?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/53d3d3d7de2c</guid>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Wed, 22 Sep 2021 17:20:22 GMT</pubDate>
            <atom:updated>2021-09-22T17:20:22.621Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ZZoVKvKJuyu5RIlgeZnYGg.png" /></figure><p>The popularity of Kubernetes is constantly increasing, with more and more companies moving their workload to this way of orchestration. Some organizations exclusively develop new applications on Kubernetes, taking advantage of the architecture designs it enables. Other organizations move their current infrastructure to Kubernetes in a lift-and-shift manner. While some tools offer native solutions to cost analysis, these can quickly become too simple of an overview.</p><p>Having your workload running in Kubernetes can bring lots of benefits, but costs become difficult to manage and monitor. In this article, we’ll examine the key reasons why cost can be so difficult to manage in Kubernetes. Plus, you’ll gain insight into how you can improve your cost management significantly.</p><h3>Traditional vs. Kubernetes Resource Management</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/0*c_dxdwAMg6EtL9YY.png" /></figure><p>Before diving into cost management, it’s important to first understand how the underlying resources differ. We’ll use the simple webshop above as an example. This webshop contains three distinct components: a frontend service, a cart service, and a product service. The frontend service is responsible for serving everything visually. The cart service is responsible for saving a customer’s order in the database. Lastly, the product service is an API that other services, like the frontend, can query in order to get product information. An actual webshop will naturally be more complicated, but we’ll stick with this as an example.</p><h3>Traditional Architecture</h3><p>Traditionally you would spin up each service on their own pools of VMs, giving these pools the appropriate sizes. This makes it easy to see the cost of each service; you just need to look at the bill. For example, you can quickly figure out the product service is taking up a lot of resources, which you can then start looking into.</p><p>Since traditional architecture has been around for so long, many tools-especially cloud providers-are used to reporting costs this way. This isn’t the case for Kubernetes.</p><h3>Kubernetes Architecture</h3><p>It’s possible to re-create the traditional architecture in Kubernetes with a dedicated node pool for each service, but this isn’t the best practice. Ideally, you should use a single or a few pools to host your applications, meaning the three distinct services can run on the same set of nodes. Because of this, your bill can’t tell you what service is taking up what amount of resources.</p><p>Kubernetes does provide you with standard metrics like CPU and RAM usage per application, but it’s still tough to decipher not only what is costing you a lot, but specifically how you can lower costs. Given Kubernetes’ various capabilities, many strategies can be implemented to lower costs.</p><p>Strategies can involve rightsizing nodes, which isn’t too different from a traditional architecture, but Kubernetes offers something new. Kubernetes lets you rightsize Pods. Using limits and requests, as well as specifying the right size of your nodes, you can make sure Pods are efficiently stacked on your nodes for optimal utilization.</p><h3>Comparing Architectures</h3><p>While Kubernetes offers many advantages over a traditional architecture, moving your workload to this orchestrator does present challenges. Kubernetes requires extra focus on cost; it won’t be possible to simply look at the bill and know what resources are costing a lot.</p><p>With Kubernetes, you should look into using specialized tools for cost reporting. Many of these tools include recommendations on how to lower your cost, which is especially useful. Let’s take a deeper dive into how you would manage cost in a Kubernetes setup.</p><h3>Managing Kubernetes Costs</h3><p>Managing costs in Kubernetes is not a one-and-done process. There are a number of pitfalls that, if overlooked, could result in businesses experiencing higher costs than what they may have predicted. Let’s talk about some areas where you should be on the lookout for opportunities to mitigate costs.</p><h3>Kubernetes Workload Considerations</h3><p>First, understand the nature of your application and how it translates to a cluster environment. Does your application consist of long-lived operations, batch operations that get triggered, are they stateful (ie, databases) or are they stateless?</p><p>The answers to these questions should inform the decision-making process around what Kubernetes objects need to be created. Ensuring that your environment only runs the necessary resources is a key step to cost optimization.</p><h3>Kubernetes Workload Resource Management</h3><p>Once you have a clear picture of your resources, you can set some limits and configure features like Horizontal Pod Autoscaling (HPA) to scale pods up and down based on utilization. HPAs can be configured to operate based on metrics like CPU and memory out of the box, and can be additionally configured to operate on custom metrics. As you analyze your workload, you can further modify the settings that determine the behavior of your resources.</p><h3>Kubernetes Infrastructure Resource Management</h3><p>Managing Kubernetes costs around infrastructure can be especially tricky as you try to figure out the right type of nodes to support your workloads. Your node types will depend on the applications, their resource requirements, and factors related to scaling.</p><p>Operators can configure monitoring and alerts to keep track of how nodes are coping and what occurrences in your workload may be triggering scaling events. These kinds of activities can help organizations save costs related to overprovisioning by leveraging scaling features and tools like <a href="https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler">Cluster Autoscaler</a> to scale nodes when necessary.</p><h3>Leveraging Observability</h3><p>In the same vein as the previous point, your organization can make more informed decisions regarding your Kubernetes cluster size and node types by monitoring custom application metrics (ie, requests per second) along with CPU, memory, network, and storage utilization by Pods.</p><h3>Optimizing Kubernetes Cost with Monitoring</h3><p>One of the main ways to optimize the costs associated with running Kubernetes clusters is to set up the correct tooling for monitoring. You’ll also need to know how to react to the information you receive, and make sure it’s given to you effectively.</p><p><a href="https://cloudforecast.io/kubernetes-eks-and-ecs-cost-management.html">Barometer</a> is coming soon to CloudForecast, which will be helpful for use cases like this.</p><h3>Monitoring Kubernetes Cluster Cost</h3><p>The first things you need to monitor in your Kubernetes Cluster are CPU and memory usage. These metrics give you a quick overview of how many resources your Kubernetes cluster is using. By making sure resources in your Kubernetes cluster are correctly tagged using labels or namespaces, you’ll quickly learn what services are costing the most in your organization.</p><p>The easiest way to monitor these metrics is via automated reporting. <a href="https://cloudforecast.io/kubernetes-eks-and-ecs-cost-management.html">CloudForecast’s upcoming tool</a> will be able to consolidate these reports and deliver them to your team by email or Slack. This ensures each team is aware of how their services are performing, and whether they’re using up too many costly resources.</p><p>Setting up a general overview is highly recommended. Additionally, you should also ensure you get notified if something out of the ordinary happens. For example, you’ll want to be notified if the product service suddenly starts costing a lot more; this allows you to troubleshoot why and work on fixes.</p><p>Kubernetes comes with various different metrics you can use to determine the cost of a specific service. Using the /metrics endpoint provided by the Kubernetes API, you can get a view into pod_cpu_utilization and pod_memory_utilization. With these metrics it becomes easier to see what workloads are drawing what costs. Tools like CloudForecast&#39;s Barometer use these metrics to calculate how many dollars every pod is spending. Having this overview and g_tting a baseline cost of your_ Kubernetes cluster, will help you know when costs are rising too rapidly, and exactly where it&#39;s happening. Knowing how cAdvisor works with Prometheus, and the metrics they collectively expose is incredibly valuable when you want to examine your clusters.</p><p>While there are many metrics that can be analyzed, RAM and CPU are typically the ones you want to focus on, as these are the ones that drive your provider to allocate more resources. You can think of RAM and CPU metrics as the symptoms of your cost. With a proper overview they will allow you to know what workloads are costing you more than normal, and from there you start drilling into the service to figure out why it’s happening.</p><h3>Acting on Monitoring Data</h3><p>Once you’ve been notified of irregularities in your Kubernetes cluster, you need to act. There are many valid strategies for lowering k8s cluster cost. As mentioned earlier, a good first step in Kubernetes is to rightsize your nodes and Pods so they run efficiently. Whatever steps you take to optimize cost, doing it manually is tough.</p><p>Tools automatically suggest why your cost is high and how to reduce it. This allows you to quickly implement cost optimizations, but also uncover solutions that otherwise wouldn’t have come to mind.</p><h3>What to Monitor</h3><p>Tools can help a lot, but they’re wasted without a good foundation. To set up a good foundation, you should determine a set of Key Performance Indicators (KPIs) to monitor. A great KPI example is the number of untagged resources. Having your resources tagged allows your tool reports to be more precise, delivering better optimizations.</p><p>You could also monitor the total cost of your untagged resources. This can act as motivation for getting your resources tagged, and remind your team to maintain a good baseline approach when setting up new resources. Tracking your KPIs before and after the introduction of a tool is a great way to determine how much it actually helps. In any case, determining KPIs will make sure you’re on top of what’s happening in your Kubernetes cluster.</p><h3>How to Develop a Unit Cost Calculator</h3><p>Crucial to understanding your cost is knowing how to use the <a href="https://calculator.aws/">AWS Pricing Calculator</a>. This helps you compare costs associated with running a self-hosted Kubernetes cluster versus with an Amazon EKS cluster.</p><p>The CPU (vCPUs) and Memory (GiB) specified in the following example are just for demonstrative purposes and will vary depending on workload requirements.</p><h3>AWS EKS Cluster Cost and Pricing Estimation</h3><p>The following calculations are for a Highly Available (HA) Kubernetes cluster with a control plane managed by AWS (EKS) and three worker nodes with 4 vCPUs and 16 GiB of memory each. The instance type used in this case is a t4g.xlarge Reserved EC2 instance (1 year period). This instance type is automatically generated as a recommendation based on the CPU and memory requirements that are specified.</p><h4>Unit Calculations</h4><ul><li>EC2 Instance Savings Plans rate for t4g.xlarge in the EU (Ireland) for 1 Year term and No Upfront is 0.0929 USDHours in the commitment: 365 days x 24 hours x 1 year = 8760.0 hours</li><li>Total Commitment: 0.0929 USD x 8760 hours = 813.8 USD</li><li>Upfront: No Upfront (0% of 813.804) = 0 USD</li><li>Hourly cost for EC2 Instance Savings Plans = (Total Commitment — Upfront cost)/Hours in the term: (813.804–0)/8760 = 0.0929 USD</li></ul><p><em>Please note that you will pay an hourly commitment for the Savings Plans and your usage will be accrued at a discounted rate against this commitment.</em></p><h4>Pricing Calculations</h4><ul><li>1 Cluster x 0.10 USD per hour x 730 hours per month = 73 USD</li><li>[worker nodes] 3 instances x 0.0929 USD x 730 hours in month = 203.45 USD (monthly instance savings cost)</li><li>30 GB x 0.11 USD x 3 instances = 9.90 USD (EBS Storage Cost)</li></ul><p>Monthly Cost: 286.35 USD Annual Cost: 3,436.20 USD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pDEzumjPWiUBPEiN.png" /></figure><h3>Self-Hosted Kubernetes Cluster Pricing and Cost Estimation</h3><p>The following calculations are for a custom Highly Available (HA) Kubernetes cluster that is self hosted in AWS, and also consists of three worker nodes with 4 vCPUs and 16 GiB of memory each. Similar to the previous analysis of EKS cluster cost estimations, this analysis will use the same instance type for the same reasons detailed above.</p><h4>Unit Conversions</h4><ul><li>EC2 Instance Savings Plans rate for t4g.xlarge in the EU (Ireland) for 1 Year term and No Upfront is 0.0929 USD</li><li>Hours in the commitment: 365 days * 24 hours * 1 year = 8760.0 hours</li><li>Total Commitment: 0.0929 USD * 8760 hours = 813.8 USD</li><li>Upfront: No Upfront (0% of 813.804) = 0 USD</li><li>Hourly cost for EC2 Instance Savings Plans = (Total Commitment — Upfront cost)/Hours in the term: (813.804–0)/8760 = 0.0929 USD</li></ul><p><em>Please note that you will pay an hourly commitment for Savings Plans and your usage will be accrued at a discounted rate against this commitment.</em></p><h4>Pricing Calculations</h4><ul><li>[control-plane nodes] 3 instances x 0.0929 USD x 730 hours in month = 203.45 USD (monthly instance savings cost)</li><li>[worker nodes] 3 instances x 0.0929 USD x 730 hours in month = 203.45 USD (monthly instance savings cost)</li><li>30 GB x 0.11 USD x 3 instances = 9.90 USD (EBS Storage Cost)</li></ul><p>Monthly Cost: 426.70 USD Annual Cost: 5,120.40 USD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n4ErIPOdtw8DcXPL.png" /></figure><h3>Conclusion</h3><p>By now you’ve learned how Kubernetes architecture differs from traditional architecture. You’ve learned what challenges arise once you start to manage costs in Kubernetes, and how to keep them under control. Features like labeling and namespacing can have a great impact on the traceability of your cost, allowing you to reap the full benefits of a Kubernetes architecture. Also, you’ve learned how using the AWS Pricing Calculator can help you estimate the costs associated with running your workloads on a custom Kubernetes cluster compared to running an EKS cluster.</p><p>Using a tool like CloudForecast’s <a href="https://cloudforecast.io/kubernetes-eks-and-ecs-cost-management.html">Barometer</a> can greatly improve the tracking of cost in your cluster. Barometer not only offers you an effective general overview, it also gives you actionable cost optimization insights.</p><p><em>Originally published at </em><a href="https://www.cloudforecast.io/blog/kubernetes-cost-management-and-analysis/"><em>https://www.cloudforecast.io</em></a><em> on September 22, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53d3d3d7de2c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/kubernetes-cost-management-and-analysis-guide-53d3d3d7de2c">Kubernetes Cost Management and Analysis Guide</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS S3 Pricing and Cost Optimization Guide]]></title>
            <link>https://medium.com/cloudforecast/aws-s3-pricing-and-cost-optimization-guide-5755510110d7?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/5755510110d7</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[amazon-s3]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[aws-s3]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Tue, 02 Feb 2021 21:27:01 GMT</pubDate>
            <atom:updated>2021-02-02T21:25:25.024Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NnpF9mpltO6sb-2kedb8A.png" /></figure><p>Amazon Simple Storage Service (referred to as AWS S3) is AWS’s cloud-based file storage platform. Amazon S3 is one of <a href="https://www.ssi-net.com/the-top-10-most-used-aws-services/">Amazon’s most widely used services</a>, but it’s also one of the easiest to overspend on. If you’re not familiar with AWS S3’s pricing model or taking advantage of lifecycle policies, you’re likely paying more for storage than you need.</p><p>In this AWS S3 Pricing and Cost Optimization Guide, you’ll learn the following:</p><ul><li>Learn about AWS S3’s pricing and cost factors.</li><li>Use AWS S3’s storage class analysis to figure out how much you could save by optimizing your storage classes.</li><li>Set up a lifecycle policy that automatically transitions files to the best storage class.</li><li>Finally, a few general tips for AWS S3 pricing and cost optimization.</li></ul><p><em>Looking for other articles to reduce your AWS Cost? Check out our</em> <a href="https://www.cloudforecast.io/blog/aws-rds-pricing-and-optimization/?utm_source=blog&amp;utm_medium=banner&amp;utm_campaign=s3guide"><em>RDS Pricing and Optimization Guide.</em></a></p><h3>AWS S3 Pricing and Cost</h3><p>Before you start optimizing your S3 buckets, you need to understand Amazon S3’s pricing model. At a high level, you are charged based on <a href="https://aws.amazon.com/s3/pricing/">five factors</a>:</p><ul><li><strong>Storage:</strong> The total size of all files in AWS S3.</li><li><strong>Requests:</strong> Every API call to save or retrieve a file from S3 incurs a cost.</li><li><strong>AWS Data Transfer:</strong> Transferring data into AWS S3 from the internet is free, but you will pay for most other file transfers.</li><li><strong>Management and Analytics:</strong> Amazon offers several S3 management tools for understanding and tagging your files as paid add-ons.</li><li><strong>Replication:</strong> In addition to the storage and transfer fees for replication, you may also need to pay for <a href="https://aws.amazon.com/about-aws/whats-new/2019/11/amazon-s3-replication-time-control-for-predictable-replication-time-backed-by-sla/">S3 Replication Time Control</a>.</li></ul><h3>S3 Storage Classes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t3Msoa5m1eSRyRjJ.png" /></figure><p><em>AWS S3 Six Storage Classes. Source:</em> <a href="https://aws.amazon.com/s3/cost-optimization/"><em>https://aws.amazon.com/s3/cost-optimization/</em></a></p><p>The amount you will pay for each of these factors depends on your S3 storage class. Using the correct S3 storage class is the easiest way to help control AWS S3 cost, so it’s important to start here. You can set the S3 storage class at the S3 bucket or individual object level.</p><h4>S3 Standard</h4><p>The Standard tier is the default class, and is the best option for most frequently accessed files. Retrieving files will take a matter of milliseconds, but this tier is the most expensive for ongoing file storage.</p><h4>S3 Intelligent-Tiering</h4><p>If you work with files over 128 KB, you can use AWS S3 Intelligent-Tiering to automatically classify your files into the best access tier based on their usage. It monitors your files and moves infrequently accessed files to Infrequent Access or Glacier tiers for you. Files smaller than the 128 KB minimum will be kept in the Standard tier for frequent access when using Intelligent-Tiering.</p><h4>S3 Infrequent Access</h4><p>For files that you need less frequently, you can cut your storage costs in half by putting them in S3 Infrequent Access (aka S3 Standard-IA) or One Zone-Infrequent Access tier. While these tiers offer millisecond access time, you will pay about twice as much for data retrieval as you would in the Standard tier.</p><p>The S3 Infrequent Access tier offers redundancy, making it good for backups or inactive account data that you might need to retrieve occasionally and don’t want to lose.</p><h4>One Zone-Infrequent Access</h4><p>The S3 One Zone-IA tier is 20 percent cheaper than the Standard-IA, but offers no redundancy. This makes it less useful for critical data like backups, but it’s appropriate for storing infrequently used data that you can recreate from the original source. For example, you might use One Zone-IA to store thumbnail copies of inactive user profile pictures.</p><h4>AWS S3 Glacier</h4><p>AWS S3 Glacier Archive tier offers resilient storage at a fraction of the price of the Standard tier but requires you to settle for one minute to twelve-hour retrieval times. You’ll have to pay more to retrieve your data and there’s a ninety-day minimum time for storage. These limitations make Amazon Glacier appropriate for things like storing original versions of large media files that are distributed in an encoded format. You might occasionally need to retrieve the originals, but likely not more than once per month.</p><h4>S3 Glacier Deep Archive</h4><p>For an even deeper discount on your storage rate, you can try S3 Glacier Deep Archive. It takes twelve hours to retrieve data from a Deep Archive, and files must be stored for 180 days at a minimum. As your data is even more expensive to retrieve, this tier is most appropriate for compliance data you are required to keep, but only need to access once or twice per year at most.</p><h3>S3 Pricing by Region</h3><p>Another way to influence pricing in Amazon S3 is to select the cheapest region that makes sense for your application. Some AWS regions are <a href="https://www.concurrencylabs.com/blog/choose-your-aws-region-wisely/">as much as 50 percent cheaper</a>, but you have to be careful if you pick a region purely based on price.</p><p>First, latency between your S3 storage and other AWS resources could be a problem. If most of your infrastructure is hosted in South America, but your S3 files are in Virginia, it could add precious milliseconds to each API call.</p><p>Similarly, putting your S3 region far from your users might degrade performance if file access is a critical part of your application. That said, if you use S3 for longer-term storage and cache most of your files in a CDN like <a href="https://aws.amazon.com/cloudfront/">CloudFront</a>, users might not notice the difference between S3 regions much.</p><p>Finally, if you’re a very light user of Amazon S3 or you’re just trying it out, it’s worth noting that AWS offers up to 5 GB of storage for twelve months as part of their <a href="https://aws.amazon.com/free/">Free tier</a>. This might be helpful for early-stage startups or side projects looking for a free way to get started with S3 file storage.</p><h3>Understanding Your S3 Access Patterns</h3><p>It’s helpful to know about the different storage classes in S3, but in practice, most organizations start by putting everything in the Standard tier. Depending on your usage patterns, you’ll be able to optimize your S3 storage classes after AWS has a few weeks of data to analyze.</p><p>In this section, I’ll show you how to use the <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/analytics-storage-class.html">Amazon S3 storage class analysis</a> to determine which Standard tier files can be moved to a less frequent access tier. This will help you decide whether it’s worth investing in a S3 lifecycle policy and how much you might save by using one. It could also be used to <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html">tag objects in S3</a> and move them to the right tier accordingly.</p><h3>Running AWS S3 Storage Class Analysis</h3><p>You can run storage class analysis on an entire bucket or a subset of your S3 files using prefixes or tags. To configure the analysis, open your S3 bucket and click the <em>Metrics</em> tab.</p><p>Scroll down and click <em>Create analytics configuration</em>. From here you can decide whether to analyze the whole bucket or just a subset of files. You can also decide whether to export the results or not.</p><h3>Understanding the Results</h3><p>AWS will start the storage class analysis, and depending on the size of your bucket, you might be able to see results fairly quickly. Here’s an example of what the results might look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8pwZgmbGM79fX7ns.png" /></figure><p>The chart and summary boxes in the image show you the total size of the objects in Amazon S3 that you are storing versus the amount that’s been retrieved. These results show you that even many of the older files in this bucket are being retrieved regularly. While this bucket only contains 13.92 GB of files over a year old, 40.84 GB have been retrieved in the past seven days.</p><p>On the other hand, the analytics in the following image show a very different usage pattern:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PVwgPvmERo0mWvLl.png" /></figure><p>In this bucket, none of the files over thirty days old have been accessed at all in the past seven days. This might indicate that you can move some of these older files to an Infrequent Access tier or even Glacier. You’ll have to be careful, though, as moving files in and out of these longer-term storage patterns incurs a cost.</p><p>Once the storage class analysis is complete, AWS will offer some recommendations as well. Here’s an example from another bucket:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*WqV0mqHTE_TasHXJ.png" /></figure><p>This shows you that the rate of retrieval (the solid orange line) drops to almost zero after a file is thirty days old, and that there is a lot of storage being consumed by older, infrequently accessed files in this bucket.</p><p>Now that you know you <em>can</em> use a better Amazon S3 storage tier, what do you do with this information? That’s where lifecycle rules come in.</p><h3>Creating an S3 Lifecycle Rule</h3><p>By creating a lifecycle rule, you are instructing Amazon S3 to move files to another storage tier or delete files automatically based on their age. These rules can be applied to all files in your bucket or you can use tags or file prefixes to filter files first.</p><p>To get started, go to the <em>Management</em> tab in your S3 bucket and click <em>Create lifecycle rule</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2nXCmcGGwMXcWCK4.png" /></figure><p>For the second example above where no files older than thirty days are being accessed, you could create a lifecycle rule that:</p><ul><li>Moves files to the Standard-IA tier after they’re forty-five days old (this gives you a little bit of margin for error, but you could probably use thirty days if you like)</li><li>Moves files to Glacier after they’re 365 days old</li><li>Expires files after they’re ten years old</li></ul><p>In this case, your access rule setup would look like this in the Amazon S3 GUI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2hUo205sC8_LVi7r.png" /></figure><p>These rules can also be <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html">created in XML and applied using the S3 API</a> if you would like to automate this process to apply policies to many buckets at once. Be careful when setting up these policies, though. Moving files in and out of Glacier frequently will get very expensive, so a small mistake in your lifecycle policy could add thousands to your AWS bill. One way to avoid this is to <a href="https://cloudforecast.io/">sign up for CloudForecast’s daily AWS spending alerts</a> to help monitor cost. In addition, CloudForecast’s <a href="https://www.cloudforecast.io/kb/docs/getting-started/zerowaste/">ZeroWaste Health Check report</a> helps you quickly identity all your S3 buckets that do not have a lifecycle policy attached.</p><p>Once you’ve set up the appropriate policies, your S3 files will be automatically moved to the desired storage tier as they age. This is one of the best ways to lower your Amazon S3 bill, but it’s not the only strategy you can take.</p><h3>AWS S3 Cost Optimization Tips</h3><p>Depending on the size of your organization and the way you’ve stored data in Amazon S3, access patterns and lifecycle policies might only get you so far. If you’re taking a deep look at optimizing your S3 cost, here are a few more things you can explore.</p><h4>Prioritize the Biggest Wins First</h4><p>If your organization manages a lot of data in Amazon S3, going through the storage class analysis, checking with the appropriate teams, and implementing lifecycle policies for every set of data can be overwhelming.</p><p>Start with the areas where you can get the biggest wins first and invest time in them accordingly. If you’re only storing a few gigabytes in a bucket, it might not be worth setting up a lifecycle policy at all.</p><h4>S3 Bucket Organization</h4><p>Because lifecycle policies can use file prefixes or tags as a filter, it’s vital that you implement an organization system across your S3 buckets. Common tagging strategies might include:</p><ul><li>Prefixing file names with an expiration date</li><li>Tagging objects with the responsible team’s name</li><li>Adding user ID’s to the front of file names so you can archive old user data</li><li>Tagging data with each customer’s company name</li></ul><p>Having a consistent system for organizing objects in Amazon S3 will ensure that you can write effective lifecycle policies, and it will prevent you from deleting important files.</p><h4>S3 Usage Reports</h4><p>If you want a granular look at how each file in your S3 bucket has been created, modified, or accessed, you can <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/aws-usage-report.html">download an AWS usage report</a>.</p><h4>Bulk Retrieval and Storage</h4><p>If you decide to move data into or out of Glacier, you might want to <a href="https://aws.amazon.com/glacier/faqs/">consider zipping your files first</a>. This will help you save money on your data transfer and make retrieving files that are likely to be needed at the same time easier. The time to compress your data is negligible compared to the five minute access time Glacier imposes.</p><h4>Partial File Uploads</h4><p>If you use Amazon S3 for multipart file uploads, be sure to remove unfinished parts from your buckets. Amazon provides <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-stop-incomplete-mpu-lifecycle-config">a lifecycle policy for this</a> in their documentation, and it’s an easy way to make sure you’re not paying for files you don’t need.</p><h4>Don’t Forget Your Logs</h4><p>If you’re storing <a href="https://aws.amazon.com/cloudwatch/">CloudWatch</a> logs in Amazon S3, one of the easiest ways to decrease your S3 cost is to expire (ie, delete) old log files automatically. You can do this manually or using a lifecycle policy (as mentioned above), but be careful. Many regulations <a href="https://knowledge.digitaledge.net/newsletters/log-management-retention-requirements/">require organizations to retain logs for months or years</a>.</p><h3>Conclusion</h3><p>Amazon S3 is one of the most widely used AWS products, and because it has such a wide array of use cases, it can be especially hard to understand its pricing. After you get a handle on the different storage tiers available, you can start to dive into your S3 buckets.</p><p>Run the storage class analysis tool to understand your usage, and create a lifecycle policy that properly transfers your S3 files to the appropriate storage class. Keep prioritization and organization in mind, though, because S3 pricing optimization is an ongoing process.</p><p>If you’d like help understanding and keeping up with your AWS spending, <a href="https://www.cloudforecast.io/">CloudForecast</a> can help. Reach out to our CTO, <a href="/service/mailto:francois@cloudforecast.io">francois@cloudforecast.io</a> to help reduce your cost or schedule a time with us.</p><p><em>Originally published at </em><a href="https://www.cloudforecast.io/blog/aws-s3-pricing-and-optimization-guide/"><em>https://www.cloudforecast.io</em></a><em> on January 27, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5755510110d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/aws-s3-pricing-and-cost-optimization-guide-5755510110d7">AWS S3 Pricing and Cost Optimization Guide</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS RDS Pricing and Optimization Guide]]></title>
            <link>https://medium.com/cloudforecast/aws-rds-pricing-and-optimization-guide-6b9505c6320c?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/6b9505c6320c</guid>
            <category><![CDATA[aws-rds]]></category>
            <category><![CDATA[aws-cost-optimization]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[aws-optimization]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Wed, 18 Nov 2020 17:57:42 GMT</pubDate>
            <atom:updated>2020-11-18T17:59:10.638Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LppLh7RFx-d-rxdOQzDcQw.png" /></figure><p>Amazon Web Services makes getting your data into their <a href="https://aws.amazon.com/rds/">Relational Database Service</a> (RDS) relatively easy. Import costs are free, and you can store up to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html#RDS_Limits.Limits">100 terabytes across all your instances</a>. AWS RDS hosts your relational databases in the cloud, and their engineers handle patching, monitoring, availability, and some security concerns.</p><p>These factors make getting started with AWS RDS easy, but understanding and controlling your costs is another matter entirely. In this article, you’ll learning the following about RDS Cost and pricing, monitoring and finally, ways to optimize your cost.</p><h3>AWS RDS Cost and Pricing</h3><p>Instance usage, storage, I/O, backups, and data transfers drive the bulk of your AWS RDS costs. Instance usage and storage are unavoidable, but generally, you should minimize them while adequately addressing your needs. AWS RDS offers some I/O and backup capability bundled into the cost of storage, but you might need more. Moving data into RDS from the internet is free, but moving it out of RDS can get expensive.</p><p>In this section, I’ll dive deeper into each of AWS RDS’s pricing factors to help you understand how your usage might affect your monthly bill.</p><h4>RDS Database Engine</h4><p>Amazon currently offers six database engines: <a href="https://aws.amazon.com/rds/aurora">Amazon Aurora</a>, <a href="https://aws.amazon.com/rds/postgresql/">PostgreSQL</a>, <a href="https://aws.amazon.com/rds/mysql/">MySQL</a>, <a href="https://aws.amazon.com/rds/mariadb/">MariaDB</a>, <a href="https://aws.amazon.com/rds/oracle/">Oracle</a>, and <a href="https://aws.amazon.com/rds/sqlserver/">Microsoft SQL Server</a>. Normally you won’t be able to change your database engine, but you can choose to optimize for memory, performance, or I/O.</p><p>The three open-source databases (Postgres, MySQL, and MariaDB) are similar in price. Depending on the size, PostgreSQL instances are five to ten percent more expensive per hour, but PostgreSQL, MySQL, and MariaDB share pricing for storage, provisioned I/O, and data transfer.</p><p>AWS Aurora is Amazon’s proprietary database, so it gets special treatment. AWS offers a <a href="https://aws.amazon.com/rds/aurora/serverless/">serverless option</a>, making it ideal for applications that don’t need to be on all the time, like test environments. Aurora also has a <a href="https://aws.amazon.com/rds/aurora/global-database/">multi-zone backup system</a> that charges per million replicated I/O operations. Storage per gibibyte (GiB) is a few cents more expensive, but if you’re dealing with intermittent usage or need fast failovers and many read replicas, Aurora can save money over implementing these features on other engines.</p><p>Oracle and SQL Server aren’t open-source or owned by Amazon. To accomodate licensing, hourly instances can cost nearly twice as much. You can self-license with Oracle, which brings the cost in line with open-source options. Other fees, like storage and data transfer, match their open-source counterparts.</p><p>While this guide isn’t intended to help you choose the best database engine, it is important to note that pricing varies based on the engine you choose.</p><h4>RDS Instance Sizes</h4><p>Once you select an engine, you have to select an RDS <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html">instance size</a> with the appropriate computational (vCPU), network (Mbps), and memory capacity (GiB RAM). RDS offers instances ranging from db.t3.micro (2 vCPUS, 1 GiB RAM, 2085 Mbps) to db.m5.24xlarge (96 vCPUS, 384 GiB RAM, 19,000 Mbps).</p><p>Selecting the right RDS instance size can be challenging. To estimate the RDS instance size you’ll need, estimate or track the amount of data your queries need (called your <a href="http://www.tocker.ca/2013/05/31/estimating-mysqls-working-set-with-information_schema.html">working set</a>), then select an instance that can fit your working set into <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_BestPractices.html#CHAP_BestPractices.Performance.RAM">memory</a>. I’ll touch on RDS monitoring and right-sizing your RDS instances later in this guide.</p><h4>Reserved Instances</h4><p>Without additional configuration, RDS instances are created on-demand. These instances are billed in one-second increments from the moment the instance starts to its termination. You can stop, start, or change an on-demand <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.DBInstance.Modifying.html">instance size</a> at any time.</p><p>The alternative to on-demand pricing is <a href="https://aws.amazon.com/aws-cost-management/aws-cost-optimization/reserved-instances/">Reserved Instances</a> for RDS. You commit to lease an RDS instance for a set period (1 or 3+ years) in exchange for discounts up to 60%. AWS offers sizing flexibility for all Reserved Instance engines except SQL Server and License Included Oracle, allowing administrators to freely change instance size within the same family. If you’re able to commit to RDS for a year or three and have monitored your requirements enough to develop a solid performance baseline, you can save money by trading away the flexibility to turn off or downsize your databases.</p><h4>RDS Storage: Aurora and Autoscaling</h4><p>For most engines, you buy storage per GiB in advance. Aurora is the exception: you only pay for what you use. It’s important to accurately predict your monthly storage needs as you <a href="https://aws.amazon.com/premiumsupport/knowledge-center/rds-db-storage-size/">cannot reduce storage</a> on an instance (<a href="https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-aurora-enables-dynamic-resizing-database-storage-space/">except Aurora</a>).</p><p>AWS can <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html#USER_PIOPS.Autoscaling">auto-scale</a> your storage when an instance has under 10% space remaining for more than 5 minutes. This option has the benefit of keeping storage costs low, but can surprise you if something unexpected happens. To protect against auto-scaling to 65,536 GiB, set a maximum storage threshold for your instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*73XYxcKECN0lYNfG.png" /></figure><p>If you can accurately predict your storage needs, manually provisioning is the cheapest option. If you’re facing unpredictability or unused storage, consider auto-provisioning and focus on maintaining a reasonable storage maximum.</p><h4>RDS Backups</h4><p>AWS backs up 100% of the storage you’ve purchased in any zone for free. If you buy 20 GiB of storage across two instances, it includes 20 GiB of backup space.</p><p>If you need more space for backups, you pay per GiB at a slightly lower rate than regular storage costs. RDS automatically backs up each storage volume every day. These backups are stored according to the backup retention period. Automated backups will not occur if the DB’s state is not AVAILABLE (for example, if the state is STORAGE_FULL). Users can also create manual backups. These never expire and count against your backup storage total.</p><h4>Regions</h4><p>As with most AWS services, RDS costs are specific to a <a href="https://aws.amazon.com/about-aws/global-infrastructure/regions_az/">region</a>. Choose your region carefully because the most expensive regions double the hourly cost of instances, increase storage by a few cents per GiB, and 5x inter-zone data transfer pricing.</p><p>On the other hand, if your database is located further from your application servers, you’ll add latency to every database call. If this latency degrades user experience, you probably don’t want to use a distant, slightly cheaper region.</p><h4>Multi-AZ Deployments</h4><p>If you need availability when an AWS regional data center encounters trouble, you can enable <a href="https://aws.amazon.com/rds/features/multi-az/">Multi-AZ deployments</a>. This creates a backup database instance and replicates your data to a second AWS data center. Be aware that this <strong>doubles your monthly instance and storage costs</strong> but enhances the reliability of critical services. If you need a Multi-AZ deployment, focus on reducing your storage needs and instance size, these gains are won twice.</p><h3>RDS Cost Monitoring</h3><p>Once you understand the many options RDS offers and set up an instance, there are a few tools that will help you audit your current usage and predict future requirements.</p><h4>AWS Cost Explorer</h4><p>The best way to audit your RDS spending is the <a href="https://aws.amazon.com/aws-cost-management/aws-cost-explorer/">AWS Cost Explorer</a>. Activating and examining your daily or monthly spend is an excellent way to visualize your organization’s priorities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LvbTH2LNeLvMRIE-.png" /></figure><p><a href="https://www.cloudforecast.io/blog/aws-tagging-best-practices/">Tagging your resources</a> helps you understand which projects and teams are using which databases. Cost Explorer also offers <a href="https://docs.aws.amazon.com/savingsplans/latest/userguide/sp-recommendations.html">suggestions</a> for using reserved instances based on your past usage.</p><h4>RDS Management Console and Enhanced Monitoring</h4><p>AWS provides a “Monitoring” tab in the RDS Console that displays free-tier CloudWatch metrics like the number of connections and CPU utilization. Keeping an eye on your usage in the console can help you prepare to right-size your storage or purchase reserved instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nvpALJX32MICPoeR.png" /></figure><p>AWS gives you the option to activate additional monitoring services.</p><p><a href="https://aws.amazon.com/rds/performance-insights/"><strong>Performance Insights</strong></a> gathers data about the database load. This tool has its own <a href="https://aws.amazon.com/rds/performance-insights/pricing">pricing model</a> with a free tier that includes 7-day retention.</p><p><a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring.OS.html"><strong>Enhanced Monitoring</strong></a> is stored and priced as <a href="https://aws.amazon.com/cloudwatch/pricing/">CloudWatch logs</a>. It reports metrics from a user agent instead of the hypervisor, allowing you to examine running processes and the OS, which is useful for examining the resource usage of individual queries.</p><p>Finally, you can enable and access <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.html">database logs</a> directly for the price of storing the files.</p><h4>CloudForecast</h4><p><a href="https://www.cloudforecast.io/">CloudForecast</a> supplements AWS’s Cost Explorer through proactive monitoring and optimization reports that keeps your RDS cost in check.</p><p>Through the <a href="https://cloudforecast.io/aws-daily-cost-report-tool.html">daily cost reports</a>, you’ll receive a daily report via email or slack that details your RDS cost in relation to your overall spend and alerts you of any cost anomalies with RDS.</p><p>The <a href="https://www.cloudforecast.io/aws-tagging-compliance-report.html">tagging compliance report</a> helps make sure your RDS instances are properly tagged and let you know exactly which RDS resources are not following compliance.</p><p>Finally, the <a href="https://www.cloudforecast.io/kb/docs/general-info/whatis/#aws-cost-optimization">ZeroWaste Health Report</a> (beta) let’s you know of possible inefficiencies by identifying all your over-provisioned and unused RDS instances in one single report.</p><h3>RDS Cost Optimization</h3><p>Armed with insights into your requirements and RDS’s abilities, it’s time to put cost-saving measures and AWS <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_BestPractices.html">best practices</a> to use. In the rest of this guide, I’ll offer some strategies for decreasing your RDS costs using the insights you gathered above.</p><h4>Right Sizing your Instances</h4><p>In short, turn off anything that’s not being used.</p><p>Every month, you pay for instances and storage and the infrastructure attached to them. You can check each database’s utilization using the connections metric in the RDS Console. On-demand RDS instances can be stopped for up to 7 days. When stopped, you aren’t charged for DB Instance hours, but you are charged for storage. You can use an AWS Lambda <a href="https://docs.aws.amazon.com/lambda/latest/dg/services-cloudwatchevents-tutorial.html">scheduled event</a> and DB Instance API calls to programmatically stop and start instances.</p><p>To practice <a href="https://aws.amazon.com/aws-cost-management/aws-cost-optimization/right-sizing/">right-sizing</a>, act on your monitoring and purchase the machine with the minimum requirements to meet your needs. Multiple RDS instances can also be consolidated into a single instance to minimize costs. This is especially helpful for development environments where a low number of users can access several databases running on a single instance. CloudForecast also makes this easy through our <a href="https://www.cloudforecast.io/kb/docs/general-info/whatis/#aws-cost-optimization">ZeroWaste Health Check Report</a> that reports on all your overprovisioned and unused RDS instances. By getting a full roll up of possible idled resources, you can easily see and turn off anything that is not being used.</p><h4>Database Hygiene</h4><p>Indexing and database sanitation are both important for controlling costs. Proper indexing is important for performance and I/O, as it allows your instance size to remain small and minimizes bottlenecks.</p><p>Removing unused tables, columns, and indexes directly impacts your storage costs. <a href="https://aws.amazon.com/caching/database-caching/">Caching</a> and batching statements can improve performance. You can use <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring.OS.html">Enhanced Monoring</a> and database-specific tools like the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html#USER_LogAccess.MySQL.Generallog">MySQL slow query log</a> to track and examine outliers that take lots of resources, then optimize them.</p><h4>RDS IOPS</h4><p>Input/Output (I/O) operations are extremely important to databases. You use an input operation to write to the database and an output operation to read data. You can monitor read and write I/O per second (IOPS) from the RDS Console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2hEcaiMMgmX0DNUY.png" /></figure><p><a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html#Concepts.Storage.GeneralSSD">General Purpose SSD</a> instances start fully stocked with 5.4 million IOPS credits, enough to perform 3,000 operations a second for 30 minutes. They also generate I/O credits at a rate of <em>3 IOPS per GiB of storage</em>.</p><p>In the first months of an RDS transition, keep <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MonitoringOverview.html#USER_Monitoring">an eye</a> on your IOPS credit balance (also reported in the RDS console) so you don’t run out.</p><p>The options for directly increasing I/O are purchasing more storage or switching to the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html#USER_PIOPS">Provisioned IOPS storage type</a>, where you pay a fixed price per thousand IOPS. Indexing and increasing the instance size can also increase I/O speed as each item in the queue is handled more efficiently.</p><h4>RDS CloudWatch Metrics</h4><p>CloudWatch is an excellent monitoring tool, but it incurs its own <a href="https://aws.amazon.com/cloudwatch/pricing/">costs</a>. It’s important to consider your monitoring frequency. Going from 5-minute monitoring to 1-minute monitoring will dramatically increase the size of your logs. If your CloudWatch budget is slowly expanding, consider modifying the <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html#SettingLogRetention">log data retention period</a> and auditing your <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html">alarms</a> to be sure they’re relevant.</p><h4>RDS Data Transfer Cost</h4><p>Importing data to RDS from the internet <a href="https://aws.amazon.com/premiumsupport/knowledge-center/rds-import-data/">is free</a>, but you often want to use your data elsewhere. Data transfer out to the internet costs between .09 and .13 cents per GiB. Data transfer prices between zones are very dependent on the chosen zone, ranging in price from .02 to .13 cents per GiB.</p><p>Be aware that you’re actually charged twice: once when the data leaves a zone and again when it enters a target zone. To reduce these costs, minimize the amount of data you’re sending. Limit queries, and don’t re-run reports. Transferring data between Amazon RDS and EC2 Instances in the same Availability Zone is free, so you can sidestep some data fees by consolidating services into a single zone.</p><h4>RDS Snapshots</h4><p>Database snapshots contribute to your storage costs as well. You can reduce backup storage by reducing your backup retention period and deleting manually created snapshots, which are never automatically removed. If you need to store snapshots, move them somewhere cheaper <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.html">like an S3 instance</a>. For MySQL, storing snapshots in S3 is about 1/5 as expensive as keeping them in RDS.</p><h4>Conclusion</h4><p>AWS RDS is extremely powerful and deeply customizable, but the complex pricing model means that your monthly spend can grow quickly and unexpectedly.</p><p>RDS optimization starts with understanding your current requirements using monitoring tools like CostExplorer and CloudWatch. Once you know the resources you need, you can select the best region, instance engine, and size. Then, keep an eye on your IOPS credits, data transfers between zones and out to the internet, and backup requirements. Database hygiene matters even more when you’re paying per GiB, so optimize queries and use efficient indexes.</p><p>If you’re struggling to understand your AWS spend, <a href="https://www.cloudforecast.io/">CloudForecast</a> can help. Reach out to our CTO, <a href="/service/mailto:francois@cloudforecast.io">francois@cloudforecast.io</a>, if you’d like help tagging your resources or implementing a long-term cost-reduction strategy.</p><p><em>This was originally posted on the CloudForecast blog on 11/17/2020: </em><a href="https://www.cloudforecast.io/blog/aws-rds-pricing-and-optimization/"><em>https://www.cloudforecast.io/blog/aws-rds-pricing-and-optimization/</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b9505c6320c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/aws-rds-pricing-and-optimization-guide-6b9505c6320c">AWS RDS Pricing and Optimization Guide</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Parquet on Athena to Save Money on AWS]]></title>
            <link>https://medium.com/cloudforecast/using-parquet-on-athena-to-save-money-on-aws-c02fed1ae5d5?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/c02fed1ae5d5</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[devops-practice]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devtools]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Fri, 26 Apr 2019 15:04:30 GMT</pubDate>
            <atom:updated>2019-04-26T15:04:02.809Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8IUFtL2MH1JnvP4WRtnruA.png" /></figure><p>To show you how you can optimize your Athena query and save money, we will use the ‘2018 Flight On-Time Performance’ dataset from the Bureau of Transportation Statistics ( <a href="https://www.cloudforecast.io/blog/using-parquet-on-athena-to-save-money-on-aws/bts.gov">bts.gov</a>). We will also drop a few interesting facts about US Airports ✈️queried from the dataset while using Athena.</p><p>All datasets and queries used in this post can be found in our <a href="https://github.com/CloudForecast/blog-posts">Github repo</a>.</p><p><strong>Let’s start with some terminology</strong></p><blockquote><strong><em>Amazon Athena</em></strong><em> is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Pricing is based on the amount of data scanned by each query.</em></blockquote><blockquote><strong><em>Apache Parquet</em></strong><em> is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.</em></blockquote><p>If pricing is based on the amount of data scanned, you should always optimize your dataset to process the least amount of data using one of the following techniques: compressing, partitioning and using a columnar file format. We will demonstrate the benefits of compression and using a columnar format.</p><p>Using compressions will reduce the amount of data scanned by Athena, and also reduce your S3 storage. It’s a Win-Win for your AWS bill. Supported formats: <em>GZIP, LZO, SNAPPY (Parquet) and ZLIB.</em></p><p>Instead of using a row-level approach, columnar format is storing data by columns. This allows Athena to only query and process the required columns and ignore the rest. If you want to learn more about columnar, check out <a href="https://www.cloudforecast.io/blog/using-parquet-on-athena-to-save-money-on-aws/%5Bhttps://en.wikipedia.org/wiki/RCFile%5D(https://en.wikipedia.org/wiki/RCFile)">Wikipedia</a> and/or the “ <a href="https://towardsdatascience.com/the-beauty-of-column-oriented-data-2945c0c9f560">The beauty of column-oriented data</a> “ article by Maxim Zaks</p><h4>Getting familiar with the data</h4><p>While using the ‘2018 Flight On-Time Performance’ dataset, we will use three different types of files to compare performance with the data processed: CSV, GZip, and Parquet files.</p><p>The files downloaded from the Bureau of Transportation Statistics are simple CSV files with 23 columns (such as FlightDate, Airline, Flight #, Origin, Destination, Delay, Cancelled, …)</p><p>Same data, 3 versions:</p><p><strong>Raw CSV:</strong></p><ul><li><em>Description</em>: No compression, just a plain set of CSV files</li><li>Files: 12 ~55MB files (one for each month)</li><li>Total dataset size: ~666MBs</li></ul><p><strong>GZip CSV</strong>:</p><ul><li><em>Description</em>: Simple CSV files compressed using GZip to compress them.</li><li><em>Files</em>: 12 ~10MB Gzipped CSV files (one for each month).</li><li><em>Total dataset size</em>: ~126MBs</li></ul><p><strong>Compressed Parquet</strong>:</p><ul><li><em>Description</em>: We converted to the CSV file to parquet using Spark. The same process could also be done with (<a href="https://aws.amazon.com/glue/">AWS Glue</a>)</li><li><em>Files:</em> 12 ~8MB Parquet file using the default compression (<a href="https://github.com/google/snappy">Snappy</a>).</li><li><em>Total dataset size:</em> ~84MBs</li></ul><p>Find the three dataset versions on our <a href="https://github.com/CloudForecast/blog-posts/tree/master/athena-parquet">Github repo</a>.</p><h4>Creating the various tables</h4><p>Since the various formats and/or compressions are different, each CREATE statement needs to indicate to Athena which format/compression it should use.</p><p><strong>Raw CSVs Gzipped CSVs</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/63f78e973977dc1c3e7e97d3d91f0da1/href">https://medium.com/media/63f78e973977dc1c3e7e97d3d91f0da1/href</a></iframe><p>Athena automatically detects the gzip format (based on the “.gz” suffix) so we can re-use the query from above. See the full query <a href="https://github.com/CloudForecast/blog-posts">here</a>.</p><p><strong>Parquet Files using Snappy</strong></p><p>Using both STORED AS PARQUET and &quot;parquet.compress&quot;=&quot;SNAPPY&quot;, Athena will be able to process our data flawlessly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83e2565071df5d78d5d521b6e7830f79/href">https://medium.com/media/83e2565071df5d78d5d521b6e7830f79/href</a></iframe><h4>Picking a couple of queries to play with</h4><p><strong>Query #1: Top route in the U.S. in 2018 alongside the number of airlines providing flights for it.</strong></p><p><strong><em>Spoiler Alert</em></strong><em>: It’s Chicago O’Hare International to New York LaGuardia with 6 unique airlines offering the route</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6371e50bcb71688c99249975b20f37f1/href">https://medium.com/media/6371e50bcb71688c99249975b20f37f1/href</a></iframe><p><strong>Query #2: Best airport to fly out from to avoid any major delay (&gt; 30 minutes)</strong></p><p><strong><em>Spoiler Alert</em></strong><em>: Honolulu International Airport has less than 4% flight delayed, giving you one more reason to visit Hawaii. Newark Airport had an average of 16.2% flights delayed (&gt;30 minutes).</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38cd1e538c64258ec2cc2a9e4d7c7f79/href">https://medium.com/media/38cd1e538c64258ec2cc2a9e4d7c7f79/href</a></iframe><h4>Comparing performances</h4><p>For this category, I ran each query 10 times and averaged the numbers. We will use the <em>Raw</em> dataset as our baseline.</p><p><strong>Query #1</strong></p><ul><li><em>Raw</em>: 665.71 MB scanned in 2.60 seconds. $0.00333 per query</li><li><em>GZip</em>: 125.54 MB scanned in 2.08 seconds. $0.00063 per query <strong>(-81% savings)</strong>.</li><li><em>Parquet</em>: 8.29 MB scanned in 1.26 seconds. $0.000050 per query (<strong>-98% savings</strong>).</li></ul><p><strong>Query #2</strong></p><ul><li><em>Raw</em>: 665.71 MB scanned in 2.60 seconds. $0.00333 per query.</li><li><em>GZip</em>: 125.54 MB scanned in 2.08 seconds. $0.00063 per query <strong>(-81% savings)</strong>.</li><li><em>Parquet</em>: 8.29 MB scanned in 0.81 seconds. $0.000062 per query (<strong>-98% savings</strong>).</li></ul><h4>Conclusions</h4><p>Thanks to Parquet’s columnar format, Athena is only reading the columns that are needed from the query. This reduces the query time by more than <strong>50+%</strong> and reduces the query price by <strong>98%</strong>.</p><p><em>If you have any questions about Athena, Parquet or CloudForecast, feel free to ping me via email </em><a href="/service/mailto:francois@cloudforecast.io"><em>francois@cloudforecast.io</em></a><em> or by Twitter: </em><a href="https://twitter.com/francoislagier"><em>@francoislagier</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.cloudforecast.io/blog/using-parquet-on-athena-to-save-money-on-aws/"><em>https://www.cloudforecast.io</em></a><em> on April 14, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c02fed1ae5d5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/using-parquet-on-athena-to-save-money-on-aws-c02fed1ae5d5">Using Parquet on Athena to Save Money on AWS</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Serverless Reapers to Lower Your AWS Bill]]></title>
            <link>https://medium.com/cloudforecast/using-serverless-reapers-to-lower-your-aws-bill-c7b5634c1fab?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/c7b5634c1fab</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Tue, 02 Apr 2019 19:46:06 GMT</pubDate>
            <atom:updated>2019-04-02T19:45:44.538Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bgm2onLqcRf5uaJqVgKTqA.png" /></figure><p>There are a lot of factors to consider when thinking about building your applications and services inside of Amazon Web Services. Often times there is a learning curve that individuals have to climb to get a full understanding. This learning curve is a large reason why I created my <a href="https://www.kylegalbraith.com/learn-aws/">Learn AWS By Using It course</a>. It’s a course that helps you learn the basics of AWS by actually diving in and using it to build a solution to a common problem.</p><p>But, there are things even the most seasoned of AWS gurus could stand to understand better. The primary one, the AWS bill. What is costing you money on your bill and what is not. Many folks monitor their bill and if they’re smart they use a service like <a href="https://www.cloudforecast.io/">CloudForecast</a> to do so.</p><p>But monitoring is only one-third of the equation.</p><p>Understanding your AWS bill and establishing a baseline for what you expect to spend month over month is another piece to the puzzle. Once you have a baseline it’s time to start taking action when things breach your baseline.</p><p>In this post, we are going to focus on how you can take proactive actions to keep your AWS bill low. We are going to automatically destroy unused resources that are left lying around. We are going to focus on a common resource I have seen left around in AWS accounts, Elastic block store (EBS) volumes.</p><p>If you recall EBS volumes are block storage volumes that we can attach and detach from EC2 instances. They allow us to write to storage on one instance, detach the volume, attach it to a different instance and read the same data.</p><p>On the surface, these aren’t all that expensive. If we focus on the SSD gp2 volumes the pricing is $0.10 per GB-month of provisioned storage. OK, so if I use 8 GB of EBS storage for a total of 6 hours in a month my cost would be:</p><p>$0.10 * 8 * 21600 seconds / (86400 seconds / 30 days) = $0.006 per month</p><p>But, what if that 8 GB volume ran 24 hours a day for the entire month. Now things get a bit more expensive:</p><p>$0.10 * 8 = $0.80 per month</p><p>A difference of a few cents isn’t going to make or break you, but imagine if that 8 GB was 800 GB. For 6 hours out of the month, we are looking at $0.66 cents a month. But, if that EBS volume was running around the clock it would be $80 dollars a month. Quite the difference right?</p><p>There are all kinds of developer errors, bugs in the code, and brain farts that can leave an EBS volume running even though it’s not used. In this context, we define used as attached to an active production instance. You are still charged for the GB’s of provisioned EBS volumes whether they are attached to an instance. These can add up if you leave lots of them lying around.</p><p>The simplest solution is…well to not let that happen. Monitor your infrastructure and have a firm understanding of what contributes to your monthly spend in AWS. This should help you detect these types of issues sooner.</p><p>We can also put some automated processes in place to prevent these types of surprises.</p><h3>Serverless Reapers</h3><p>Turning mundane manual things into well oiled automated machines is a prime use case for serverless and AWS Lambda. So we are going to build out a quick Lambda function that finds available EBS volumes and destroys them.</p><p>At a high-level, our reaper is going to follow this idea:</p><ul><li>Launch off some CloudWatch rule that is running off a schedule</li><li>List all available EBS volumes in a given region</li><li>Destroy the volumes that are in an available state</li></ul><p>This reaper is going to provide us a huge benefit, automation. We write the code once, set up the schedule trigger, and then volumes left lying around in an available state are automatically destroyed. We never have to think about this problem again.</p><p>Let’s dive into some code.</p><h3>Implementing a reaper in AWS Lambda</h3><p>For this blog post we are going to make use of the Serverless Framework for provisioning our Lambda function. I enjoy the interface the framework provides and it makes representing your infrastructure as code a breeze. That said, it is optional but you will need to deploy your function by some other means if you choose not to use it.</p><p>If you want to see the code, I have created a GitHub repo that has this code and steps to deploy this into your own AWS account. <a href="https://github.com/kylegalbraith/aws-reaper-lambda">https://github.com/kylegalbraith/aws-reaper-lambda</a></p><p>The first step is getting our serverless template the way we want. Let’s start by creating an initial project from the command line.</p><p>$ serverless create --template aws-nodejs --path aws-reaper-lambda</p><p>Cool, we now have our initial code configuration in handler.js and an initial serverless.yml file. Remember, we want our function to be triggered off of a CloudWatch rule that is running on some kind of cadence. We can add that into our serverless template in the events section.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2f6aa469b17a39d622bc84d1d76206d/href">https://medium.com/media/f2f6aa469b17a39d622bc84d1d76206d/href</a></iframe><p>For this reaper function, we also need to grant it access to list and destroy EBS volumes. We can add that to our serverless template as well in the role statements section.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e148c571f6108c6ab69da7b71b2da1cc/href">https://medium.com/media/e148c571f6108c6ab69da7b71b2da1cc/href</a></iframe><p>Now that we have our serverless template file all ready to go, let’s write the actual code in our function handler.</p><p>Our reaper function is going to launch once per day. When it launches it is going to issue a describe volumes call to the EC2 service for the volumes that are in an available state. We can then loop over that list of volumes and delete any that are in an available state.</p><p>Let’s take a look at what that code looks like in our handler.js file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f530dd1a6ab68fcd657dc4d930e971a/href">https://medium.com/media/6f530dd1a6ab68fcd657dc4d930e971a/href</a></iframe><p>We start off by getting a new ec2Client from the aws-sdk. Next, we declare the parameters object that we are doing to pass to the describeVolumes API. Notice here that we are using the Filters syntax to only grab the EBS volumes that have a status of available.</p><p>Now inside of our try block we can call describeVolumes and ask the SDK for a promise back. Once the promise returns we will have our response from the API call and we can loop over the Volumes property.</p><p>For each volume that was returned, we are going to issue a deleteVolume API call for that VolumeId and push the resulting promise onto an array of promises we are watching.</p><p>Once we are done looping through the volumes we can wait for all promises in our array to finish resolving by using Promise.all(). When that returns we know that we have deleted the volumes that were in an available state and we exit the function with a successful callback,callback(null, Successfully removed ${availableVolumes.Volumes.length} EBS volumes.);.</p><p>With that one function, we now have an automated process that runs every day and cleans up any EBS volumes that are lying around in an available state. This could save us a bit of money on our AWS storage costs if we are making heavy use of EBS volumes and they are being left around.</p><h3>Where could this go from here</h3><p>This is a rather simple example of a serverless reaper, but this one function could save you significant amounts of money on your AWS bill. I had a client that had over 150 EBS volumes lying around in one region at any given point. Deploying this function allowed them to save hundreds of dollars on their monthly bill.</p><p>Of course, the longer term solution is to figure out why those volumes are being left lying around. But sometimes it’s better to save a bit of money and then start digging to the bottom of the root cause.</p><p>This is only one example of a serverless reaper.</p><p>This same idea can be applied to other areas of your AWS account. Perhaps you don’t want to just remove any EBS volume that is an available state but only those that are missing a certain tag. You can do that by changing your describeVolumes filter.</p><p>Or maybe you don’t want to apply this idea to EBS volumes but rather expensive EC2 instances, you can do that too. You need to describe the instances in your region, filter for the ones you want to terminate, and then terminate them.</p><p>This idea can help you with any use case where you want to automatically destroy resources.</p><h3>Conclusion</h3><p>Leaving EBS volumes lying around likely isn’t something that is going to break your bank account. But as organizations scale and more developers begin provisioning resources this type of problem can become exponentially worse.</p><p>Leaving instances or RDS databases lying around that aren’t used can be 10–100x more expensive than EBS volumes depending on the instance sizes.</p><p>What we have laid out in this blog post is one approach to avoiding those surprise AWS bills. Using reaper serverless functions we can detect when resources like EBS volumes are left lying around and delete them so we are no longer charged for them. This idea could be expanded and extended to include things like EC2 instances or RDS databases. With that approach, I have seen clients save 25–50% on their AWS bill just by destroying resources that were no longer being used.</p><p>But, this isn’t the only process you should think about implementing relating to this problem. Resources left lying around unused is usually a sign that you aren’t using Infrastructure-as-Code. Surprised by your AWS bill at the end of the month? You should consider monitoring your costs more closely using a service like CloudForecast.</p><p>As is the case with most things in AWS software development, there are many approaches, but you have to decide on the ones that work for you.</p><p><em>If you have any questions about serverless reaper, feel free to ping me via twitter: </em><a href="https://twitter.com/kylegalbraith"><em>@kylegalbraith</em></a><em>. If you have questions about CloudForecast, feel free to ping Tony: </em><a href="/service/mailto:tony@cloudforecast.io"><em>tony@cloudforecast.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7b5634c1fab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/using-serverless-reapers-to-lower-your-aws-bill-c7b5634c1fab">Using Serverless Reapers to Lower Your AWS Bill</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interview with GitLab CEO, Sid Sijbrandij: Keys to Developer Marketing]]></title>
            <link>https://medium.com/cloudforecast/interview-with-gitlab-ceo-sid-sijbrandij-keys-to-developer-marketing-39877fe9f82?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/39877fe9f82</guid>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[gitlab]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Tue, 12 Mar 2019 15:20:15 GMT</pubDate>
            <atom:updated>2019-03-12T14:40:35.895Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vNHKJnEg6gu9ajO-Ubw2xw.png" /></figure><p>The CloudForecast team recently had a chance to do a “Pick Your Brain” session with the CEO of GitLab, Sid Sijbrandij. The range of topics included, keys to marketing to developers and growth strategies.</p><p>The full interview can be found below this post. Here are a few highlights from the conversations:</p><p><strong>What are some common early mistakes that you see software service companies make with marketing. Which one did you personally fall for? Are there any specific things you can think of with developer marketing that you see that works really well?</strong></p><p>You have to trust yourself that you understand your audience. For example, I post a lot on Hacker News and I saw that as a hobby of mine. I’m seeing the huge effect it has on people being fans of GitLab through my posting on HackerNews.</p><p>Not measuring is a pitfall. If you do any marketing actions you want to make sure you measure what the effect is. It’s important to figure out ways to measure those marketing actions effects.</p><p><em>Examples: What is your qualification criteria? What is a marketing qualified lead? What is the accepted sales accepted opportunity?</em></p><p>Another pitfall is only focusing on the number of leads. If you are only focused on the count, you’ll get a lot of small opportunities. It’s important to mix in the total size of pipeline opportunity.</p><p><strong>Are there certain themes that you see that resonates with developers with content marketing?</strong></p><p>I’m a big fan of <a href="https://news.ycombinator.com/">HackerNews</a>, so I am biased towards that. If you read something on HackerNews, it’s because you want to learn something. There will be a lot of lessons and stories to tell as you build out your product. These lessons and stories are things you tell your co-founders, <em>“Hey! This is interesting and this was the effect.”.</em> Writing and telling those stories in a public setting is an underserved market. I’ve found that they are mostly done by hobbyists. I find that too many companies want to present company pitch in their post.</p><p>Telling about a hard technical problem that you solved, or something that totally didn’t work as you intended tends to be more interesting for developers. It’s very relatable and you feel like you’ve benefited from someone else making a mistake instead of you making the mistake yourself. That’s a gift that just keeps on giving. Try to keep it to topics that you discuss with developer friends when you grab a beer together.</p><p><strong>How did you measure what was successful in your early days at GitLab with acquiring customers?</strong></p><p>We followed the revenue and tried different things really quickly.</p><p>We tried many things at GitLab in the early days. We tried to sell support. Nobody wanted that. We tried donations. We got it to grow from seven dollars to less than a thousand dollar a month. We tried paid development but it was super hard to coordinate. Then we tried licensed software and that went bananas compared to the rest. We also tried different things with our price with testing a higher tier and then people bought that. Then we tested another tier that is even higher and that worked well too.</p><p><strong>You all really tried a lot of things. How quickly did you all go from one idea to another?</strong></p><p>We hopped from one idea to another pretty fast in the early days. For example,<strong> </strong>we tried donations for two months before moving on.<strong> </strong>What they advised us at YCombinator, an incubator program we went to, is to set a metric to destroy and try to prove it. Our metric at GitLab was to increase revenue by 20% every two weeks.</p><p>That really keeps you honest. If you can’t do that, then you have to make a dramatic change with your product. It’s also great because you might think that any change should take at least three months to implement. If you are forced to execute a change in one week and need to have the revenue to show for it after the two weeks, you start taking all these shortcuts to try to make it happen. These are not bad shortcuts, but good shortcuts since you’re trying out things very fast and learning.</p><p><strong>Are you paying the price for all the shortcuts? </strong><em>Example: Needing to hire more engineers or putting more engineer resources to fix tech debt , or working with clients to fix things due to the shortcuts that were made.</em></p><p>Not really! You’d think that. We don’t take shortcuts with things like code quality. We do take shortcuts with the scope of a feature. It’s sometimes embarrassing with how much we cut scope. For example, we now have a Jaeger integration, which is a new cool tracing methodology. The integration is you can put in the URL and it will link to to Jaeger. It’s a trivial feature but it was a bigger thing. We only had time to ship a basic version of it.</p><p>Another good example is introducing a new pricing plan. You can make changes to your pricing page to see how it converts. You can also take out some ads and see what click thru rate is with different pages or do interviews with relevant people and ask them what it is worth to them. We’ve always had these huge two months plan but we cut them down in scope. This still gave us feedback but in a lot quicker way.</p><p>We define scope based on how much time we have and not the dream vision.</p><p><strong>Did you feel like your growth strategy kind of evolved a bit to stay in line with your product and vision? Like how did that change a little bit from your early days to what it is today, over the last eight years?</strong></p><p>Again in the beginning we always said, “Hey the majority of the world isn’t using GitLab yet. So, we’ll optimize for everyone not using GitLab.” That meant that we had to greatly expand the scope of our product in the beginning which is pretty controversial. That has worked out really well for us because now that the scope is so big, people now see the advantage of a single application for the whole devops lifecycle.</p><p>Prioritization is getting harder now that we are bigger. It was whatever features we thought were needed in the early days. Then you get your first customers and you focus building features for them. We had one customer where we built everything they asked for. It turned out to be a really good thing because everyone else was asking for the same features later. Now we have this mixed bag of features that needs to be refactored into new features or new features that need to be extended. An extension of scope.</p><p>The most useful thing I can tell you is to have a single person that determines prioritization of features. At GitLab, product manager’s makes all final decisions on features. If you don’t agree, you can always talk to the product manager. We have certain guidelines with how they should make decisions. You don’t want the decision be between 10 different people.</p><p><strong>Any blogs or writers that you recommend that you personally enjoy reading and enjoy kind of looking over whenever you get a free chance?</strong></p><p>The blog of <a href="https://stratechery.com/">Ben Thompson</a> is great. If you’re starting a SaaS business, I would read <a href="https://www.amazon.com/Predictable-Revenue-Business-Practices-Salesforce-com/dp/0984380213">Predictable Revenue</a> by Aaron Ross.</p><p><strong>Any final advice to SaaS companies in the dev tool space?</strong></p><p>I think VC’s don’t like to invest in dev tools because it’s notoriously hard to make money. If you make a dev tool, make sure it’s a broad product. If it only has a single feature then you’re just selling a C.I. and it will be very hard to make money.</p><p><em>Thank you to Sid and the GitLab team for setting this up. GitLab is a single application for the entire software development lifecycle. From project planning and source code management to CI/CD, monitoring, and security. For more information on GitLab, please visit their website: </em><a href="http://www.gitlab.com/"><em>www.gitlab.com</em></a><em>.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FqVVgKT1quMU&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DqVVgKT1quMU&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FqVVgKT1quMU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d74b6b7a86ac78cd99ef58607f9b46db/href">https://medium.com/media/d74b6b7a86ac78cd99ef58607f9b46db/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39877fe9f82" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/interview-with-gitlab-ceo-sid-sijbrandij-keys-to-developer-marketing-39877fe9f82">Interview with GitLab CEO, Sid Sijbrandij: Keys to Developer Marketing</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Features: Send AWS Cost Reports via Slack and Other Shiny New Features.]]></title>
            <link>https://medium.com/cloudforecast/new-features-send-aws-cost-reports-via-slack-and-other-shiny-new-features-fab94da205c?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/fab94da205c</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Thu, 14 Feb 2019 06:17:21 GMT</pubDate>
            <atom:updated>2019-02-14T06:17:21.159Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OGOAh8VBD1dSRahkoXvBhw.png" /></figure><p>We have four new features available today on CloudForecast. Here is a quick rundown of everything new and shiny:</p><p><strong>TL:DR</strong></p><ol><li><strong>Slack Integration: </strong>Deliver our cost reports to a Slack channel of your choice.</li><li><strong>PagerDuty &amp; Opsgenie Integration</strong>: Page stakeholders of any major increase of AWS cost via PagerDuty or Opsgenie</li><li><strong>Frequency: </strong>Only send cost reports if it’s cloudy or stormy.</li><li><strong>Historical Reports : </strong>View past cost reports starting from Jan 2019.</li></ol><p><strong>Slack Integration</strong></p><p>Slack is one the best collaboration and communication tools for organizations. With a friendly developer ecosystem and wide adoption amongst our customers, we knew it was a no brainer for us to build an integration to deliver our daily cost reports to a Slack channel of your choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OGOAh8VBD1dSRahkoXvBhw.png" /><figcaption>Encourage collaboration and discussion around your AWS cost.</figcaption></figure><p>We’re seeing different organizations (tech and non-tech) within companies collaborate and discuss our reports sent to their Slack channels during our beta test. It gives a way for your whole organization to have more awareness around AWS cost.</p><blockquote>“We have the Ops Team as well as the bean-counters subscribed to the Slack channel where the alerts are published”</blockquote><p>Getting our Slack integration takes less than 2 minutes to implement if you are an existing customer. Click here to learn more.</p><p><em>Example use case: Do you have a financial stakeholder that would like to see the cost reports? Create a new custom report and a new slack channel to deliver our reports directly to them. Our reports are easy enough to understand and digest for non-tech/eng teams.</em></p><p><strong>PagerDuty and Opsgenie Integration</strong></p><p>In addition to our Slack integration, we developed integrations with two most widely used incident management platforms, PagerDuty and OpsGenie. Within our integration dashboard, you’ll be able to set the frequency to only page a team member if it’s “cloudy/stormy only report” or “stormy only report”.</p><p><em>Example use case: If you have someone on your team responsible for a tag or specific sub-account in AWS and cost has increased a significant amount, you can build a report around that and page the person of the increase. The page will have details of where the increase is and the %.</em></p><p><strong>Frequency</strong></p><p>Sick of seeing sunny reports and want to only send cloudy/stormy reports? With all our old and new ways to deliver reports, email, slack, PagerDuty and Opsgenie, there is now an option called “frequency” that gives you the ability to only send the report if it’s Cloudy or Stormy!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*oy_enDmh7UlMU-b1s_OFbA.gif" /></figure><p><strong>Reports History</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yV5calg6Wg-RnmfkJVJ6Vg.png" /><figcaption>You can now go back and look at old cost reports</figcaption></figure><p>You now have a way to go back and view all your historical reports starting from Jan 2019 within the report section of the dashboard. You can sift through a list of reports by month, see quickly if it was a sunny/cloudy/stormy report, get the total spend for that day and view the full report right in your dashboard. This makes it super easy if you need to go back and analyze something on a certain day.</p><p>Phew. That was a long list!</p><p>Special thanks to all our customers who have given us feedback and suggestion. We’re already heads down for our next feature sprint to help you save even more money on AWS.</p><p>If you have any feedback or suggestion for us, feel free to reach out directly to us: <a href="/service/mailto:hello@cloudforecast.io">hello@cloudforecast.io</a></p><p>-Tony for the CloudForecast Team</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fab94da205c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/new-features-send-aws-cost-reports-via-slack-and-other-shiny-new-features-fab94da205c">New Features: Send AWS Cost Reports via Slack and Other Shiny New Features.</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2018 was Awesome. 2019 Will Be Even Better!]]></title>
            <link>https://medium.com/cloudforecast/2018-was-awesome-2019-will-be-even-better-b045edcd54d1?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/b045edcd54d1</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Tue, 08 Jan 2019 14:39:30 GMT</pubDate>
            <atom:updated>2019-01-08T14:40:53.514Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TnRr_0dM35shw9KfmLzP-w.jpeg" /></figure><p>With a new year, it’s always important to take the time to reflect but also look forward to a new year.</p><p>2018 marked the first full calendar year for CloudForecast as a company, and we couldn’t be more grateful for the experience so far. There were a lot of ups and downs for us, but we survived 2018. What drives us everyday are our customers, and making sure we are hustling for them. What started as a bootstrapped side hustle, CloudForecast has grown into a valuable tool to help teams monitor their AWS cost. We wrote the following in our January 2018 post: <a href="https://medium.com/cloudforecast/benefit-from-what-we-learned-spending-2-million-annually-on-aws-f747e6a3bbb3"><em>“Benefit from what we learned spending 2 million annually on AWS.”</em></a><em>.</em></p><blockquote>The idea of CloudForecast came from Kacy and Francois’ extensive experience managing AWS cost at scale during our time at Perfect Audience (YC S11). The question that kept coming up in their heads was, “Why isn’t there an affordable cloud cost monitoring tool for the rest of us?”. We’re confident that CloudForecast can fill this niche and be an indispensable tool for companies using AWS that need a second eye on their cost.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*2VJz6adKW6Jf8IUgS6zxUg.png" /><figcaption>Our first design attempt of the daily cost report - March 2017</figcaption></figure><p><strong>A few accomplishments in 2018:</strong></p><ul><li>Features built:<strong> </strong>Custom report tool, support for sub-accounts and tags, revamped our user interface, <a href="https://cloudforecast.io/cloudoutage.html">CloudOutage</a><em> (free real-time Amazon Web Service &amp; Google Cloud Platform service alerts in Slack)</em>, and made a fun little AWS logo quiz for Re:invent.(<a href="https://cloudforecast.io/awslogoquiz/"><em>Link</em></a>)</li><li>Launched officially on HackerNews in July 2018 (<a href="https://news.ycombinator.com/item?id=17551796"><em>Link</em></a>)</li><li>Featured on HackerNoon: (<a href="https://news.ycombinator.com/item?id=17551796"><em>Link</em></a>)<em>(Thanks </em><a href="https://hackernoon.com/@davis1"><em>Dru</em></a><em> for the write up!)</em></li><li>Hustled and built a strong customer base who has given us very solid product feedback and ways we can improve.</li><li>Completed YCombinator’s 10 week Startup School program</li><li>Applied and interviewed for YC W19. We got rejected, but we got very tangible and positive feedback — <em>We’ll write more about our experience soon!</em></li></ul><p>So with all that said, 2018 was a pretty complete year for us even though we worked on this part-time! To end our year, we sent out an end of a year survey to all our customers to gauge their satisfaction of the product. We got great product feedback on how to improve and affirmation from all our customers:</p><blockquote>Enabled both our Ops and Billing teams to be informed daily of our activities in AWS. This has created a better communication and predictability of our largest monthly cost.</blockquote><blockquote>The email makes it easy to spot anomalies, particularly in dev/test accounts where someone might spin up something like an aurora cluster and forget to turn it back off.</blockquote><blockquote>Helped us become aware and triage unnecessary expenses.</blockquote><blockquote>By helping us keep an eye on our spend in an easy method with the growth rate change over time. Makes it very clear.</blockquote><blockquote>Extremely helpful to keep tabs on our spending…I like the cloud vs sun in the email to quickly know if things are ok. The monthly estimate is great.</blockquote><blockquote>You guys are doing an awesome job, can’t wait to see what 2019 brings!</blockquote><blockquote>Great work on this. I find value here, and look forward to next year!</blockquote><p>We definitely feel the love! However, this does not mean that we are content with our progress. If anything, it has lit a fire and gave us more motivation to do even more for our customers. To help accelerate our growth and build even more value for our customers in 2019 with more features, we are diving into this full-time! This was not an easy decision for us to make. We spent the last two months planning how this might look for us personally and how it might look for the company. Ultimately, we all came to the same conclusion that the timing was right.</p><p><strong>Here are a few things you can expect from us in the coming months:</strong></p><ul><li><strong>More features! </strong>We have some great features lined up in the short term to make CloudForecast more robust and helpful. We plan on integrating with popular services such as Slack and build more features around bringing more detailed visibility to your daily email reports.</li><li><strong>Customers First:</strong> Need help with something? Have feedback? We’re even more available now to help you our customers out. We’ll aim to cut down our support response time even further.</li><li><strong>More content: </strong>More helpful content related to startup life, technical pieces, and CloudForecast updates. <em>If you are interested in partnering up with us on content, let us know!</em></li></ul><p>We’re excited about 2019 and the journey we are about to take going full-time. There are tons of unknowns with running a startup, but you can expect us to hustle, and provide even more value to all our current and prospective customers.</p><p>If you have any questions, feedback or would love to learn more, feel free to reach out to us: <a href="/service/mailto:hello@cloudforecast.io">hello@cloudforecast.io</a>. We would love to hear from you!</p><p><em>Special thanks to a few people who have helped us in 2018: </em><a href="https://bradflora.com/"><em>Brad Flora</em></a><em>, Jordan Buller, and </em><a href="https://lastweekinaws.com/"><em>Corey Quinn</em></a><em>. Thank you for believing in us. It’s only the beginning but we’re thankful for all your advice and encouragement.</em></p><p>-CloudForecast Team</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b045edcd54d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/2018-was-awesome-2019-will-be-even-better-b045edcd54d1">2018 was Awesome. 2019 Will Be Even Better!</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Founder Interviews: Francois and Kacy of CloudForecast]]></title>
            <link>https://medium.com/cloudforecast/founder-interviews-francois-and-kacy-of-cloudforecast-c26e870dce8e?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/c26e870dce8e</guid>
            <category><![CDATA[founders]]></category>
            <category><![CDATA[davis-baer]]></category>
            <category><![CDATA[founder-stories]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[hackernoon]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Wed, 05 Dec 2018 16:04:48 GMT</pubDate>
            <atom:updated>2018-12-05T15:56:35.559Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*RsFdKgbwTujSGOxeMUHWJQ.png" /></figure><p><em>This was previously featured on </em><a href="https://hackernoon.com/founder-interviews-francois-and-kacy-of-cloudforecast-8ddcebe805c5"><em>Hacker Noon</em></a><em> and written by </em><a href="https://hackernoon.com/@davis1"><em>Davis Baer</em></a><em>. Davis Baer is the Co-founder of </em><a href="https://www.oneupapp.io/"><em>oneupapp.io</em></a><em> 💻 &amp; Host of the </em><a href="https://hackernoon.com/tagged/founders"><em>Hacker Noon Founder Interview series</em></a><em> 💬. Thanks for the write up, Davis!</em></p><p><a href="https://twitter.com/daviswbaer?lang=en"><strong>Davis Baer</strong></a><strong>: What’s your background, and what are you working on?</strong></p><p><a href="https://twitter.com/CloudForecastIO?lang=en"><strong>Francois and Kacy</strong></a>: We’re CloudForecast, and we’re building simple and affordable cloud cost monitoring tools. Kacy and Francois are the co-founders, and we met 5 years ago working on a previous YC startup that got acquired. We’re offering enterprise quality cloud monitoring to be used by everyone no matter what the size of the company. We’re just short of $1k in MRR, and we’ve experienced 187% growth in converted, paying customers since July.</p><p>CloudForecast is a still a side gig for us and we have full-time jobs. Kacy currently manages a team of engineers after a decade working in Infrastructure Engineering. Francois is a Senior Software Engineer in a big tech company in SF and is currently focused on building Machine Learning models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bz49ow4UWikJEqDLdzhiFA.png" /><figcaption>The Daily Cost Report notifies you of any significant cost increases in AWS. <a href="http://bit.ly/cloudforecastemail">Click here</a> for a full screenshot of the CloudForecast “Daily Cost Report” email.</figcaption></figure><p><strong>What motivated you to get started with your company?</strong></p><p>We were end users of existing paid and free products in the market, but we realized that nothing in the market really fit the needs of small and medium businesses. Our competitors are expensive, and they are justifying the high price by offering a large suite of features that often go unused and don’t add value. They also require contracts that are designed to lock in companies. Existing open source products were clunky and not actively maintained.</p><p>At the company that we worked for, we came up with creative ways to monitor and alert on our AWS spend. We built a few small in-house services to help out with this. To validate the idea, we built an MVP in a few weekends and found out that a few people within our network were interested in paying for the service. We were pretty excited about that!</p><p><strong>What went into building the initial product?</strong></p><p>The MVP only required a few weeks worth of work. Francois focused on building the backend data processing side in Scala on AWS Lambda, and Kacy focused on building a web front-end (Ruby on Rails) that could simply onboard customers and set them up. It also helped that Francois’ wife is a UX engineer. She helped us with the design of the daily email and the front-end website. Since we were building this on AWS Lambda, running the backend is <em>really</em> cheap. Our Heroku bill for the Rails app was pretty low too. We were able to support three initial paying customers with the MVP for under $40 a month.</p><p>We really wanted to make the MVP as simple as possible, so we restricted it to be a bare bones product. The web front-end would collect the AWS credentials, a credit card, and an email address. The lambda functions would process the report on a daily basis. We met in person a few times to collaborate on the architecture and overall idea, but most of the work for the MVP was split out pretty evenly. We used that time to reach out to potential clients in our network and did more market research.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fqm7ivs-41O67bHW_7USrQ.png" /><figcaption>Full customization of the “Daily Cost Report” email. Adjust alert thresholds to notify you of any significant cost increases in AWS.</figcaption></figure><p><strong>How have you attracted users and grown your company?</strong></p><p>Marketing for developer tools is not an easy task! We hustled to get our first five trialing customers from our personal network. Getting the customers from our personal network was half the battle. We needed to make sure they felt loved, taken care of and that we listened to their feedback. That meant hustling to reiterate and improve our product as quickly as possible with every bit of feedback that came. Every improvement we did for these five customers meant that they were likely to become a paying customer.</p><p>All five eventually became paying customers, which gave us the confidence to do an official launch via a “Show Hacker News post in July 2018 (<a href="https://news.ycombinator.com/item?id=17551796">https://news.ycombinator.com/item?id=17551796</a>). The Show HN post produced 63 signups, 20 free trials and converted 6 additional paying customers which we were pretty excited about! Out of the 14 trials that did not convert, many of them were too small and just wanted to test out the product. Unfortunately, some of them didn’t see the value in our product, so we worked with them to improve it.</p><p>We ran a few ads on various platforms with a small budget ($50 each) to test the water with the goal to invest more if we found a good angle. Reddit has performed well for us. However, LinkedIn did not yield any results, so we shut it down pretty quickly. We will revisit LinkedIn in the future.</p><p>Here is our current focus for user acquisition:</p><ul><li>Content marketing by creating helpful content around AWS and cost management.</li><li>Cold outreach targeting our ideal customers, CTO/VP Eng of startups/SMBs spending up to $1M annually on AWS.</li><li>Onboarding YC Startup School companies</li><li>Building a strong referral program.</li></ul><p>Our long term focus:</p><ul><li>Build relationships with AWS influencers and various online AWS communities.</li><li>Attend and sponsor AWS community events, meetup groups, and conferences.</li><li>Build a set of free tools in the AWS space to create qualified leads.</li></ul><p>We highly recommend reading Paul Graham’s essay on “<a href="http://paulgraham.com/ds.html">Do Things that Don’t Scale</a>”. We took a lot of advice from the essay and applied it to our first set of customers. Doing things that don’t scale is still a very important part of our customer acquisition plan.</p><p><strong>What’s your business model, and how have you grown your revenue?</strong></p><p>We have two subscription tiers: $49/month and $149/month. For companies spending more than $100k/month on AWS, we created a custom plan (always &lt;1%).</p><p>With our current product offering, converting 1% of our target companies (400,000 in total) will get us to a minimum of $200k MRR.</p><p>The revenue growth will increase as we build out more features to detect waste on AWS. The total market for cloud computing was $260.2 Billion in 2017. Based on existing research, it’s estimated that 35% of cloud spend goes to waste. If we capture a slice of that by saving people money, we will penetrate a market that can save $91 Billion annually. This number will continue to grow as the market is expected to reach $411 Billion by 2020.</p><p>Our monthly operating costs are pretty much fixed. We have a fixed Heroku bill, a banking monthly fee, Intercom (customer support), and Google Suite. Our AWS costs are pretty darn low too; we don’t like wasting money. The best part about this business is that our margins increase with more revenue!</p><p><strong>What are your goals for the future?</strong></p><p>We’re planning to grow into a fully-featured cloud monitoring service. However, our immediate focus is targeting the underserved market of small businesses and startups in this space. We also plan to implement GCP and Azure cost analysis within the next year. We have a few unique products on the horizon, and we can’t wait to share with everyone.</p><p>We are also heavily focusing on our marketing strategy to reach $2,500 MRR by the end of the year.</p><p><strong>What are the biggest challenges you’ve faced and obstacles you’ve overcome? If you had to start over, what would you do differently?</strong></p><p>Setting the right expectations was definitely a struggle for us at the beginning. We wanted to move quickly, but we were time-bounded by our full-time jobs. Clear focus on priorities is key to have decent progress and visible impact on your product.</p><p>We could/should have started building a list of potential leads that we could easily convert when our MVP was ready. In other words, we started our marketing strategy a bit late.</p><p><strong>Have you found anything particularly helpful or advantageous?</strong></p><p>Our original ShowHN post on Hacker News has been instrumental to our success so far. We’ve found that comments in the post and private messages we’ve received over email have been very helpful. It’s very validating that folks are finding our product useful enough to give us candid feedback and are invested in helping us make the product better.</p><p>We recently joined the YC Startup School community. YC is providing us with great content and an experienced community. It’s great to bounce ideas, learn from others, and motivate each other. YC is generously offering the content for free, and you can find all the videos here: <a href="https://bit.ly/2p32gJJ">https://bit.ly/2p32gJJ</a></p><p>One of the books that inspired our frugalness was “The $100 Startup”. Many of the business decisions we’ve made over the last year have been inspired by this book. Our marketing strategy is following the method described in the “Traction” book. The book explains various marketing channels and a system to follow to prioritize them.</p><p><strong>What’s your advice for entrepreneurs who are just starting out?</strong></p><p>Don’t be afraid to ask for candid feedback early! If you’re developing an MVP, try to identify a few ideal customers in your network.</p><p>Although some may disagree, we believe that building beautiful products that are easy to use is just as important as building out a fancy technical implementation.</p><p>Momentum is everything! Early in your project, you will be excited to build something awesome. However, that honeymoon phase doesn’t last long but talking to clients and showing them your product will fuel your motivation.</p><p><strong>Where can we go to learn more?</strong></p><p>If you would like to learn more about us, catch us over at <a href="https://cloudforecast.io/">https://cloudforecast.io</a>.</p><p>We also have a blog that we updates about once a month at <a href="https://medium.com/cloudforecast">https://medium.com/cloudforecast</a></p><p>We’re also on Twitter at <a href="https://twitter.com/cloudforecastio">https://twitter.com/cloudforecastio</a></p><p>If you have any other questions, please reach out to us at <a href="/service/mailto:hello@cloudforecast.io">hello@cloudforecast.io</a></p><p>If you are using AWS and want to try us, just signup and reach out to us at <a href="/service/mailto:hello@cloudforecast.io">hello@cloudforecast.io</a> and we will extend your trial!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c26e870dce8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/founder-interviews-francois-and-kacy-of-cloudforecast-c26e870dce8e">Founder Interviews: Francois and Kacy of CloudForecast</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we saved over $1,000 by building CloudForecast.io with Serverless and AWS Lambda.]]></title>
            <link>https://medium.com/cloudforecast/how-we-saved-over-1-000-by-building-cloudforecast-io-with-serverless-and-aws-lambda-8c9c244882e8?source=rss----615927e38fe5---4</link>
            <guid isPermaLink="false">https://medium.com/p/8c9c244882e8</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[bootstrapping]]></category>
            <dc:creator><![CDATA[CloudForecast Team]]></dc:creator>
            <pubDate>Tue, 18 Sep 2018 20:42:59 GMT</pubDate>
            <atom:updated>2018-09-11T15:34:29.578Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ysIIgObxCCx0vZrm" /></figure><p>In a previous blog <a href="https://medium.com/cloudforecast/benefit-from-what-we-learned-spending-2-million-annually-on-aws-f747e6a3bbb3">post</a>, we described our journey to launch <a href="https://www.cloudforecast.io/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=serverless">CloudForecast </a>and some of the questions every startup face. In this post, we will answer the question “What’s the right way to build this product with our resources?”</p><p><strong>The “$100 Startup” concept</strong></p><p>I’ve been obsessed with a book called “$100 Startup” by Chris Guillebeau for over a year now. In his book, Chris advises his readers to get their product out fast while being cost-effective to limit the damage if your idea fails (<em>definitely possible, unfortunately!</em>). We try to apply this advice on every side of our business including our technical decisions.</p><p><strong>Our goals/requirements</strong></p><p>With the $100 startup concept in mind, we started by listing our requirements that were important to us while building CloudForecast:</p><ul><li><strong>Ship it out fast</strong>: We were excited to build CloudForecast to help companies save $$$ on AWS. We wanted to put it in front of our clients ASAP to keep our excitement going. Spending time debugging config, deploy process, etc. will not make the end product better. We were looking for an out of the box solution that we can grow with and allowed us to focus on our MVP the right away.</li><li><strong>Cost effective</strong>: As an early startup, we wanted to avoid fixed costs by building a system that will scale seamlessly based on our client base.</li></ul><p><strong>AWS Lambda + Serverless = Easy + Focus + Cost Effective</strong></p><p>With our next steps, we brainstormed on how can we achieve our goals and requirements. We discussed managing our own instances, using containers and half a dozen other ideas but we wanted something simple so we decided to use a Serverless solution (or FaaS). While there are few downsides (e.g cold starts, …), we believe going Serverless better suited our use case (zero administration, pay-per-execution with no idle cost and auto-scaling).</p><p>For the dev setup and deploy process, we decided to use <a href="https://serverless.com/">Serverless Framework</a> with AWS Lambdas for the following reasons:</p><ul><li>We can focus on writing our product and let Serverless manage the rest (permission, event management)</li><li>Easy to configure and easy to deploy</li><li>Serverless supports multiple platforms (GCP, AWS, … ) which could prevent headaches in the future. We picked AWS Lambdas to start with since our clients data will be stored in S3. It was a logical decision in order to reduce the reduce the network cost</li><li>Great documentation and community (Github, Gitter, Slack)</li></ul><p>Our original requirements were fairly simple: We needed 4 AWS lambda functions each on their own cron job, and each lambda would need to talk to various AWS products (RDS, DynamoDb, SQS et al.). All that with an easy way to manage multiple environments (dev vs prod) easily with an easy/effective way to manage resource permissions.</p><p>Here is how we do it:</p><ul><li>We used 4 <a href="https://serverless.com/framework/docs/providers/aws/guide/functions/">functions</a> with <a href="https://serverless.com/framework/docs/providers/aws/guide/variables#reference-variables-in-other-files">external JSON files</a> variables based on the environment:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E58m7w8y2BYeDgV9" /></figure><ul><li>Used the <a href="https://serverless.com/framework/docs/providers/aws/guide/iam/#custom-iam-roles">iamRoleStatements</a> to configure all the permissions</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x0fjcElVTbIeVz_3" /></figure><p>With these two code snippets, we were able to set up most of our architecture. 4 functions that will interact with each other through SNS. Two functions will run check on a cron schedule (via the ‘schedule.rate’ params) to check a file need to be reprocessed and trigger another function via an SNS. This configuration will be able to scale effortlessly while keeping our cost under control. We are able to fully silo-ed our environments using iamRoleStatements to configure the permissions. We used an external variable to manage environment variables from a simple YAML file.</p><p>We considered running a couple of small instances to do a job which would cost us at least $1k for the year. Meanwhile, the AWS Lambda cost is meaningless since we are only running tens of thousands of functions a day.</p><p><strong>What’s Next?</strong></p><p>Using Serverless helped us get <a href="https://www.cloudforecast.io/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=serverless">CloudForecast.io</a> off the ground quickly. We saved our first clients thousands of dollars by giving them visibility into their AWS spend.</p><p>We are in the process of integrating AWS Athena into CloudForecast for our next project. Integrating with AWS Athena will enable new features that will allow end users to dive deeper into their AWS spend. Every company has unique spending patterns, and this will allow them to fully understand where their $$$ is going. We will write more about our Athena integration soon.</p><p>If you have any questions related to this post or what we do at <a href="https://www.cloudforecast.io/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=serverless">CloudForecast.io</a>, feel free to reach out to me at <a href="/service/mailto:francois@cloudforecast.io">francois@cloudforecast.io</a>. We would love to hear from you!</p><p>-Francois &amp; the CloudForecast Team</p><p><em>Special Thanks to Kyle Galbraith for reviewing this draft. Check out his weekly Newsletter, </em><a href="https://www.kylegalbraith.com/learn-by-doing/"><em>Learn By Doing</em></a><em>. Learn by Doing is a free weekly curated cloud, blockchain, and coding newsletter delivered to your inbox every week.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c9c244882e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudforecast/how-we-saved-over-1-000-by-building-cloudforecast-io-with-serverless-and-aws-lambda-8c9c244882e8">How we saved over $1,000 by building CloudForecast.io with Serverless and AWS Lambda.</a> was originally published in <a href="https://medium.com/cloudforecast">CloudForecast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>