<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amarjit Singh on Medium]]></title>
        <description><![CDATA[Stories by Amarjit Singh on Medium]]></description>
        <link>https://medium.com/@amarjits?source=rss-71d5fd56834a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*RfxJ-wdufgNO6cNS.</url>
            <title>Stories by Amarjit Singh on Medium</title>
            <link>https://medium.com/@amarjits?source=rss-71d5fd56834a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:58:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amarjits/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[At Branch we process and store vast amounts of data; data generated from over 15 billion user…]]></title>
            <link>https://medium.com/branch-engineering/at-branch-we-process-and-store-vast-amounts-of-data-data-generated-from-over-15-billion-user-85e32b5aac31?source=rss-71d5fd56834a------2</link>
            <guid isPermaLink="false">https://medium.com/p/85e32b5aac31</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[statefulsets]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Amarjit Singh]]></dc:creator>
            <pubDate>Fri, 05 Apr 2019 22:32:39 GMT</pubDate>
            <atom:updated>2019-04-05T22:32:39.847Z</atom:updated>
            <content:encoded><![CDATA[<h3>Managing Stateful Workloads in Kubernetes</h3><p>At Branch we process and store vast amounts of data; data generated from over 15 billion user events every day. Successfully doing so — while maintaining a <a href="https://status.branch.io/">near 100% SLA</a> — requires running hundreds of stateless microservices, which we run in <a href="https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/">Kubernetes</a> at scale. On any given work day at Branch, you can find an open terminal interacting with Kubernetes on any given engineer’s laptop.</p><p>In conjunction, all of these stateless services communicate with multiple persistent, low latency, redundant and always available databases and data processing workloads such as our self-managed NoSQL Aerospike databases, Kafka, Elasticsearch etc. Find out more on how we use <a href="https://medium.com/branch-engineering/from-zero-to-40-billion-links-our-journey-migrating-to-dynamodb-4ce598926382">DynamoDB</a> and <a href="https://medium.com/branch-engineering/optimizing-kafka-hardware-selection-within-aws-amazon-web-services-69fb4b10198">Kafka</a> to efficiently support our clients at scale.</p><h3>Challenges Scaling Stateful Workloads</h3><p>Our stateful workloads — on the other hand — are orchestrated with other various tools and often require specialized and dedicated interaction to properly handle growth and emergency situations. For a company like Branch, which is growing extremely fast resulting in a tremendous growth of data, fragmentation in orchestration and configuration management starts to appear very quickly. These types of fragmentation make it extremely difficult to maintain standard operating procedures for ensuring site reliability.</p><p>If the performance and availability of such stateful workloads can be met with the orchestration and configuration management of Kubernetes, we could solve some of the above pain points. Enter the use of Kubernetes’ <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">Statefulset</a> scheduling. Let’s discuss where we started and how we plan to scale them with proper selection of underlying storage layer in AWS cloud.</p><h3>Choosing Persistent Volumes</h3><p>Persistent Volumes attach to pods orchestrated by statefulsets scheduler in Kubernetes. When a pod is restarted (because of deletion or scaling ), the Kubernetes scheduler has to assign the same volume on which it was running previously. In AWS, Kubernetes can leverage EBS or Ephemeral storage (as local volume in Kubernetes statefulsets) to provide Persistent Volumes to pods.</p><p>While pods will restart and live through an instance failure/termination when using EBS persistent volumes, they would not be able to live through an instance failure when using ephemeral/instance store as local volumes. Such failures would require manually deleting PVCs before affected statefulset pod would restart.</p><h3>Provisioning Statefulsets Based on Workload Profile</h3><p>The following decision tree provides a starting point when choosing type of volume for stateful set workloads in AWS cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gv90bXiezATUWXIo" /></figure><p>Ref:<a href="https://www.slideshare.net/AmazonWebServices/stg306deep-dive-on-amazon-ebs/14"> https://www.slideshare.net/AmazonWebServices/stg306deep-dive-on-amazon-ebs/14</a></p><h3>Average Performance Workloads</h3><p>For average performance workloads we decided to use AWS EBS gp2 volume as a statefulset persistent data store. We run the Elasticsearch and Kafka workload storing stateful data in EBS volumes. Since EBS volume is network based and can be attached to any instance (or Kubernetes node), losing a node does not result in losing data. In such an event, the volume is detached from the failed node and attached to a new node where the impacted pod gets started.</p><p>At high level, common steps to create statefulsets using EBS as persistent volumes are:-</p><ol><li>Create a Storage class specifying type of EBS volume i.e gp2, io1, sc1, st1 (<a href="https://kubernetes.io/docs/concepts/storage/storage-classes/">ref this doc for help</a>)</li><li>Create a Persistent Volume Claim which needs Persistent Volume either created manually or dynamiclally. (<a href="https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/">more information here</a>).</li><li>Consume PVC and create a statefulset workload (<a href="https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/">example and more information here</a>)</li></ol><h3>High performance Workloads</h3><p>High performance stateful set workloads require instance based storage; e.g NVMe storage on AWS i3 instances. This requires using a special type of Persistent Volumes called <a href="https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/">Local Volumes</a> in Kubernetes. In other words, local persistent volumes in Kubernetes can be seen as mapped to instance-based storage also known as ephemeral storage.</p><p>Ephemeral storage is fast, but comes with a small disadvantage; data is lost, when an instance is terminated or lost (in case of spots or scaling down activities). In such an event, statefulset pod is lost and would be restarted on a new node. This might require deleting old PVC associated with terminated instances. A new statefulset pod with fresh volume would mean that underlying application/workload would have to rebalance and recreate lost copies of data on new instance(with new volume).</p><p>Statefulsets with Local Persistent Volumes can be created with the same steps as described above, however the following steps must first be performed before following the above-mentioned steps. When creating one more important Local ephemeral storage disks, they need to be mounted before they can be used by Kubernetes nodes.</p><p>At a high level, steps to create Local Persistent Volume based statefulsets would be:-</p><ol><li>Partition NVME disks when Kubernetes node is added to cluster. (see below for details)</li><li>Format and mount partition at node level.(see below for details)</li><li>Create Storage Class specifying local persistent volumes.</li><li>Run external dynamic provisioner .Kubernetes does not support dynamic provisioning for local persistent volumes natively but can be achieved by using an external provisioner. External dynamic provisioner would look for the volumes mounted under a specific mount point, create PVs from them and make them ready to be consumed by above Storage Class. We use the one listed<a href="https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/"> here</a>. Using helm, you would generate a yaml to create ConfigMap, RBAC and DaemonSet which runs external provisioned. Refer to an example shows at the end of this article.</li><li>Create statefulset workload as needed.</li></ol><p>#1 and #2 can be automated by adding a user-data script which gets executed when a new instance is provisioned. If Kubernetes is provisioned via kops, the following yaml can be used to create an instance group which provides local instance storage based persistence volumes.</p><blockquote><em>apiVersion: kops/v1alpha2</em></blockquote><blockquote><em>kind: InstanceGroup</em></blockquote><blockquote><em>metadata:</em></blockquote><blockquote><em>labels:</em></blockquote><blockquote><em>kops.k8s.io/cluster: your_cluster_name</em></blockquote><blockquote><em>name: your_node_name</em></blockquote><blockquote><em>spec:</em></blockquote><blockquote><em>additionalUserData:</em></blockquote><blockquote><em>- name: mountvolumes.sh</em></blockquote><blockquote><em>type: text/x-shellscript</em></blockquote><blockquote><em>content: |</em></blockquote><blockquote><em>#!/bin/bash</em></blockquote><blockquote><em>cp /etc/fstab /etc/fstab.orig</em></blockquote><blockquote><em>lsblk|grep “^nv”|awk ‘{print $1}’|while read A</em></blockquote><blockquote><em>do</em></blockquote><blockquote><em>fdisk /dev/$A &lt;&lt; EOF</em></blockquote><blockquote><em>n</em></blockquote><blockquote><em>p</em></blockquote><blockquote><em>1</em></blockquote><blockquote><em>+400G</em></blockquote><blockquote><em>n</em></blockquote><blockquote><em>p</em></blockquote><blockquote><em>2</em></blockquote><blockquote><em>+400G</em></blockquote><blockquote><em>n</em></blockquote><blockquote><em>p</em></blockquote><blockquote><em>3</em></blockquote><blockquote><em>+400G</em></blockquote><blockquote><em>n</em></blockquote><blockquote><em>p</em></blockquote><blockquote><em>w</em></blockquote><blockquote><em>EOF</em></blockquote><blockquote><em>mkfs.ext4 /dev/${A}p1</em></blockquote><blockquote><em>mkfs.ext4 /dev/${A}p2</em></blockquote><blockquote><em>mkfs.ext4 /dev/${A}p3</em></blockquote><blockquote><em>mkfs.ext4 /dev/${A}p4</em></blockquote><blockquote><em>mkdir /mnt/${A}p1</em></blockquote><blockquote><em>mkdir /mnt/${A}p2</em></blockquote><blockquote><em>mkdir /mnt/${A}p3</em></blockquote><blockquote><em>mkdir /mnt/${A}p4</em></blockquote><blockquote><em>echo “/dev/${A}p1 /mnt/${A}p1 ext4 defaults 1 2” &gt;&gt;/etc/fstab</em></blockquote><blockquote><em>echo “/dev/${A}p2 /mnt/${A}p2 ext4 defaults 1 2” &gt;&gt;/etc/fstab</em></blockquote><blockquote><em>echo “/dev/${A}p3 /mnt/${A}p3 ext4 defaults 1 2” &gt;&gt;/etc/fstab</em></blockquote><blockquote><em>echo “/dev/${A}p4 /mnt/${A}p4 ext4 defaults 1 2” &gt;&gt;/etc/fstab</em></blockquote><blockquote><em>done</em></blockquote><blockquote><em>mount -a</em></blockquote><blockquote><em>cloudLabels:</em></blockquote><blockquote><em>TAG1: Value1</em></blockquote><blockquote><em>TAG2: Value2</em></blockquote><blockquote><em>image: ac_number/path/to/image</em></blockquote><blockquote><em>machineType: i3.2xlarge</em></blockquote><blockquote><em>nodeLabels:</em></blockquote><blockquote><em>app: your_node_label</em></blockquote><blockquote><em>instance_type: i3</em></blockquote><blockquote><em>kops.k8s.io/instancegroup: your_instance_group_name</em></blockquote><blockquote><em>role: Node</em></blockquote><blockquote><em>subnets:</em></blockquote><blockquote><em>- us-west-1a</em></blockquote><p><strong>Dynamic Provisioner</strong></p><blockquote>— -</blockquote><blockquote><em>#Source:provisioner/templates/provisioner.yaml</em></blockquote><blockquote><strong>apiVersion</strong>:v1</blockquote><blockquote><strong>kind</strong>:ConfigMap</blockquote><blockquote><strong>metadata</strong>:</blockquote><blockquote><strong>name</strong>:local-provisioner-config</blockquote><blockquote><strong>namespace</strong>:your_namespace</blockquote><blockquote><strong>labels</strong>:</blockquote><blockquote><strong>heritage</strong>:<strong>”Tiller”</strong></blockquote><blockquote><strong>release</strong>:<strong>”RELEASE-NAME”</strong></blockquote><blockquote><strong>chart</strong>:provisioner-2.3.0</blockquote><blockquote><strong>data</strong>:</blockquote><blockquote><strong>storageClassMap</strong>:|</blockquote><blockquote>fast-disks:</blockquote><blockquote>hostDir:/mnt</blockquote><blockquote>mountDir:/mnt</blockquote><blockquote>blockCleanerCommand:</blockquote><blockquote>-”/scripts/shred.sh”</blockquote><blockquote>-”2&quot;</blockquote><blockquote>volumeMode:Filesystem</blockquote><blockquote>fsType:ext4</blockquote><blockquote>— -</blockquote><blockquote><strong>apiVersion</strong>:apps/v1</blockquote><blockquote><strong>kind</strong>:DaemonSet</blockquote><blockquote><strong>metadata</strong>:</blockquote><blockquote><strong>name</strong>:local-volume-provisioner</blockquote><blockquote><strong>namespace</strong>:your_namespace</blockquote><blockquote><strong>labels</strong>:</blockquote><blockquote><strong>app</strong>:local-volume-provisioner</blockquote><blockquote><strong>heritage</strong>:<strong>”Tiller”</strong></blockquote><blockquote><strong>release</strong>:<strong>”RELEASE-NAME”</strong></blockquote><blockquote><strong>chart</strong>:provisioner-2.3.0</blockquote><blockquote><strong>spec</strong>:</blockquote><blockquote><strong>selector</strong>:</blockquote><blockquote><strong>matchLabels</strong>:</blockquote><blockquote><strong>app</strong>:local-volume-provisioner</blockquote><blockquote><strong>template</strong>:</blockquote><blockquote><strong>metadata</strong>:</blockquote><blockquote><strong>labels</strong>:</blockquote><blockquote><strong>app</strong>:local-volume-provisioner</blockquote><blockquote><strong>spec</strong>:</blockquote><blockquote><strong>serviceAccountName</strong>:local-storage-admin</blockquote><blockquote><strong>nodeSelector</strong>:</blockquote><blockquote><strong>instance_type</strong>:i3</blockquote><blockquote><strong>tolerations</strong>:</blockquote><blockquote><strong>-effect</strong>:NoSchedule</blockquote><blockquote><strong>key</strong>:app</blockquote><blockquote><strong>operator</strong>:Equal</blockquote><blockquote><strong>value</strong>:your_namespace</blockquote><blockquote><strong>priorityClassName</strong>:system-node-critical</blockquote><blockquote><strong>containers</strong>:</blockquote><blockquote><strong>-image</strong>:<strong>”quay.io/external_storage/local-volume-provisioner:v2.3.0&quot;</strong></blockquote><blockquote><strong>name</strong>:provisioner</blockquote><blockquote><strong>securityContext</strong>:</blockquote><blockquote><strong>privileged</strong>:true</blockquote><blockquote><strong>env</strong>:</blockquote><blockquote><strong>-name</strong>:MY_NODE_NAME</blockquote><blockquote><strong>valueFrom</strong>:</blockquote><blockquote><strong>fieldRef</strong>:</blockquote><blockquote><strong>fieldPath</strong>:spec.nodeName</blockquote><blockquote><strong>-name</strong>:MY_NAMESPACE</blockquote><blockquote><strong>valueFrom</strong>:</blockquote><blockquote><strong>fieldRef</strong>:</blockquote><blockquote><strong>fieldPath</strong>:metadata.namespace</blockquote><blockquote><strong>-name</strong>:JOB_CONTAINER_IMAGE</blockquote><blockquote><strong>value</strong>:<strong>”quay.io/external_storage/local-volume-provisioner:v2.3.0&quot;</strong></blockquote><blockquote><strong>volumeMounts</strong>:</blockquote><blockquote><strong>-mountPath</strong>:/etc/provisioner/config</blockquote><blockquote><strong>name</strong>:provisioner-config</blockquote><blockquote><strong>readOnly</strong>:true</blockquote><blockquote><strong>-mountPath</strong>:/dev</blockquote><blockquote><strong>name</strong>:provisioner-dev</blockquote><blockquote><strong>-mountPath</strong>:/mnt</blockquote><blockquote><strong>name</strong>:fast-disks</blockquote><blockquote><strong>mountPropagation</strong>:<strong>”HostToContainer”</strong></blockquote><blockquote><strong>volumes</strong>:</blockquote><blockquote><strong>-name</strong>:provisioner-config</blockquote><blockquote><strong>configMap</strong>:</blockquote><blockquote><strong>name</strong>:local-provisioner-config</blockquote><blockquote><strong>-name</strong>:provisioner-dev</blockquote><blockquote><strong>hostPath</strong>:</blockquote><blockquote><strong>path</strong>:/dev</blockquote><blockquote><strong>-name</strong>:fast-disks</blockquote><blockquote><strong>hostPath</strong>:</blockquote><blockquote><strong>path</strong>:/mnt</blockquote><blockquote>— -</blockquote><blockquote><em>#Source:provisioner/templates/provisioner-service-account.yaml</em></blockquote><blockquote><strong>apiVersion</strong>:v1</blockquote><blockquote><strong>kind</strong>:ServiceAccount</blockquote><blockquote><strong>metadata</strong>:</blockquote><blockquote><strong>name</strong>:local-storage-admin</blockquote><blockquote><strong>namespace</strong>:your_namespace</blockquote><blockquote><strong>labels</strong>:</blockquote><blockquote><strong>heritage</strong>:<strong>”Tiller”</strong></blockquote><blockquote><strong>release</strong>:<strong>”RELEASE-NAME”</strong></blockquote><blockquote><strong>chart</strong>:provisioner-2.3.0</blockquote><blockquote>— -</blockquote><blockquote><em>#Source:provisioner/templates/provisioner-cluster-role-binding.yaml</em></blockquote><blockquote><strong>apiVersion</strong>:rbac.authorization.k8s.io/v1</blockquote><blockquote><strong>kind</strong>:ClusterRoleBinding</blockquote><blockquote><strong>metadata</strong>:</blockquote><blockquote><strong>name</strong>:local-storage-provisioner-pv-binding</blockquote><blockquote><strong>labels</strong>:</blockquote><blockquote><strong>heritage</strong>:<strong>”Tiller”</strong></blockquote><blockquote><strong>release</strong>:<strong>”RELEASE-NAME”</strong></blockquote><blockquote><strong>chart</strong>:provisioner-2.3.0</blockquote><blockquote><strong>subjects</strong>:</blockquote><blockquote><strong>-kind</strong>:ServiceAccount</blockquote><blockquote><strong>name</strong>:local-storage-admin</blockquote><blockquote><strong>namespace</strong>:your_namespace</blockquote><blockquote><strong>roleRef</strong>:</blockquote><blockquote><strong>kind</strong>:ClusterRole</blockquote><blockquote><strong>name</strong>:system:persistent-volume-provisioner</blockquote><blockquote><strong>apiGroup</strong>:rbac.authorization.k8s.io</blockquote><blockquote>— -</blockquote><blockquote><strong>apiVersion</strong>:rbac.authorization.k8s.io/v1</blockquote><blockquote><strong>kind</strong>:ClusterRole</blockquote><blockquote><strong>metadata</strong>:</blockquote><blockquote><strong>name</strong>:local-storage-provisioner-node-clusterrole</blockquote><blockquote><strong>labels</strong>:</blockquote><blockquote><strong>heritage</strong>:<strong>”Tiller”</strong></blockquote><blockquote><strong>release</strong>:<strong>”RELEASE-NAME”</strong></blockquote><blockquote><strong>chart</strong>:provisioner-2.3.0</blockquote><blockquote><strong>rules</strong>:</blockquote><blockquote><strong>-apiGroups</strong>:[<strong>“”</strong>]</blockquote><blockquote><strong>resources</strong>:[<strong>“nodes”</strong>]</blockquote><blockquote><strong>verbs</strong>:[<strong>“get”</strong>]</blockquote><blockquote>— -</blockquote><blockquote><strong>apiVersion</strong>:rbac.authorization.k8s.io/v1</blockquote><blockquote><strong>kind</strong>:ClusterRoleBinding</blockquote><blockquote><strong>metadata</strong>:</blockquote><blockquote><strong>name</strong>:local-storage-provisioner-node-binding</blockquote><blockquote><strong>labels</strong>:</blockquote><blockquote><strong>heritage</strong>:<strong>”Tiller”</strong></blockquote><blockquote><strong>release</strong>:<strong>”RELEASE-NAME”</strong></blockquote><blockquote><strong>chart</strong>:provisioner-2.3.0</blockquote><blockquote><strong>subjects</strong>:</blockquote><blockquote><strong>-kind</strong>:ServiceAccount</blockquote><blockquote><strong>name</strong>:local-storage-admin</blockquote><blockquote><strong>namespace</strong>:your_namespace</blockquote><blockquote><strong>roleRef</strong>:</blockquote><blockquote><strong>kind</strong>:ClusterRole</blockquote><blockquote><strong>name</strong>:local-storage-provisioner-node-clusterrole</blockquote><blockquote><strong>apiGroup</strong>:rbac.authorization.k8s.io</blockquote><blockquote>— -</blockquote><blockquote><em>#Source:provisioner/templates/namespace.yaml</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85e32b5aac31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/branch-engineering/at-branch-we-process-and-store-vast-amounts-of-data-data-generated-from-over-15-billion-user-85e32b5aac31">At Branch we process and store vast amounts of data; data generated from over 15 billion user…</a> was originally published in <a href="https://medium.com/branch-engineering">branch-engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Optimize AWS Data Transfer Costs]]></title>
            <link>https://medium.com/branch-engineering/how-to-optimize-aws-data-transfer-costs-9d4f2c085f0?source=rss-71d5fd56834a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d4f2c085f0</guid>
            <category><![CDATA[vpc]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[network]]></category>
            <dc:creator><![CDATA[Amarjit Singh]]></dc:creator>
            <pubDate>Fri, 25 May 2018 20:38:31 GMT</pubDate>
            <atom:updated>2018-05-25T21:01:36.406Z</atom:updated>
            <content:encoded><![CDATA[<p>When system administrators were only dealing with local data centers, they didn’t have to pay attention to data transfer costs. Once the infrastructure was set up, data transfers were free, so design changes over the long-term didn’t impact network operational costs. But in the cloud environment, every data transfer is measured. When you are transferring information between virtual private cloud (VPC), availability zone (AZ) or external links, you are getting charged for every bit of data. So system administrators need to pay more attention to the factors that affect data transfer in an AWS environment.</p><p>AWS doesn’t make the data transfer charges easy to understand. As a result, for most organizations, data transfer costs on AWS are shrouded in a cloud of mystery. From the AWS Console, it’s not possible to monitor the data transfers of your instances and applications. You can use the AWS generated or user-defined <a href="https://aws.amazon.com/answers/account-management/aws-tagging-strategies/">cost allocation tags</a> to gain more insights, but it can be a time-consuming process. AWS offers various data points that can be analyzed to get better visibility into data transfer costs, however.</p><p><strong>Understanding Data Transfer Charges</strong></p><p>On AWS, data transfer cost depends on the source and on the destination. The below diagram shows the various cost points from a data flow perspective:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*lJ--aVHTTkKb2b2O." /></figure><p>As long as cost allocation tags are enabled, the <a href="https://aws.amazon.com/aws-cost-management/aws-cost-explorer/">AWS Cost Explorer</a> can show data transfer for each tagged instance. Having a granular view of the various components of your system is important for understanding your overall data transfer costs. The tool will help you visualize the information and better manage your resources. Below is a screenshot of the AWS Cost Explorer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8OEZQ-GoF48dGTIA." /></figure><p>You can use the following criteria to further refine and analyze the costs:</p><p>· Data transfer cost as an aggregate number for VPCs/AZs/Regions.</p><p>· Aggregate cost for outbound data transfer to the Internet.</p><p>· Cost of data transfer by an application or instance.</p><p>· Internal data communication cost. (Remember, any communication over public IP, even in the same AZ or VPC is charged.)</p><p><strong>Cost-Optimization Through the Right Architecture</strong></p><p>The placement of your resources will play a vital role in your data transfer costs. Here are some key factors to keep in mind during the architecture design:</p><p>· <strong>Availability:</strong> A multi-AZ architecture is highly available.</p><p>· <strong>Disaster:</strong> A multi-region architecture provides higher availability during a disaster in a single region.</p><p>· <strong>Security:</strong> Resources split across accounts or VPCs provide better security.</p><p>· <strong>Cost:</strong> Anytime resources are split across AZs, Regions or VPCs, data transfer cost would increase.</p><p>You should choose the architecture according to the needs of your application. For example, if a particular resource cannot provide high availability during an AZ or region failure, your whole application would be unavailable. In such cases, you would be paying higher costs of data transfer without the benefits of availability. Similarly, if VPC peering, routing, and VPN connectivity are not carefully designed, data transfer costs would increase without the benefits of isolation or security.</p><p>Use the following table as a guideline to decide what matters most to your architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E9zKnhSI0EdtZ3DgZpDtMA.png" /></figure><p><strong>In Conclusion</strong></p><p>In the cloud, system administrators have to rethink their approach to system architecture. Even though AWS provides a lot of new tools and resources, system administrators have to understand how their design will impact data transfer costs. If not optimized, the costs can add up and impact your bottom line.</p><p><em>Interested in the engineering challenges discussed here? Come help us out, and check out </em><a href="https://branch.io/careers/"><em>https://branch.io/careers/</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d4f2c085f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/branch-engineering/how-to-optimize-aws-data-transfer-costs-9d4f2c085f0">How to Optimize AWS Data Transfer Costs</a> was originally published in <a href="https://medium.com/branch-engineering">branch-engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>