<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fajri Abdillah on Medium]]></title>
        <description><![CDATA[Stories by Fajri Abdillah on Medium]]></description>
        <link>https://medium.com/@clasense4?source=rss-2f6060739de2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*47x2FcYgP_p5h6tq.png</url>
            <title>Stories by Fajri Abdillah on Medium</title>
            <link>https://medium.com/@clasense4?source=rss-2f6060739de2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:00:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@clasense4/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introduction to AWS VPC with Terraform]]></title>
            <link>https://medium.com/aku-dev/introduction-to-aws-vpc-with-terraform-2924c937544e?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2924c937544e</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[iaac]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[vpc]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Mon, 06 Jul 2020 11:17:55 GMT</pubDate>
            <atom:updated>2020-07-06T11:17:55.495Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NaXOhT-sbAeG-zmS.jpg" /></figure><p><a href="https://www.terraform.io/">Terraform</a> is an Infrastructure as a Code product from <a href="https://www.hashicorp.com/">Hashicorp</a>. In this article, we are going to use Terraform to create a complete VPC with Public Subnet, Internet Gateway, Route Table, and spread across all our defined Availability Zones on AWS. With Terraform, you can easily create the whole new infrastructure by only creating variables, and destroy it with just a single command. This article is using Terraform version v0.12.25. If you are only looking for the complete code, it is available <a href="https://github.com/akuio/terraform-vpc-demo">here</a>.</p><h3>Prerequisites</h3><ul><li>Terraform is installed</li><li>AWS Account is connected and having proper permission</li></ul><h3>Amazon VPC</h3><p><a href="https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html">Amazon VPC</a> is a <strong>logically isolated</strong> virtual network that is dedicated to an AWS account. This virtual network <strong>closely</strong> resembles a <strong>traditional network</strong> that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.</p><p><strong>VPC Components</strong></p><ul><li>VPC</li><li>Subnet</li><li>Routing Table</li><li>Internet Gateway</li></ul><h3>VPC Hello world</h3><p>Create main.tf.</p><pre>provider &quot;aws&quot; {<br>  region = &quot;ap-southeast-1&quot;<br>}<br>resource &quot;aws_vpc&quot; &quot;this&quot; {<br>  cidr_block           = &quot;10.0.0.0/16&quot;<br>  enable_dns_hostnames = <strong>true</strong><br>  enable_dns_support   = <strong>true</strong><br>  tags = {<br>    Name = &quot;akudev-vpc&quot;<br>  }<br>}</pre><blockquote><em>❗ </em><strong><em>What is CIDR block?</em></strong><em><br></em><a href="https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing"><em>Classless Inter-Domain Routing</em></a><em> (CIDR /ˈsaɪdər, ˈsɪ-/) is a method for allocating IP addresses and for IP routing. Go to </em><a href="http://jodies.de/ipcalc"><em>http://jodies.de/ipcalc</em></a><em> to calculate the numbers of IP available by a CIDR block.</em></blockquote><p>Let’s run our scripts by executing the commands below.</p><pre>terraform init<br>terraform plan<br>terraform apply</pre><p>After execution is done, let’s see the resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y8EGnWobFuNCbI15.png" /></figure><p>Great. Now let’s refactor our code by introducing the variables. 🚧</p><h3>Variables</h3><p>As per the official <a href="https://www.terraform.io/docs/configuration/variables.html">documentation</a>, input variables serve as <strong>parameters</strong> for a Terraform module, allowing aspects of the module to be customized <strong>without altering</strong> the module’s own source code, and allowing modules to be <strong>shared</strong> between different configurations.</p><p>Update main.tf.</p><pre>resource &quot;aws_vpc&quot; &quot;this&quot; {<br>  cidr_block           = var.cidr_block<br>  enable_dns_hostnames = <strong>true</strong><br>  enable_dns_support   = <strong>true</strong><br>  tags = {<br>    Name = &quot;${var.project}-vpc&quot;<br>  }<br>}</pre><p>Create providers.tf file.</p><pre>provider &quot;aws&quot; {<br>  region = var.region<br>}</pre><p>Create variables.tf file.</p><pre>variable &quot;project&quot; {<br>  description = &quot;Project Name&quot;<br>  type        = string<br>}<br>variable &quot;region&quot; {<br>  description = &quot;AWS Region&quot;<br>  type        = string<br>}<br>variable &quot;cidr_block&quot; {<br>  description = &quot;The CIDR block for the VPC&quot;<br>  type        = string<br>}</pre><p>Create dev.tfvars file. This file is where we define all variables value.</p><pre>project    = &quot;akudev&quot;<br>region     = &quot;ap-southeast-1&quot;<br>cidr_block = &quot;10.0.0.0/16&quot;</pre><p>The file structure should now look like below.</p><pre>.<br>├── dev.tfvars<br>├── main.tf<br>├── providers.tf<br>└── variables.tf</pre><p>Now execute our Terraform infrastructure with these commands.</p><pre>terraform plan -var-file=dev.tfvars<br>terraform apply -var-file=dev.tfvars</pre><p>No changes in our infrastructure are expected as we are only refactoring it to use variables.</p><blockquote><em>❗ One of Terraform’s powerful features is the </em><a href="https://www.terraform.io/docs/state/index.html"><em>state</em></a><em> management. Terraform stores state in a specific file that it generates automatically. Let’s inspect the </em><em>terraform.tfstate file and see what’s inside.</em></blockquote><h3>Subnets</h3><p>A subnet is a <strong>range of IP addresses</strong> in your VPC. We can launch AWS resources into a specified subnet. In this case we create the public subnet that can be accessed by the internet.</p><p>Update main.tf, add subnet resource.</p><pre>resource &quot;aws_subnet&quot; &quot;public&quot; {<br>  count = length(var.public_subnets)</pre><pre>  vpc_id                  = aws_vpc.this.id<br>  cidr_block              = element(concat(var.public_subnets, [&quot;&quot;]), count.index)<br>  availability_zone       = element(concat(var.azs, [&quot;&quot;]), count.index)<br>  map_public_ip_on_launch = <strong>true</strong><br>  tags = {<br>    Name = &quot;${var.project}-public-${count.index}&quot;<br>  }<br>}</pre><p>Update variables.tf file. Add the block below.</p><pre>variable &quot;public_subnets&quot; {<br>  description = &quot;List of public subnets for the VPC&quot;<br>  type        = list(string)<br>}<br>variable &quot;azs&quot; {<br>  description = &quot;List of availability zones names in the region&quot;<br>  type        = list(string)<br>}</pre><p>Update dev.tfvars file. Add the block below.</p><pre>public_subnets = [&quot;10.0.32.0/20&quot;, &quot;10.0.96.0/20&quot;, &quot;10.0.160.0/20&quot;]<br>azs            = [&quot;ap-southeast-1a&quot;, &quot;ap-southeast-1b&quot;, &quot;ap-southeast-1c&quot;]</pre><blockquote><em>❗ Learn more about Terraform syntax &amp; function: </em><a href="https://www.terraform.io/docs/configuration/resources.html#count-multiple-resource-instances-by-count"><em>count</em></a><em>, </em><a href="https://www.terraform.io/docs/configuration/resources.html#count-multiple-resource-instances-by-count"><em>element</em></a><em>, </em><a href="https://www.terraform.io/docs/configuration/functions/concat.html"><em>concat</em></a><em>.</em></blockquote><p>Now let’s deploy our infrastructure with these commands.</p><pre>terraform plan -var-file=dev.tfvars<br>terraform apply -var-file=dev.tfvars</pre><p>After the execution is complete, let’s inspect the resources created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ymSK62dG-jvANNSV.png" /></figure><h3>Route Table and Internet Gateway</h3><p>As per the official <a href="https://docs.aws.amazon.com/vpc/latest/userguide/how-it-works.html#what-is-route-tables">documentation</a>, a route table contains a set of <strong>rules</strong>, called <strong>routes</strong>, that are used to <strong>determine</strong> where network traffic from your subnet or gateway is <strong>directed</strong>.</p><p>An internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between your VPC and the internet.</p><p>Update main.tf. Add a new resource.</p><pre>resource &quot;aws_route_table&quot; &quot;public&quot; {<br>  vpc_id = aws_vpc.this.id</pre><pre>  tags = {<br>    Name = &quot;${var.project}-rt&quot;<br>  }<br>}<br>resource &quot;aws_route&quot; &quot;public_igw_route&quot; {<br>  route_table_id         = aws_route_table.public.id<br>  destination_cidr_block = &quot;0.0.0.0/0&quot;<br>  gateway_id             = aws_internet_gateway.this.id</pre><pre>  timeouts {<br>    create = &quot;5m&quot;<br>  }<br>}<br>resource &quot;aws_route_table_association&quot; &quot;public&quot; {<br>  count = length(var.public_subnets)</pre><pre>  subnet_id      = element(aws_subnet.public.*.id, count.index)<br>  route_table_id = aws_route_table.public.id<br>}<br>resource &quot;aws_internet_gateway&quot; &quot;this&quot; {<br>  vpc_id = aws_vpc.this.id</pre><pre>  tags = {<br>    Name = &quot;${var.project}-igw&quot;<br>  }<br>}</pre><p>Now execute our Terraform infrastructure with these commands.</p><pre>terraform plan -var-file=dev.tfvars<br>terraform apply -var-file=dev.tfvars</pre><p>After execution is done, let’s see the resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3koh3Bng2Hf8ZAYI.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VdFn_gPR2LTK-XyX.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AAUYRaMLanO_GUWg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YR5GbZivauQ3_Zg8.png" /></figure><h3>Output</h3><p><a href="https://www.terraform.io/docs/configuration/outputs.html">Output values</a> are like the return values of a Terraform module, and have several uses:</p><ul><li>A child module can use outputs to <strong>expose</strong> a subset of its resource attributes to a parent module.</li><li>A root module can use outputs to <strong>print</strong> certain values in the CLI output after running terraform apply.</li><li>When using remote state, root module outputs can be <strong>accessed</strong> by other configurations via a terraform_remote_state data source.</li></ul><p>Create outputs.tf.</p><pre>output &quot;vpc_id&quot; {<br>  description = &quot;VPC ID&quot;<br>  value       = aws_vpc.this.id<br>}</pre><pre>output &quot;public_subnet_0&quot; {<br>  description = &quot;Public Subnet 0&quot;<br>  value       = aws_subnet.public[<strong>0</strong>].id<br>}</pre><pre>output &quot;public_subnet_1&quot; {<br>  description = &quot;Public Subnet 1&quot;<br>  value       = aws_subnet.public[<strong>1</strong>].id<br>}</pre><pre>output &quot;public_subnet_2&quot; {<br>  description = &quot;Public Subnet 2&quot;<br>  value       = aws_subnet.public[<strong>2</strong>].id<br>}</pre><p>Now let’s execute the command.</p><pre>terraform apply -var-file=dev.tfvars</pre><p>And examine the output</p><pre>....<br>Apply complete! Resources: 0 added, 0 changed, 0 destroyed.</pre><pre>Outputs:</pre><pre>public_subnet_0 = subnet-070d08c9539dd0fa9<br>public_subnet_1 = subnet-0398c19c758693d39<br>public_subnet_2 = subnet-07020dead4947732f<br>vpc_id = vpc-02d66796c64e9f486</pre><p>We will use the information above to create a new EC2 instance.</p><h3>Default public security group</h3><p>As per the official <a href="https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html">documentation</a>, a security group acts as a <strong>virtual firewall</strong> for your instance to control <strong>inbound and outbound</strong> traffic. When you launch an instance in a VPC, you can assign <strong>up to five</strong> security groups to the instance. Security groups act at the <strong>instance level</strong>, not at the subnet level.</p><p>Update main.tf. Add a new resource.</p><pre>resource &quot;aws_security_group&quot; &quot;default_public&quot; {<br>  name        = &quot;${var.project}_default_public_sg&quot;<br>  description = &quot;${var.project} default public SG&quot;<br>  vpc_id      = aws_vpc.this.id</pre><pre>  ingress {<br>    from_port   = <strong>22</strong><br>    to_port     = <strong>22</strong><br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = var.vpn_cidr_block<br>    description = &quot;VPN IP&quot;<br>  }</pre><pre>  ingress {<br>    from_port   = <strong>80</strong><br>    to_port     = <strong>80</strong><br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>    description = &quot;Allow http access&quot;<br>  }</pre><pre>  egress {<br>    from_port   = <strong>0</strong><br>    to_port     = <strong>0</strong><br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }</pre><pre>  tags = {<br>    Name = &quot;${var.project}_public_sg&quot;<br>  }<br>}</pre><p>Update variables.tf file. Add the block below.</p><pre>variable &quot;vpn_cidr_block&quot; {<br>  description = &quot;CIDR block to access public instance&quot;<br>  type        = list(string)<br>}</pre><p>If you have a VPN, it is great. Just add /32 to the end of your VPN IP address. But if you don’t have one, you can grant <strong>TEMPORARY</strong> access to your public internet. Remember, <strong>TEMPORARY</strong>.</p><pre>curl ifconfig.me<br>203.81.204.171</pre><p>Update dev.tfvars file. Add the block below.</p><pre>vpn_cidr_block       = [&quot;203.81.204.171/32&quot;]</pre><p>Update outputs.tf. Add the block below.</p><pre>output &quot;public_subnet_sg&quot; {<br>  description = &quot;Public Subnet Security Group&quot;<br>  value       = aws_security_group.default_public.id<br>}</pre><p>Now re-execute our Terraform commands.</p><pre>terraform plan -var-file=dev.tfvars<br>terraform apply -var-file=dev.tfvars</pre><p>You now have the ability to create and destroy infrastructure as needed easily, thanks to the powerful IAAC capabilities provided by Terraform. To destroy your infrastructure, simply execute the command below.</p><pre>terraform destroy -var-file=dev.tfvars</pre><blockquote><em>❗ You can also add </em><em>-auto-approve when executing </em><em>terraform apply or </em><em>terraform destroy to skip the validation.</em></blockquote><p>Example output</p><pre>Apply complete! Resources: 11 added, 0 changed, 0 destroyed.</pre><pre>Outputs:</pre><pre>public_subnet_0 = subnet-070d08c9539dd0fa9<br>public_subnet_1 = subnet-0398c19c758693d39<br>public_subnet_2 = subnet-07020dead4947732f<br>public_subnet_sg = sg-0788cc2823796c618<br>vpc_id = vpc-02d66796c64e9f486</pre><h3>Test create instance</h3><p>Now we have everything in place, let’s create a single ec2 instance to test our infrastructure.</p><p>Create ec2.tf.</p><pre>resource &quot;aws_instance&quot; &quot;this&quot; {<br>  ami                         = &quot;ami-07ce5f60a39f1790e&quot;<br>  instance_type               = &quot;t2.micro&quot;<br>  key_name                    = &quot;fajri_akudev&quot;<br>  associate_public_ip_address = <strong>true</strong><br>  security_groups             = [&quot;sg-0788cc2823796c618&quot;]<br>  subnet_id                   = &quot;subnet-070d08c9539dd0fa9&quot;<br>  tags = {<br>    Name        = var.project<br>    Description = &quot;Managed by terraform&quot;<br>  }<br>}</pre><pre>output &quot;public_dns&quot; {<br>  value = aws_instance.this.public_dns<br>}</pre><blockquote><em>❗ Make sure you have the key_name in place, in this case I already had one.</em></blockquote><p>Now execute our Terraform infrastructure with these commands.</p><pre>terraform plan -var-file=dev.tfvars<br>terraform apply -var-file=dev.tfvars</pre><p>Example output</p><pre>.....<br>Apply complete! Resources: 1 added, 0 changed, 0 destroyed.</pre><pre>Outputs:</pre><pre>public_dns = ec2-18-136-209-92.ap-southeast-1.compute.amazonaws.com<br>public_subnet_0 = subnet-070d08c9539dd0fa9<br>public_subnet_1 = subnet-0398c19c758693d39<br>public_subnet_2 = subnet-07020dead4947732f<br>public_subnet_sg = sg-0788cc2823796c618<br>vpc_id = vpc-02d66796c64e9f486</pre><p>After execution is complete, let’s look at the resources created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mryff2ezNbsbZROl.png" /></figure><p>Test ssh to the instance</p><pre>mv ~/Downloads/fajri_akudev.pem ~/.ssh<br>cd ~/.ssh<br>chmod 0400 fajri_akudev.pem<br>ssh -i &quot;fajri_akudev.pem&quot; ubuntu@ec2-18-136-209-92.ap-southeast-1.compute.amazonaws.com</pre><blockquote><em>❗ Now everything has been successfully deployed, but how do we now remove the ec2 instance. A simple way to do this is to simply comment out the </em><em>ec2.tf code, and execute </em><em>terraform apply command again. The ec2 should be destroyed.</em></blockquote><h3>Add environment and workspace</h3><p>Now we want to create a new environment for our infrastructure, but we have not defined anything on env or environment, except for a dev.tfvars file. We will also need to look into managing our state in terraform.tfstate file, otherwise, there would be a conflict.</p><p>Update main.tf. Add -${var.env} suffix to our resources.</p><pre>resource &quot;aws_vpc&quot; &quot;this&quot; {<br>  cidr_block           = var.cidr_block<br>  enable_dns_hostnames = <strong>true</strong><br>  enable_dns_support   = <strong>true</strong><br>  tags = {<br>    Name = &quot;${var.project}-vpc-${var.env}&quot;<br>  }<br>}</pre><pre>resource &quot;aws_subnet&quot; &quot;public&quot; {<br>  count = length(var.public_subnets)</pre><pre>  vpc_id                  = aws_vpc.this.id<br>  cidr_block              = element(concat(var.public_subnets, [&quot;&quot;]), count.index)<br>  availability_zone       = element(concat(var.azs, [&quot;&quot;]), count.index)<br>  map_public_ip_on_launch = <strong>true</strong><br>  tags = {<br>    Name = &quot;${var.project}-public-${count.index}-${var.env}&quot;<br>  }<br>}</pre><pre>resource &quot;aws_route_table&quot; &quot;public&quot; {<br>  vpc_id = aws_vpc.this.id</pre><pre>  tags = {<br>    Name = &quot;${var.project}-rt-${var.env}&quot;<br>  }<br>}<br>resource &quot;aws_route&quot; &quot;public_igw_route&quot; {<br>  route_table_id         = aws_route_table.public.id<br>  destination_cidr_block = &quot;0.0.0.0/0&quot;<br>  gateway_id             = aws_internet_gateway.this.id</pre><pre>  timeouts {<br>    create = &quot;5m&quot;<br>  }<br>}<br>resource &quot;aws_route_table_association&quot; &quot;public&quot; {<br>  count = length(var.public_subnets)</pre><pre>  subnet_id      = element(aws_subnet.public.*.id, count.index)<br>  route_table_id = aws_route_table.public.id<br>}<br>resource &quot;aws_internet_gateway&quot; &quot;this&quot; {<br>  vpc_id = aws_vpc.this.id</pre><pre>  tags = {<br>    Name = &quot;${var.project}-igw-${var.env}&quot;<br>  }<br>}</pre><pre>resource &quot;aws_security_group&quot; &quot;default_public&quot; {<br>  name        = &quot;${var.project}_default_public_sg-${var.env}&quot;<br>  description = &quot;${var.project} default public SG&quot;<br>  vpc_id      = aws_vpc.this.id</pre><pre>  ingress {<br>    from_port   = <strong>22</strong><br>    to_port     = <strong>22</strong><br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = var.vpn_cidr_block<br>    description = &quot;VPN IP&quot;<br>  }</pre><pre>  ingress {<br>    from_port   = <strong>80</strong><br>    to_port     = <strong>80</strong><br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>    description = &quot;Allow http access&quot;<br>  }</pre><pre>  egress {<br>    from_port   = <strong>0</strong><br>    to_port     = <strong>0</strong><br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }</pre><pre>  tags = {<br>    Name = &quot;${var.project}_public_sg-${var.env}&quot;<br>  }<br>}</pre><p>Update variables.tf. Add the block below.</p><pre>variable &quot;env&quot; {<br>  description = &quot;Environment Name&quot;<br>  type        = string<br>}</pre><p>Update dev.tfvars. Add the block below.</p><pre>env = &quot;dev&quot;</pre><p>As before, let’s execute these commands once more and validate the output.</p><pre>terraform plan -var-file=dev.tfvars<br>terraform apply -var-file=dev.tfvars</pre><p>Following output should be expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nCbc30aYudGeTzlC.png" /></figure><p>Now all infrastructure provisioned by our script carries a suffix, but we need a way to manage the state. We will introduce the use of <a href="https://www.terraform.io/docs/state/workspaces.html">workspace</a>. Destroy the infrastructure, delete terraform.tfstate and terraform.tfstate.backup file, as we no longer need them.</p><pre>terraform destroy -var-file=dev.tfvars<br>rm -rf terraform.tfstate<br>rm -rf terraform.tfstate.backup</pre><p>Update main.tf add this block below at the top of the file.</p><pre>terraform {<br>  backend &quot;local&quot; {<br>    path          = &quot;terraform.tfstate&quot;<br>    workspace_dir = &quot;workspace&quot;<br>  }<br>}</pre><p>And execute these commands</p><pre>terraform init<br>terraform workspace new dev</pre><p>Now this is our directory structure</p><pre>.<br>├── dev.tfvars<br>├── ec2.tf<br>├── main.tf<br>├── outputs.tf<br>├── providers.tf<br>├── variables.tf<br>└── workspace<br>    └── dev</pre><h3>Recreate dev environment</h3><p>Great, now we have successfully implemented the environment and workspace, but it still not very organized. Let’s create a directory called vars and move dev.tfvars to that directory.</p><pre>mkdir vars<br>mv dev.tfvars vars</pre><p>Now this is our directory structure</p><pre>.<br>├── ec2.tf<br>├── main.tf<br>├── outputs.tf<br>├── prod.tfvars<br>├── providers.tf<br>├── variables.tf<br>├── vars<br>│   └── dev.tfvars<br>└── workspace<br>    └── dev</pre><p>And apply these changes</p><pre>terraform plan -var-file=vars/dev.tfvars<br>terraform apply -var-file=vars/dev.tfvars -auto-approve</pre><p>As you can guess, there are no changes to be expected in the infrastructure as we have only moved the variable file into its own directory</p><h3>Create prod environment</h3><p>Now we have our dev environment working fine, we can create prod environment.</p><p>Create vars/prod.tfvars.</p><pre>project        = &quot;akudev&quot;<br>env            = &quot;prod&quot;<br>region         = &quot;ap-southeast-1&quot;<br>cidr_block     = &quot;10.0.0.0/16&quot;<br>public_subnets = [&quot;10.0.32.0/20&quot;, &quot;10.0.96.0/20&quot;, &quot;10.0.160.0/20&quot;]<br>azs            = [&quot;ap-southeast-1a&quot;, &quot;ap-southeast-1b&quot;, &quot;ap-southeast-1c&quot;]<br>vpn_cidr_block = [&quot;203.81.204.171/32&quot;]</pre><p>And execute these commands</p><pre>terraform workspace new prod<br>terraform workspace select prod<br>terraform plan -var-file=vars/prod.tfvars<br>terraform apply -var-file=vars/prod.tfvars -auto-approve</pre><p>Let’s examine the resources created once execution is complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*68Gx7Lg3O0tAIZQA.png" /></figure><p>And here is our final file and directory structure.</p><pre>.<br>├── ec2.tf<br>├── main.tf<br>├── outputs.tf<br>├── providers.tf<br>├── variables.tf<br>├── vars<br>│   ├── dev.tfvars<br>│   └── prod.tfvars<br>└── workspace<br>    ├── dev<br>    │   └── terraform.tfstate<br>    └── prod<br>        └── terraform.tfstate</pre><pre>4 directories, 10 files</pre><p>To go back to dev environment, use the following commands.</p><pre>terraform workspace select dev<br>terraform plan -var-file=vars/dev.tfvars</pre><h3>Conclusion</h3><p>In this article, we have explored how we can create a complete VPC with Terraform. Creating IT infrastructure manually is prone to human errors such as a typo or missing a step which is very likely to happen. By describing and managing our infrastructure as code, they can be better managed and also be reviewed by others, along with other best practices that exist in the software development lifecycle.</p><p>But when working as a team, the state needs to be stored somewhere in a common location so that everyone can access it, for example, <a href="https://www.terraform.io/docs/backends/types/s3.html">s3</a> is supported for that. I will dive deep in this specific area in my next article so stay tuned!</p><p>We also explored how to use Terraform Workspace. With this feature, we can easily create another environment simply by duplicating an existing file and changing the values accordingly. So when someone in the team asked for a new environment, the process would only require minutes. Also when the environment is done, we can easily destroy it with a single command.</p><p>The journey is still very long, but trust me, Terraform is fun 🔥</p><p><em>Originally published at </em><a href="https://aku.dev/terraform-vpc-introduction/"><strong><em>aku.dev</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2924c937544e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aku-dev/introduction-to-aws-vpc-with-terraform-2924c937544e">Introduction to AWS VPC with Terraform</a> was originally published in <a href="https://medium.com/aku-dev">aku-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Big Data Journey — from zero to hero]]></title>
            <link>https://medium.com/serverless-indonesia/aws-big-data-journey-building-mobile-apps-analytics-service-82aedf9643a1?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/82aedf9643a1</guid>
            <category><![CDATA[laravel]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[php]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Thu, 05 Apr 2018 15:59:02 GMT</pubDate>
            <atom:updated>2020-09-21T03:46:04.799Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>We want to share our experience in building Big Data System that is built using PHP &amp; Laravel with nobody in our team that has experience to Big Data and Scaling it to ~50 Million requests per day.</blockquote><p>To put it simply, we build <strong>Google Analytics for Mobile</strong> with additional features, such as <strong>Campaign</strong> and <strong>Segment</strong>. And at the end of this project, the data will be sent to our Ads Platform (<a href="https://www.aerospike.com/docs/connectors/ashadoop/usecases/asForRTB.html">RTB</a>/DMP/DSP).</p><p>Let’s think from a partner perspective. For example, we have <strong>Partner X</strong>. Partner X<strong> </strong>is having mobile apps that already published in <em>Google play store</em> + <em>Apple apps store</em> and having <strong>10000 Users</strong>. Partner X business is a convenience store and having hundreds of branches across 3 countries: <em>Indonesia</em>, <em>Malaysia,</em> and <em>Singapore</em>.</p><p>They want to <strong><em>monetize their users</em></strong> and <strong><em>having more insight</em></strong> into their users but don’t want to use traditional ads. So they embed our <strong>MAAS (Mobile Analytics as a Service) SDK</strong>, add new terms &amp; agreement (explain that we are capturing data), publish a new version and it is done. Partner X<strong> </strong>users won’t feel any changes, they won’t see any ads on their screen, no UI/UX is changing. Partner X<strong> </strong>can be logged in to our <strong>MAAS Dashboard</strong>, and see some metrics, for example, Daily/Weekly/Monthly Active/Inactive Users, Total Users, Message Sent/Open/Read, etc.</p><p><strong>Campaign </strong>and<strong> Segment </strong>are<strong> </strong>the most important features in this project. Partner X can create Campaign, for example: “We want to reach our Android users by <em>Push Notification</em> were located in <strong><em>Bandung, Indonesia </em></strong><em>a</em>nd give them promotion code, 10% of discount”. From that information, we have segment Android<strong> </strong>and<strong> </strong>Bandung, Indonesia. The Campaign itself, is a promotion code, a 10% discount.</p><blockquote>Not only that, the partner can schedule when the Campaign will be sent : once, daily, weekly, monthly, at exact time, or by using trigger. For example when devices is detected by geofence, beacon or dwelltime.</blockquote><p>This picture below is, to sum up, what we are doing for 18 months. It is an example of Sample App + MAAS SDK push notification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*toiqR39lyscEhnsYOjm7Fw.png" /><figcaption>If you know Sundanese language, you will find this is funny :)</figcaption></figure><h3>Stage 1 —Initial Version</h3><p><strong>Features :</strong></p><ul><li>Android / IOS SDK</li><li>Campaign &amp; Segmentation</li><li>Channel: Push Notification</li></ul><p><strong>Stack :</strong></p><ul><li>Elastic Load Balancer (Classic)</li><li>Elastic Beanstalk — REST API ( Lumen) / m4.large</li><li>Elastic Beanstalk — Dashboard (Yii) / t2.medium</li><li>EC2 for Push Notification (Node Js App) / t2.large</li><li>1 RDS PostgreSQL / r3.xlarge</li><li>1 Elasticache Redis / r3.large</li></ul><p><strong>Why </strong><a href="https://lumen.laravel.com/"><strong>Lumen</strong></a><strong>? </strong>At that time, we need to use a framework that most of our team members understand well. And having the best performance for PHP micro-framework. Not only that, but Lumen is also based on Laravel that is actively maintained very well. Because we believe that this part will be becoming an entry point. If this part is not working well, we will not receive any data, instead, we will get tons of complaints, because our SDK is slowing down the partner apps. We have 3 people to work on this part.</p><p><strong>Why </strong><a href="https://www.yiiframework.com/"><strong>Yii</strong></a><strong>? </strong>Yii is great for creating a Dashboard like a web application, as long as all great features are used, and following the best practices. We have 2 people to work on this part.</p><p><strong>Why Node JS for Push Notification? </strong>Well, it is because of the management decision and found one of the employees is having this experience, so it will be great boosts to this project.</p><p><strong>Why PostgreSQL? </strong>The Management said these projects need to have capabilities to ingest any kind of data. So we think our data must be in a JSON format and easy to query. We need to choose between <strong>MongoDB </strong>or <strong>PostgreSQL + JSONB</strong>. At that time, we cannot find anyone in the company that having MongoDB experience. We also have some research about PostgreSQL JSONB and GIN index. The performance is not that bad compared to MongoDB. And we are using <a href="https://postgis.net/">PostGIS</a> for location features. To us, PostgreSQL<strong> </strong>is<strong> RDBMS with steroids</strong>.</p><p><strong>Why Redis? </strong>For our use case, Redis is used for :</p><ul><li>Yii Query Cache and Metrics</li><li>Dwelltime features by utilizing GET/SET, INCR/DECR, SETEX, SADD, SREM, SCARD, SMEMBERS, and LPUSH/RPOP</li><li>Internal use to calculate how many hit for each endpoint</li><li>Delayed job for sending campaign</li></ul><p>The Initial version is completed at <em>Fri May 20 17:31:26 2016 +0700</em>. It took about 4 months, and of course, it is not working perfectly. We make the PostgreSQL do everything, from ingest and store it on jsonb + gin index, doing OLTP query and doing OLAP query by querying from jsonb. At that time, this architecture is working as expected. Well, because we haven’t so many partners at that time and the data is not big enough.</p><p>NodeJS for sending push notification is looking expensive, too much time &amp; cost to maintain. In the next version, we will utilize SNS to send push notification. The image below is the complete architecture for the initial version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3G8daDUQZfy5QlOIAJ7DNA.png" /><figcaption>MAAS Stage 1 — Initial Version</figcaption></figure><h4>Stage 2 — New Features &amp; Improve Durability</h4><p><strong>New Features :</strong></p><ul><li>Geofence &amp; Beacon</li><li>Dwelltime calculation</li></ul><p><strong>What is Geofence? </strong>In our project, we only utilize <strong>Static Geofence</strong>. <a href="https://github.com/tidwall">Josh Baker</a> has created a gif to show how Static Geofence is work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*kxns7PL8kIt0QCS1B2knWg.gif" /><figcaption>Static Geofence. From <a href="http://tile38.com/">http://tile38.com/</a></figcaption></figure><p>If you notice, there are <strong>3 Red Points</strong>. For our use case, when the devices come across the area, the Partner can do something about it. The first Red Points, we can call it as an event, let say, “enter area”. The second Red Point is “dwelltime”. The third Red Point is “exit area”. Then the partner can create a campaign when the devices are entering the area, for example, <em>“Hey, come to our store, and use coupon code BLABLABLA to receive 50% off discount”</em>. When the devices are on an event (enter/exit/dwelltime), the SDK will send a trigger, and the devices will receive a Push Notification. To put it simply, this is our approach to <strong>Macro Location</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*nRfkdeP_aXW_ACOAAwaprw.png" /></figure><p><strong>What is Beacon?</strong> From <a href="https://en.wikipedia.org/wiki/Bluetooth_low_energy_beacon">Wikipedia</a>, Bluetooth beacons are hardware transmitters — a class of <a href="https://en.wikipedia.org/wiki/Bluetooth_low_energy">Bluetooth low energy</a> (LE) devices that broadcast their identifier to nearby portable electronic devices. For our use case, it is similar to how geofence works but we are using beacon proximity to detect devices. For example, Partner X<strong> </strong>has one huge store in some city, its area is 1000 square meters and has a section to sell <strong>Android Phone</strong>. So, the Partner can create a campaign, for example, <em>“Hey, want to upgrade your Android Phone? Come to the Phone Section, and show this code, and we will give you the best price for your used phone”</em>. When the devices detect the beacon (enter/exit/dwelltime), the SDK will send a trigger, and the devices will receive a Push Notification. To put it simply, this is our approach to <strong>Micro Location</strong>.</p><p><strong>What is Dwelltime?</strong> Let’s continue the example above. When someone goes into a huge store, usually they took 30 minutes, 1 hour or more. When the devices are inside either Geofence or Beacon area for a particular time, it is called dwelltime. The Partner can create a campaign, for example, <em>“Hey, it is your 30 minutes in our store, go to the information centre to retrieve bottled water for free”.</em></p><p><strong>Added Stack :</strong></p><ul><li>EC2 for Laravel Queue Worker, using Redis as Broker (m4.large)</li><li>SNS for push notification</li></ul><p>With this version, we add the Laravel queue. The main reason is, <strong>we don’t want to lose any data</strong>. It is fine if it slow, at least the data is not lost. And we start with 1 listener for all events. Yes, that is not a good approach. For example, we have an <em>auth event</em> and <em>send_data event</em>. In 1 minute, we receive 10 auth and 50 send_data. If we use 1 listener, then the response for the <em>auth event</em> will be very slow. Next week, we create several Laravel queue listener and categorized it by event group. For example, the <em>auth event</em> needs to be executed fast, then we create a listener with 10 workers and group it as a <strong>high priority</strong>. For <em>send_data event</em>, because it doesn’t need to be executed at that time, we can use a listener with fewer workers count and group it as <strong>low priority</strong>. Even though we set 48 to our Supervisor <a href="https://laravel.com/docs/5.6/queues#supervisor-configuration">numprocs</a>, the worker is not scale-out. Then we found <a href="https://deliciousbrains.com/scaling-laravel-aws-elastic-beanstalk-part-1-setting-up-laravel/">deliciousbrains blog</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*E0O5t7LAo71KdoFziIzRfg.png" /><figcaption>I’m asking the author because we are having the same problem.</figcaption></figure><blockquote><strong>Dwelltime</strong> is the most challenging feature to us. Thanks to Laravel queue + Redis. We need a special section to discuss this further.</blockquote><p><strong>SNS</strong> for sending push notification is a good approach. Because we don’t need to manage any server, it is service, paid only for what we use. Fortunately, we haven’t hit any <a href="https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html">limitation of SNS</a> until this project ended.</p><p>The image below is the complete architecture for Stage 2 version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*soHhb94NBnJgHhFrrV982Q.png" /><figcaption>MAAS Stage 2 — New Features &amp; Improve Durability</figcaption></figure><h4>Stage 3 — EC2 Worker &amp; New RDS</h4><p><strong>Added Stack :</strong></p><ul><li>SQS + Elastic Beanstalk <a href="https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html">Worker Tier</a> (m4.large &amp; m3.medium)</li><li>New RDS to store <em>send_data event </em>(r3.large)</li></ul><p>In this new version, we replace Redis with SQS. With this architecture, we have no worries to scale the worker to consume SQS Messages. We can create <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html">Cloudwatch alert</a> to add new 1 instances to worker pool when messages &gt; 10000 and remove 1 instance from the worker pool when messages &lt; 1000.</p><blockquote>Please do remember, that number is work for us, because 1000 messages in SQS is can easily consumed by 1 instances in few seconds.</blockquote><p>Not only we gain more performance, but we also gain more durability. When there is something wrong with the code, storage, or unknown error, we can always <strong>retry</strong> it by utilizing the <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html">SQS Dead Letter Queue</a>.</p><p>After introducing SQS to our architecture, our SQS is receiving <strong>11.2 million messages</strong>. We forget how many instances &amp; how much time needed to process these messages, We’re sorry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KMorFVgVdxUDakBWbbcD_Q.png" /></figure><blockquote>Big thanks to <a href="https://github.com/dusterio/laravel-aws-worker">Denis Mysenko</a> for creating this awesome package.</blockquote><p>Our worker tier code is still the same, but the only changes are at routing. The worker has routing only for consuming the SQS messages.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/408866f6f97c6a2a928903f492f8a02e/href">https://medium.com/media/408866f6f97c6a2a928903f492f8a02e/href</a></iframe><p>After we are having no problem with scaling the worker, now we have a problem with RDS. Our storage is almost full.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ZWUl3vgCv012BaqwhiotA.png" /><figcaption>Only a few days to its maximum storage of 500 GB</figcaption></figure><p>We can’t just upgrade the storage of RDS without having downtime. Unfortunately, at that time, RDS Aurora is only available for the MySQL version. Our goal is <strong>not to lose any data</strong>, so our plan is to add new RDS with bigger storage (<strong>2000 GB</strong>), normal IOPS, then change the destination of /send_data endpoint to our new RDS. Then our Dashboard has consumed the data from that new RDS with extra date condition because our old data is not there.</p><blockquote>Old RDS is still used for OLTP query, like authentication, storing device token, etc.</blockquote><p>The image below is the complete architecture for Stage 3 version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nqn8grAW9U9_1lCTz8LPUw.png" /><figcaption>MAAS Stage 3 — SQS and Elastic Beanstalk Worker Tier</figcaption></figure><h4>Stage 4 — Firehose &amp; Redshift</h4><p><strong>Added Stack :</strong></p><ul><li>New SQS Worker</li><li>Firehose + Redshift</li></ul><p><strong>Redshift</strong> is a product from AWS that specialize for doing OLAP query. Mostly used as Data Warehouse. The challenge by utilizing Redshift in our architecture is the <strong>ETL</strong> part. Because we are having JSON, while Redshift must be in columnar format, then we needed a specific worker to send data to Redshift. The worker is doing the transformation from JSON to column format. Not only that, but we also need to change, big change, to our query that is used by the Dashboard, it is really a big job.</p><blockquote>We made mistake by doing full ETL from RDS to Redshift while the result is not being used by anyone. It cost us development time &amp; cost of cloud provider.</blockquote><p>We are also designing the table by adding suffix <strong>_{year}{month}</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*ELdIAz7mbUUswA1qYefBMg.png" /><figcaption>Separating the tables by year and month.</figcaption></figure><p>The drawback part of this method is we need to create separate firehose delivery stream and separate Redshift table manually. Because in our worker, it will automatically detect <strong>what month &amp; year right now?</strong> and send it to the respective delivery stream. At one moment, We completely forget to create that process, as result, our SQS queue is taking too many messages. Our DLQ contains <strong>7.5 Million</strong> messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RgZpwXoXbbJZbWMnC5KjEQ.png" /><figcaption>Fortunately, we are having SQS</figcaption></figure><p>Loading data to Redshift is made easy by using <a href="https://docs.aws.amazon.com/ses/latest/DeveloperGuide/event-publishing-redshift-firehose-stream.html#event-publishing-redshift-firehose-stream-copy">Firehose</a>. Firehose will create a temporary S3 file, and then Firehose will invoke Redshift COPY command.</p><blockquote>Please do remember, ETL process before sending data to S3 is really important. Having 1 line of error in random S3 CSV file will result whole COPY command is failing. Please make sure the data is clean.</blockquote><p>We are also screwed up with Redshift by using it to do the wrong Query. If We remember correctly, it is OLTP query to search some string with LIKE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*gkGciN0qmDps6h8koLTtZQ.png" /><figcaption>Never-ending query, we need to kill it manually</figcaption></figure><p>At that time, Redshift is only available in 3 Region, and one of them is in Ireland. Fortunately (and also unfortunate), most of our service is in Tokyo, because our users are coming from South East Asia. The latency to Ireland from Tokyo is not really high. Then the unfortunate part is data transfer cost because we send data to another region.</p><blockquote>Our awesome CTO is invite AWS Solution Architect from Singapore to come to our Bandung office to help us solve the Redshift problem.</blockquote><p>The image below is the complete architecture for Stage 4 version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MApb-U9UpNPBQJzI35Gy0A.png" /><figcaption>MAAS Stage 4 — Firehose &amp; Redshift</figcaption></figure><h4>Stage 5 — Enter Serverless</h4><p><strong>Added Stack :</strong></p><ul><li>Kinesis Stream + Lambda via <a href="https://github.com/Miserlou/Zappa">Zappa</a></li><li>EMR + Athena</li><li>Terraform</li></ul><p>The story continues when our cost is too much. That being said, we need to think about how we are storing the data. After we understand how to handle <strong>Hot</strong> data, <strong>Warm</strong> data, and <strong>Cold</strong> Data, we are removing Redshift from our architecture, it cost too much while it doesn’t give extra value to the company.</p><p>Now we are planning to send raw data to S3. While the metrics for the dashboard is coming from Redis and PostgreSQL in pre-calculated format by EMR and some worker. Our big mistake is doing the calculation for Dashboard metrics from the raw data table and naively said query cache will be working well.</p><p>Now we are creating 2 Data Pipeline, <strong>Realtime, </strong>and <strong>Batch</strong>. We still use SQS for Realtime Pipeline because we need the retry feature. On the other side, Batch Pipeline is utilizing Kinesis Stream with Lambda to consume the messages. The combination of Kinesis Stream &amp; Lambda is the best choice for us. Because we don’t need to think about the server, especially when we split the Kinesis stream shard, lambda will take care of that problem. We use <a href="https://www.zappa.io/">Zappa</a> to deploy our Lambda function.</p><p>It is really fortunate that we move the raw data to S3 because in a few next days we get alert that <strong>storage-full</strong> on RDS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tu10JwaZewPYOqLH3uyZkg.png" /><figcaption>We are really doomed</figcaption></figure><p>When we are building the Batch Pipeline, we are actually doing a double write. Just to make sure we are not losing any data. The consequences is the cost doubling up and it cost more time to validate &amp; cleaning the data between S3 &amp; RDS. It is fine because we are having a dedicated team to work for this part.</p><p>The most entertaining part is when we are utilizing <a href="http://terraform.io/">Terraform</a> to our architecture. It is really fun to do everything in code. The most important thing is we can easily replicate the architecture and minimizing human error. We have no time to terraforming everything, but the data pipeline part is already in Terraform.</p><p><strong>Kinesis Stream</strong> is received a small chunk of events from our application. Lambda is doing aggregation and combine it to big chunk then send to firehose via PutRecordBatch. Our Kinesis Stream is located the same as our main application (Tokyo), while Firehose and S3 are located in North Virginia.</p><blockquote>Please do remember, Kinesis Stream &amp; Kinesis Firehose is have limitation. I write how to “scaling” Kinesis Firehose. And this is work for us.</blockquote><p><a href="https://medium.com/@clasense4/how-to-scaling-aws-kinesis-firehose-4ef4bea590e0">How to Scaling AWS Kinesis Firehose</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*lzjKEtjQyGl23A9XJdVXGw.png" /></figure><p>After a few days we implemented the Batch Pipeline, we receive <strong>5GB Daily in S3 with CSV format</strong>. And it is processed by the Data team by utilizing Apache Spark. <a href="https://medium.com/u/4be0cd334cc2">Ridwan Fajar</a> is creating a dedicated page about how this is working.</p><p><a href="https://medium.com/serverless-indonesia/a-story-about-managing-tracking-log-files-from-maas-with-aws-kinesis-firehose-aws-s3-apache-spark-9ffa2c2b6195">A Story About Managing Tracking Log Files from MAAS with AWS Kinesis Firehose, AWS S3, Apache Spark…</a></p><p>We also utilizing Athena for quick &amp; simple queries in S3. The data team is converting the CSV to ORC, so it is easy for us to query. Not long after Athena released, <a href="https://aws.amazon.com/redshift/spectrum/">Redshift Spectrum</a> is out. We want to try it, but no luck*.</p><p>The highest traffic we receive is <strong>~50 Million hit</strong> / <strong>Day</strong> on Elastic Beanstalk. We are limiting it to Minimum 6 and Maximum 10. The instance type is m4.xlarge. And please remember, that is a PHP Application, Lumen Micro Framework with default Elastic Beanstalk setting. If we tune everything or change to another programming language that is better than PHP, maybe we can cut the cost and improving our response time. But at that time, if we have to choose to improve, we are really considering to make use of <a href="https://www.swoole.co.uk/"><strong>Swoole</strong></a><strong>. </strong>Because it is running well and it is not broken, we don’t need to fix it. We can invest our time to build another part of this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtUGIp9VhbIcKJ9biwr4cQ.png" /><figcaption>Not a bad number</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*FT4VFip5p8zQKap9MOCb9w.png" /><figcaption>Our S3 Data before Bandung office and its employee is shutting down</figcaption></figure><p>After 4 months implemented this new architecture, our management is having a problem and wants to close the Indonesia office which is located in Bandung and Jakarta. We are doing great on cutting cloud provider costs by utilizing S3 and by Reserved Instances, also Spot Instances. But that’s still not enough, too much hidden cost besides the cloud provider cost.</p><p><strong>Why not 100% serverless architecture? </strong>Our CTO is planned to move everything to serverless, but at that time we don’t have any scaling problem on the Rest API part. We try to calculate our API Gateway cost for 50 Million hits/ day, and the cost is too much high. It is $3.50 / Million hit.</p><p><strong>Why not trying containers? </strong>If only we had more time. We want to try ECS, because we have some research on it, and asking 3 AWS Solution Architect, they all recommended us to move from Elastic Beanstalk to ECS. The important thing is AWS will manage the container for us, and ECS itself is free.</p><p>The image below is the complete architecture for Stage 5 version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BdSLywipDJtB7JrTAebOig.png" /><figcaption>MAAS Stage 5 — Enter Serverless</figcaption></figure><h3>Lesson Learned</h3><ol><li>If you having a Database, and not planning it to do some scale up/down, then you need to use Reserved Instances, not On Demand.</li><li>Try to utilize Spot Instances as much as your team can.</li><li>Separate the AWS Account for Development &amp; Production. It is for security concerns and easier to calculate the cost.</li><li>If having multiple AWS Account is not possible, try to utilize AWS Tag to track your project&#39;s cost.</li><li>Pay attention to REST API Response. Do not send too much unnecessary response, it will be accumulated into your <a href="https://aws.amazon.com/ec2/pricing/">AWS Data Transfer</a> bill ($0.09/GB). If you only need only HTTP Header, then go with it.</li><li>Make sure the database you are using is the right tool for the right job.</li><li>Having/building internal analytics for application is really important, it is to make sure that the feature your team build is used by the client.</li><li>If you are having a repeated task that needs to be done for more than 1 minute, automate it.</li><li>If you are building a big project, especially from scratch, having too many screenshot/camera photos for those projects is completely fine.</li><li>If you are using Slack, you can create a bot to receive every message, and export it to permanent storage.</li></ol><h3>Conclusion</h3><p>This is the saddest part of our journey, we haven’t done everything we can, but the management chooses to shut down the operation. We build these projects from zero (even minus) to (not yet) hero. None of us is having experience with AWS, especially the data we are talking about is not small, and will eventually growing as the number of clients is increasing.</p><p>We are really grateful for this experience, not every engineer can have the same experience as us, what we can do is share our little experience. We hope nobody is doing the same mistake as us.</p><p>That whole architecture is a part of the “Input” process. We have another team to work on “Process” and “Output”. We will create a dedicated page for that.</p><h3>Many thanks for The Team</h3><blockquote>This architecture is built by the team. So we have to say thank you to all persons that involved on built this architecture. Here are our friends that built this awesome architecture : Our best CTO <a href="https://medium.com/u/f58aa75e17d">Elaia Raj</a>, Our best Manager Toshi Firmansyah &amp; Budi Satria, <a href="https://medium.com/u/ae0b7472e817">Tajhul Faijin Aliyudin</a>, <a href="https://medium.com/u/2f6060739de2">fajri abdillah</a>, Wira Sakti, Rinaldi Guarsa &amp; <a href="https://medium.com/u/4be0cd334cc2">Ridwan Fajar</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82aedf9643a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/serverless-indonesia/aws-big-data-journey-building-mobile-apps-analytics-service-82aedf9643a1">AWS Big Data Journey — from zero to hero</a> was originally published in <a href="https://medium.com/serverless-indonesia">Serverless Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pengguna Layanan AWS Terbesar]]></title>
            <link>https://medium.com/serverless-indonesia/pengguna-layanan-aws-terbesar-d7e689fa78a3?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7e689fa78a3</guid>
            <category><![CDATA[netflix]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-user]]></category>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[airbnb]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Tue, 27 Mar 2018 02:43:56 GMT</pubDate>
            <atom:updated>2018-03-27T02:43:56.786Z</atom:updated>
            <content:encoded><![CDATA[<h3>Slack</h3><p>Dengan menggunakan AWS Global Deployment, Slack saat ini memiliki <strong>5 Juta Daily Active User</strong> dan lebih dari <strong>30 Juta messages mengalir melalui Slack setiap minggu</strong> baik itu melalui integrasi dengan layanan lain, maupun di dalam internal slack.</p><p>Banyak orang yang menggunakan Slack hingga <strong>10 Jam atau lebih</strong>, berbeda dengan aplikasi seperti Facebook atau Twitter, yang orang hanya sesekali mengecek disaat meeting, atau waktu senggang lainnya.</p><p>Slack menggunakan beberapa produk AWS seperti EC2 dengan tipe instance I3 yang support NVME SSD, Elastic Load Balancer, EMR, S3, VPC, Route53, Athena, SNS, SQS, CloudFront dan CloudWatch.</p><p>Salah satu keunggulan Slack adalah kemudahan membuat aplikasi diatas Slack, seperti integrasi dengan github, jenkins dan pagerduty. Saat ini AWS <a href="https://aws.amazon.com/lex/">Lex</a> sudah terintegrasi dengan Slack. Dengan Lex, membuat bot menjadi sangat mudah.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2yjW483z2BoUB2Ey.png" /></figure><p>Sumber :</p><ul><li><a href="https://www.youtube.com/watch?v=vdRizqgyOPE">https://www.youtube.com/watch?v=vdRizqgyOPE</a></li><li><a href="https://aws.amazon.com/solutions/case-studies/slack/">https://aws.amazon.com/solutions/case-studies/slack/</a></li></ul><h3>Netflix</h3><p>Dengan menggunakan AWS Global Deployment, Netflix mampu menjangkau banyak negara, mampu membuat banyak instance aws untuk dipergunakan dalam skenario testing maupun development dalam hitungan menit.</p><p>Pada tahun 2009, Netflix <strong>beralih menuju Infrastruktur cloud AWS</strong>. Developer / Engineer Netflix diberi <strong>kebebasan</strong> untuk menjalankan instance di atas AWS. Sedangkan di masa lalu, mereka harus bicara kepada tim Infra, dan ini membutuhkan waktu yang sangat lama.</p><p>Layanan AWS bisa membuat Netflix untuk melakukan scale secara <strong>elastis</strong>, yaitu ketika traffic meningkat, mereka bisa menambah instance, dan ketika traffic sudah turun mereka bisa mengurangi jumlah instance, dan ini adalah hal yang sangat penting bagi perusahaan internet global seperti Netflix</p><p><strong>Netflix dalam angka</strong></p><ul><li>&gt; 81 Juta Anggota</li><li>&gt; 125 Juta Jam Video Streaming / hari</li><li>Lebih dari 1/3 dari Traffic Internet di Amerika Utara</li><li>600 Miliar Event / hari</li><li>40 Peta Bytes di dalam Data Warehouse</li><li>3 Peta Bytes Read / hari</li><li>300 TB Write / hari</li><li>Di tahun 2016 sudah menjangkau banyak negara, negara besar kecuali china</li></ul><p>Sumber :</p><ul><li><a href="https://www.youtube.com/watch?v=lQGHsBOZJBw">https://www.youtube.com/watch?v=lQGHsBOZJBw</a></li><li><a href="https://www.youtube.com/watch?v=hTfIAWhd3qI">https://www.youtube.com/watch?v=hTfIAWhd3qI</a></li></ul><h3>Airbnb</h3><p>Airbnb adalah sebuah community marketplace yang memperbolehkan pemilik properti dan traveler untuk saling terhubung satu sama lain dengan tujuan menyewa tempat berlibur yang unik di seluruh dunia. Aktivitas pengguna komunitas Airbnb kebanyakan dilakukan melalui website perusahaan dan melalui aplikasi iPhone / Android. Saat ini Airbnb mendukung penyewaan property di <strong>25,000 kota</strong> dan dalam <strong>192 Negara</strong>.</p><p>Airbnb menggunakan AWS <strong>sejak hari pertama mereka memulai bisnis</strong>, dan hingga tahun 2013 Airbnb sudah memiliki <strong>9 Juta</strong> pengunjung, dari angka tersebut tidak menunjukan sedikitpun pengurangan pengunjung, hingga pada suatu malam, jumlah pesanan Airbnb mencapai angka <strong>150,000</strong>.</p><p>Beberapa produk yang digunakan Airbnb diantaranya EC2, ELB, EMR, RDS, S3, DynamoDB, Elasticache, dan Route53.</p><p>Pada tahun 2013 data Airbnb sudah mencapai <strong>2 Miliar baris data</strong> yang tersimpan dalam RDS. Jumlah instance EC2 sudah mencapai <strong>1000</strong>. <strong>50 TB</strong> data foto tersimpan di S3. Yang lebih menarik, untuk menangani infrastruktur sebesar itu, cukup ditangani oleh <strong>5 orang yang berada di divisi operations</strong>.</p><p>Sumber :</p><ul><li><a href="https://aws.amazon.com/solutions/case-studies/airbnb/?hp=tile">https://aws.amazon.com/solutions/case-studies/airbnb/?hp=tile</a></li></ul><h3>Kesimpulan</h3><p>Slack, Netflix dan Airbnb memiliki mimpi yang sama, yaitu ingin memberikan pelayanan yang terbaik kepada para penggunanya di seluruh dunia. Dan mereka semua memilih AWS sebagai partner, karena mereka yakin dengan layanan dan kemudahan yang ditawarkan oleh AWS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7e689fa78a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/serverless-indonesia/pengguna-layanan-aws-terbesar-d7e689fa78a3">Pengguna Layanan AWS Terbesar</a> was originally published in <a href="https://medium.com/serverless-indonesia">Serverless Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Berkenalan dengan Amazon Web Services (AWS)]]></title>
            <link>https://medium.com/serverless-indonesia/berkenalan-dengan-amazon-web-services-aws-13c65f3b0e5f?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/13c65f3b0e5f</guid>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Tue, 27 Mar 2018 02:41:57 GMT</pubDate>
            <atom:updated>2018-03-27T02:41:57.778Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apa itu AWS?</h3><p>Dimanapun Anda menjalankan bisnis, baik itu bisnis retail, real estate, finance, social media, Anda membutuhkan aplikasi yang berjalan dengan baik dalam menunjang bisnis tersebut. Aplikasi harus <em>reliable</em>, <em>secure</em> dan <em>cost efficient</em>.</p><p><a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS) adalah penyedia layanan cloud yang aman, AWS menawarkan tenaga komputasi, ruang penyimpanan database, “<em>content delivery network</em>” dan fungsionalitas lainnya yang membantu banyak bisnis untuk berkembang dan menjalankan aplikasi dengan baik.</p><p>AWS memberikan banyak pilihan produk yang sangat memudahkan dalam membangun bisnis. <em>Infrastructure as a service</em>, adalah istilah yang tepat untuk AWS. Istilah lainnya ada juga <a href="https://m.oursky.com/saas-paas-and-iaas-explained-in-one-graphic-d56c3e6f4606"><em>Software as a service dan Platform as a service</em></a>.</p><p>Beberapa produk AWS telah didesain untuk memudahkan pengguna, contohnya <a href="https://aws.amazon.com/ec2/"><strong>Elastic Compute Cloud</strong></a> atau EC2. Pada dasarnya EC2 adalah sebuah server virtual. Kemudahan yang ditawarkan oleh EC2 yaitu jika kita bandingkan dengan server tradisional.</p><p>Pada server tradisional, kita harus mengetahui secara detail spesifikasi server yang kita inginkan seperti apa, <em>support</em>, <em>spare part</em>, lalu penempatan di <em>data center</em>, dan proses tersebut bukanlah proses yang mudah. Jika kita menyiapkan terlalu banyak server, sedangkan kebutuhan komputasi tidak banyak, kita akan <strong>membuang banyak biaya</strong>. Sedangkan jika kita menyiapkan sedikit server sedangkan kebutuhan komputasi sangat banyak, hal ini akan <strong>menggangu jalannya bisnis</strong>.</p><p>Jika hari ini membutuhkan <strong>10 server</strong>, sedangkan besok membutuhkan <strong>20 server</strong>, lalu lusa menjadi <strong>1 server</strong>. Hal ini sangat dimungkinkan oleh EC2. Bahkan, beberapa bisnis besar seperti <a href="https://medium.com/netflix-techblog/the-evolution-of-container-usage-at-netflix-3abfc096781b">Netflix</a>, mereka sanggup membangun <strong>lebih dari 1 juta container</strong> diatas <strong>500 r3.8xl instances</strong>.</p><p>AWS EC2 terdiri dari bermacam-macam jenis instance, seperti tipe <strong>&quot;compute&quot;</strong>yang menggunakan kode : c4.large atau c3.xlarge, tipe <strong>&quot;general purpose&quot;</strong>yang menggunakan kode : m4.large atau m3.xlarge. Kita bisa memilih sesuai dengan keperluan.</p><p>Lalu contoh berikutnya dari produk AWS adalah <a href="https://aws.amazon.com/kinesis/streams/"><strong>Kinesis Streams</strong></a>. Kinesis Streams adalah produk AWS yang berbasis Stream. Salah satu produk open source yang mendukung teknologi stream yaitu <a href="https://kafka.apache.org/"><strong>Apache Kafka</strong></a>. Jika menggunakan Apache Kafka, kita harus mengetahui cara mengoperasikannya, cara melakukan <em>scaling</em>, memastikan <em>data durability</em>, <em>disaster recovery</em>, <em>high availability</em> dan lainnya.</p><p>Dengan menggunakan Kinesis Streams, kita tidak perlu memikirkan <em>scaling</em>, tidak perlu memikirkan <em>data durability</em>, <em>disaster recovery</em>, <em>high availability</em> dan kita bisa dengan segera menggunakan layanan streaming tersebut.</p><h3>Siapa pengguna AWS?</h3><p>Netflix, Slack, dan Airbnb merupakan contoh pengguna AWS, mereka menggunakan <em>Global Deployment</em> untuk menjangkau pengguna di seluruh dunia.</p><p><strong>Baca juga: Pengguna layanan AWS Terbesar</strong></p><h3>Kapan menggunakan AWS?</h3><p><a href="https://www.youtube.com/watch?v=xQBNn67rL_o">Airbnb</a> menggunakan AWS sejak pertama kali bisnis mereka dijalankan. Bisnis seperti Airbnb yang berkembang sangat pesat, membutuhkan layanan penyedia cloud yang sangat cepat juga. Kata cepat tersebut bisa kita ambil contoh <strong>ketersediaan server</strong>. Ketika saat itu diperlukan <strong>100 server</strong>, aws bisa melakukannya dengan mudah dan dalam hitungan menit. Proses <em>scaling</em>tersebut sangat berdampak pada berjalannya bisnis, kita tidak membuat sesuatu kesalahan pada aplikasi yang disebabkan kekurangan server atau server tidak mampu menangani <em>request</em>.</p><p><a href="https://www.youtube.com/watch?v=lQGHsBOZJBw">Netflix migrasi pada tahun 2009</a> dari server tradisional ke cloud provider AWS. Jika sudah pernah mengalami sulitnya berhadapan dengan data center tradisional, dengan bermigrasi ke cloud provider aws, pasti akan merasa lebih nyaman, karena tidak perlu terlalu memikirkan server dan tidak diperlukannya <em>standby</em> di Data Center.</p><p>Jika target rencana besar Anda adalah seluruh dunia, maka AWS adalah salah satu provider cloud yang bisa <strong>mewujudkan mimpi Anda</strong>. Saat ini, AWS memiliki <strong>16 Region</strong> yang tersebar di beberapa wilayah, seperti Amerika Utara, Amerika Selatan, Eropa, Asia Pasifik. Region baru yang sedang dalam proses yaitu Swedia, Prancis, China, Hong Kong dan Gov Cloud. Region yang populer bagi pelaku bisnis di Indonesia adalah region Singapore dan Tokyo.</p><h3>Kenapa menggunakan AWS?</h3><p>Selain menawarkan kemudahan, salah satu keutamaan dari AWS adalah fitur “<em>pay as you go</em>”, yaitu bayar ketika dipergunakan. Seperti AWS EC2, <a href="https://aws.amazon.com/ec2/pricing/"><em>pricing model</em></a> yang digunakan adalah per jam untuk jenis deployment on-demand. Lalu untuk <a href="https://aws.amazon.com/kinesis/streams/pricing/">Kinesis Streams</a> adalah <em>shard hour</em>, <em>PUT Payload Unit</em>, dan <em>extended data retention</em>.</p><blockquote>AWS menyediakan <a href="https://calculator.s3.amazonaws.com/index.html">simple monthly calculator</a> yang bisa kita pergunakan untuk memprediksi berapa biaya yang akan dikeluarkan</blockquote><p>Dan dengan biaya yang cukup murah, kita memiliki layanan yang powerful, contohnya kinesis streams, tanpa perlu me-<em>manage</em> aplikasi streaming seperti apache kafka, <strong>kita bisa menyelesaikan masalah yang sama dengan waktu yang lebih singkat</strong>. Efeknya, kita bisa men-<em>deliver</em> fitur lebih cepat, kita bisa fokus pada pengguna, kita bisa fokus pada bisnis, dan kita tidak perlu memikirkan &quot;<strong>scaling</strong>&quot;.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13c65f3b0e5f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/serverless-indonesia/berkenalan-dengan-amazon-web-services-aws-13c65f3b0e5f">Berkenalan dengan Amazon Web Services (AWS)</a> was originally published in <a href="https://medium.com/serverless-indonesia">Serverless Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Berkenalan dengan DevOps]]></title>
            <link>https://medium.com/serverless-indonesia/berkenalan-dengan-devops-15c9e80adab2?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/15c9e80adab2</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops-engineer]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[iaac]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Tue, 27 Mar 2018 02:36:44 GMT</pubDate>
            <atom:updated>2018-03-27T02:39:32.518Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*eGN4spZp1KvcbbKRQlCW6g.jpeg" /></figure><h3>Apa Itu DevOps</h3><p>DevOps adalah serangkaian praktik yang mengotomatiskan proses antara pengembangan aplikasi dan tim pengembang agar mereka dapat melakukan proses <em>build</em>, <em>test</em> dan <em>release</em> perangkat lunak lebih cepat dan lebih handal. DevOps yang sudah berjalan dengan baik akan menghasilkan produk yang <strong>stabil</strong>dan <strong>meningkatkan nilai</strong> dari produk itu sendiri.</p><h3>Kenapa DevOps Sangat Penting dalam Industri IT</h3><p>Salah satu best practices dalam DevOps yaitu melakukan <em>update</em> produk yang kecil dan cepat. Dalam industri IT, ini bisa disebut dengan <em>minor</em> / <em>patch</em> <em>update</em>. Dengan melakukan <em>update</em> produk yang kecil, organisasi/perusahaan dapat melakukan inovasi lebih cepat bagi produknya, juga dapat melakukan <em>rollback</em>dengan mudah jika ditemukan <em>bug</em> yang tidak ter-<em>cover</em> oleh <em>automated testing</em>maupun tim QA. Dengan praktek ini, Organisasi/Perusahan dapat me-<em>monitor</em>produk dan mendapatkan data yang paling berharga yaitu data <strong>pengguna</strong>.</p><h3>Bagaimana Praktek DevOps?</h3><h3>Continuous Integration</h3><p><em>Continuous Integration</em> adalah praktek <em>software development</em> dimana para pengembang dengan <em>regular</em> melakukan perubahan kode dan melakukan <em>merge</em>pada sebuah <em>repository</em>, dan ketika terdapat <em>push commit</em> baru, <em>automated build &amp; test</em> dijalankan, dan hasil akan segera diterima. Jika terjadi kesalahan/bug dalan build/test, tim pengembang dapat dengan cepat melakukan update code sehingga <strong>kualitas produk bisa ditingkatkan</strong>, dan <strong>mengurangi waktu</strong> untuk melakukan validasi sebuah update.</p><h3>Continuous Delivery</h3><p><em>Continuous Delivery</em> adalah praktek <em>software development</em> dimana para pengembang yang melakukan perubahan pada <em>code</em>, sudah melakukan <em>build &amp; test</em> yang dijalankan otomatis oleh <em>Continuous Integration</em> dan siap untuk <em>deploy</em> ke <em>environment production</em> namun memilih untuk tidak <em>deploy</em> yang umumnya karena masalah bisnis. Ketika <em>Continuous Delivery</em> dijalankan dengan baik, tim pengembang akan selalu memiliki <em>build artifact</em> yang siap untuk di <em>deploy</em> ke <em>environment production</em>.</p><h3>Continuous Deployment</h3><p><em>Continuous Deployment</em> merupakan salah satu rangkaian setelah <em>Continuous Integration</em> dan <em>Continuous Delivery</em> selesai dijalankan. Umumnya organisasi/perusahaan memiliki <em>environment test / development</em>, dan disinilah fungsi utama <em>continuous deployment</em>, yaitu ketika hasil dari <em>Continuous Integration</em> sudah dinyatakan baik, tim pengembang dapat segera melihat perubahan pada <em>environment test / development / production</em>.</p><p><em>Continuous Deployment</em> merupakan cara yang sangat baik untuk meningkatkan jumlah <em>feedback</em> yang diterima oleh pengguna aplikasi dan mengurangi tekanan karena tidak ada lagi <strong>“Hari Rilis”</strong>. Tim pengembang dapat fokus dalam membangun produk dan dapat <strong>melihat perubahan code</strong> mereka <strong>beberapa menit</strong> setelah mereka selesai mengerjakannya.</p><p>Aplikasi populer yang mendukung dalam <em>Continuous Integration</em>/<em>Continuous Delivery</em>/<em>Continuous Deployment</em> diantaranya <a href="https://jenkins.io/">Jenkins</a>, <a href="https://about.gitlab.com/features/gitlab-ci-cd/">Gitlab CI/CD</a> <a href="https://bitbucket.org/product/features/pipelines">Bitbucket Pipeline</a>, <a href="https://semaphoreci.com/">Semaphore CI</a>, <a href="https://travis-ci.org/">Travis CI</a>, <a href="https://circleci.com/">Circle CI</a>, <a href="https://aws.amazon.com/codebuild/">AWS Code Build</a> dan <a href="https://aws.amazon.com/codepipeline/">AWS Code Pipeline</a>.</p><h3>Configuration Management</h3><p><em>Configuration Management</em> adalah praktek dalam proses <em>System Engineering</em>yang memiliki tujuan untuk me-<em>maintain</em> konfigurasi sebuah produk, dan memastikan konsitensinya dalam seluruh <em>environment</em>. Dengan menggunakan <em>Configuration Management</em>, proses konfigurasi produk dapat diotomatisasi, distandardisasi dan mengurangi proses konfigurasi yang manual. Tahap selanjutnya, <em>Configuration Management</em> akan mempermudah dalam konfigurasi banyak server dan dapat <strong>meminimalisir kesalahan</strong>, karena konfigurasi <strong>ditulis dalam code</strong>, tidak lagi menjalankan perintah manual.</p><p>Aplikasi populer yang mendukung dalam Configuration Management diantaranya <a href="https://www.ansible.com/">Ansible</a>, <a href="https://www.chef.io/">Chef</a>, <a href="https://puppet.com/products/open-source-projects">Puppet</a>, <a href="http://capistranorb.com/">Capistrano</a>, <a href="http://www.fabfile.org/">Fabric</a> dan <a href="https://saltstack.com/salt-open-source/">SaltStack</a>.</p><h3>Infrastructure as a Code</h3><p><em>Infrastructure as a Code</em> adalah sebuah praktek dalam <em>System Architecture</em> yang mana infrastruktur sebuah produk didefinisikan dalam code yang dapat diprogram, distandardiasikan dan mudah untuk di duplikasi. Produk skala menengah, mungkin membutuhkan lebih dari satu mesin. Dengan <em>IAAC</em>, tim pengembang dapat dengan mudah menambah mesin <strong>melalui satu baris kode</strong>.</p><p>Produk dalam skala besar, umumnya bergantung pada beberapa produk yang ditawarkan oleh<em>provider cloud</em>, seperti provider aws yang menawarkan produk seperti <a href="https://aws.amazon.com/ec2/">AWS EC2</a> dan <a href="https://aws.amazon.com/kinesis/data-streams/">AWS Kinesis Data Stream</a>. Dengan <em>IAAC</em>, tim pengembang tidak lagi melakukan operasi manual pada <em>console provider cloud</em>, cukup dengan merubah <strong>beberapa baris kode</strong>, hal yang lebih penting yaitu dapat <strong>meminimalisir kesalahan</strong>.</p><p>Aplikasi populer yang mendukung IAAC diantaranya <a href="https://www.terraform.io/">Terraform</a>, <a href="https://www.ansible.com/products/tower">Ansible Tower</a>, <a href="https://www.chef.io/">Chef</a>, <a href="https://puppet.com/products/open-source-projects">Puppet</a>, <a href="https://saltstack.com/salt-open-source/">SaltStack</a> dan <a href="https://aws.amazon.com/cloudformation/">AWS CloudFormation</a>.</p><h3>Monitoring</h3><p>Sebuah produk haruslah di-<em>monitoring</em> untuk mengetahui bagaimana produk digunakan oleh pengguna. Dalam praktek DevOps, <em>monitoring</em> merupakan hal yang <strong>sangat penting</strong>. Tim pengembang harus mengetahui bagaimana perubahan kodenya berdampak pada <strong>produk</strong> juga <strong>penggunanya</strong> melalui <em>monitoring tools</em>.</p><p>Aplikasi monitoring yang populer diantaranya <a href="https://www.nagios.org/">Nagios</a>, <a href="https://www.zabbix.com/">Zabbix</a>, <a href="https://sensuapp.org/">Sensu</a>, <a href="https://www.cacti.net/">Cacti</a>, <a href="https://opennms.org/">OpenNMS</a>, <a href="https://www.icinga.com/">Icinga</a>, <a href="https://prometheus.io/">Prometheus</a>, <a href="https://grafana.com/">Grafana</a> dan <a href="https://aws.amazon.com/cloudwatch/">AWS CloudWatch</a>.</p><h3>Logging</h3><p>Log aplikasi adalah salah satu cara untuk mengetahui apakah produk kita berjalan dengan baik atau tidak. Namun seiring dengan tingkat kompleksitas sebuah produk, ada banyak log komponen yang harus diterima dan dianalisis. Dan log tersebut haruslah terpusat, tidak terpisah-pisah. Maka istilah <em>Centralized Logging</em> merupakan hal yang <strong>tidak dapat dipisahkan</strong> dalam praktek DevOps.</p><p>Aplikasi logging yang populer diantaranya <a href="https://www.elastic.co/elk-stack">ElasticSearch + Logstash + Kibana</a>, <a href="https://www.splunk.com/">Splunk</a>, <a href="https://www.graylog.org/">Graylog</a>, <a href="https://aws.amazon.com/cloudwatch/">AWS Cloudwatch</a> dan <a href="https://aws.amazon.com/kinesis/data-firehose/">AWS Kinesis Data Firehose</a>.</p><h3>Communication &amp; Collaboration</h3><p>Salah satu aspek utama dalam praktek DevOps yaitu meningkatnya <strong>komunikasi dan kolaborasi</strong> dalam sebuah organisasi/perusahaan, baik dalam bentuk fisik maupun non fisik. Praktek DevOps yang berjalan dengan baik, akan meningkatkan aspek komunikasi dan kolaborasi tidak hanya pada tim pengembang, namun juga tim marketing, sales, operations, dan tim lain yang ada didalam organisasi/perusahaan.</p><p><a href="https://slack.com/">Slack</a> adalah aplikasi yang banyak digunakan sebagai sebuah wadah untuk ber-komunikasi dalam bentuk chat. Dengan banyaknya jumlah <a href="https://slack.com/apps">integrasi</a> yang bisa dilakukan oleh slack, tidak heran slack menjadi favorit. Aplikasi seperti <a href="https://web.skype.com/">Skype</a>, <a href="https://hangouts.google.com/">Google Hangouts</a> dan <a href="https://meet.google.com/">Google Meets</a> masih menjadi favorit untuk komunikasi dalam bentuk video/voice call. Tools kolaborasi yang banyak menjadi favorit diantaranya <a href="https://trello.com/">Trello</a>, <a href="https://www.atlassian.com/software/jira">Jira</a>, <a href="https://www.pivotaltracker.com/">Pivotal Tracker</a> dan <a href="https://www.meistertask.com/">Meistertask</a>.</p><h3>Profesi DevOps Engineer</h3><p>Organisasi/perusahaan yang melakukan praktek DevOps, mereka bisa mengurangi jumlah kesalahan, <em>recovery</em> lebih cepat, dan <em>deploy</em> lebih banyak. DevOps Engineer akan menjadi <strong>tim yang posisinya sangat penting</strong>, karena DevOps Engineer ini akan bekerja sangat dekat dengan Tim Engineer untuk memastikan <strong>produk berjalan dengan baik</strong>.</p><p>Selain itu, hasil dari survey <a href="https://insights.stackoverflow.com/survey/2017#salary">StackOverflow pada tahun 2017</a>, menunjukan posisi DevOps Specialist menempati ranking pertama dalam rata-rata gaji, yaitu <strong>$60,215</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*134Kl_4pconmkCEmzlT20Q.png" /></figure><p>Juga masih dari survey <a href="https://insights.stackoverflow.com/survey/2017#work-salaries-by-geography">StackOverflow pada tahun 2017</a>, posisi DevOps di United States memiliki rata-rata gaji <strong>$100,000</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*RhrThzIR9FZOVq95ggGxQQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15c9e80adab2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/serverless-indonesia/berkenalan-dengan-devops-15c9e80adab2">Berkenalan dengan DevOps</a> was originally published in <a href="https://medium.com/serverless-indonesia">Serverless Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Arsitektur Serverless? Siapa yang sudah sukses?]]></title>
            <link>https://medium.com/serverless-indonesia/arsitektur-serverless-siapa-yang-sudah-sukses-85fd7eb6076e?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/85fd7eb6076e</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Thu, 29 Jun 2017 08:44:25 GMT</pubDate>
            <atom:updated>2017-06-29T08:44:25.373Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Pastikan anda sudah memahami </em><a href="https://blog.serverless.id/post/arsitektur-serverless-kenapa-harus-beralih/"><em>kenapa harus beralih</em></a><em> dan </em><a href="https://blog.serverless.id/post/arsitektur-serverless-siapa-yang-harus-beralih"><em>siapa yang harus beralih</em></a><em>.</em></blockquote><h3>Membangun Website Sensus Australia yang lebih baik dengan Arsitektur Serverless</h3><p>Pada tanggal 9 Agustus tahun 2016, penduduk Australia melakukan sensus. 16 juta penduduk Australia — 65% dari total populasi, atau lebih dari 6 juta rumah tangga — diharapkan untuk menyelesaikan sensus penduduk dengan cara daring. Proyek ini menggunakan dana lebih dari <strong>$9,000,000</strong> dan menggunakan dana lebih dari <strong>$400,000</strong> untuk melakukan “Load Testing”. Klaim dari pemerintah, server telah di load hingga 150% dari kapasitas, namun dari sekian banyak uang tersebut, website “crash”, dan banyak orang komplain seperti diberitakan oleh <a href="http://www.news.com.au/national/frustration-as-abs-census-website-unable-to-be-reached/news-story/298363b96a4feab132698075cec306b2">news.com.au</a> dan <a href="https://www.lifehacker.com.au/2016/08/the-australian-census-website-is-down/">lifehacker.com.au</a>.</p><p>Austin Wilshire dan Bernd Hartzer menerima perhatian dunia ketika mereka membuat sebuah website alternatif “Australian Bureau of Statistics” dengan biaya kurang dari <strong>$500</strong>. Mereka mengerjakan website ini dalam hackaton dan selesai <strong>kurang dari 24 Jam</strong>. Yang menarik, Austin Wilshire belum pernah menyentuh AWS sama sekali, apalagi teknologi serverless.</p><p>Lalu darimana mereka bisa mengklaim lebih baik dari website aslinya? Mereka melakukannya dengan opensource load testing tool <a href="https://goad.io/">Goad</a>. Goad menggunakan AWS Lambda untuk melakukan Load Testing, dan sanggup mencapai angka 100,000 request/detik.</p><p>Bahkan Werner Vogel, CTO Amazon men-tweet tentang kesuksesan 2 anak muda ini.</p><h3>Werner Vogels on Twitter</h3><p>cool: QUT Students built an Australian Census system for $500 that does not #CensusFail https://t.co/ieaxP4Ix1s #AWS</p><blockquote><em>Kunjungi artikel lengkapnya </em><a href="https://serverless.com/blog/building-a-better-australian-census-site/"><em>“Building A Better Australian Census Website with Serverless Architecture”</em></a></blockquote><h3>30,000 “Pageview” dengan biaya $0.21 dan serverless</h3><p><a href="https://fantasymovieleague.com/">Fantasy Movie League</a> adalah game fantasy untuk orang-orang yang tidak suka dengan olahraga. Perkiraan pengguna yang sudah registrasi pada saat artikel aslinya dibuat yaitu 24,000.</p><p>Aplikasi yang dibuat adalah <a href="http://analyzer.fmlnerd.com/lineups/">“Lineup Calculator”</a>. Dengan biaya yang murah, dan tidak perlu manage server, arsitektur serverless adalah cerita yang sangat menarik.</p><blockquote><em>Kunjungi artikel lengkapnya </em><a href="https://fmlnerd.com/2016/08/16/30k-page-views-for-0-21-a-serverless-story/"><em>“30K Page Views for $0.21: A Serverless Story”</em></a></blockquote><h3>Perjalanan OpsGenie menuju arsitektur serverless</h3><p><a href="https://www.opsgenie.com/">OpsGenie</a> adalah sebuah start-up yang dimulai pada tahun 2011 dengan tiga orang senior full stak developers yang sangat berpengalaman dalam membangun aplikasi Java Enterprise, produk on-premise yang memiliki spesialisasi dalam integrasi, data enrichment dan manajemen dari generasi pertama aplikasi monitoring infrastruktur. Mereka melihat ada kesempatan, dan memutuskan menggunakan pengalaman mereka untuk membangun layanan internet untuk “Alert/Incident Management”.</p><p>OpsGenie mulai memindahkan banyak service mereka dari Java ke serverless, dan membuat artikel berseri mengenai perjalanan mereka dari <a href="https://engineering.opsgenie.com/migrating-a-monolithic-java-application-to-aws-serverless-architecture-1180ca87b0b0"><strong>Aplikasi Monolitik Java ke Arsitektur Serverless AWS</strong></a>.</p><blockquote><em>Kunjungi artikel lengkapnya </em><a href="https://read.acloud.guru/opsgenie-journey-to-serverless-architecture-785540261ec3"><em>“OpsGenie is on a journey to reap the benefits of serverless architecture”</em></a></blockquote><h3>Cloudsploit tidak lagi memanage EC2 Server</h3><p><a href="https://cloudsploit.com/">CloudSploit</a> adalah “Security and Configuration Scanner” yang bisa mendeteksi ratusan ancaman dalam akun AWS.</p><p>Menariknya, Cloudsploit <strong>belum menggunakan framework</strong>, karena mereka ingin aplikasi web yang bisa di deploy dimanapun, bahkan jika di masa depan mereka harus meninggalkan AWS API Gateway dan AWS Lambda. Maka mereka pun <strong>membuat sebuah nodejs framework sendiri</strong>. Alasan lainnya adalah pada saat mereka memulai proses menuju arsitektur serverless, belum ada framework yang stable, serverless framework saat itu belum memasuki versi 1.0.</p><p>Lalu ke bagian yang lebih penting, yaitu Billing. Sebelumnya mereka full menggunakan server EC2, baik digunakan atau tidak, <strong>mereka membayar untuk 24 Jam / 7 Hari Seminggu</strong> dan saat ini mereka membayar ketika service mereka digunakan. Mereka percaya arsitektur serverless ini sangat membantu khususnya pada proses scaling yang <strong>“tidak terbatas”</strong>. Dan siapa yang merasa nyaman ketika pemberitahuan <strong>“the server is out of disk space”</strong> pada pukul 3AM tidak lagi muncul?</p><blockquote><em>Kunjungi artikel lengkapnya </em><a href="https://blog.cloudsploit.com/we-made-the-whole-company-serverless-5a91c27cd8c4"><em>We Made the Whole Company “Serverless”</em></a></blockquote><h3>Postflight mengurangi biaya hosting hingga $10,000 per bulan</h3><p>Adam Pash adalah Lead Engineer di Postflight, dan mendapatkan tugas khusus, yaitu “Menulis ulang Readability Parser API”. Readability API versi gratis membuat perusahaan harus membayar sekitar <strong>$10,000</strong> tiap bulan. Javascript adalah pilihan bahasa pemrograman yang akan digunakan, karena banyak Engineer di Postflight yang pernah memiliki pengalaman web.</p><p>Pilihan mereka adalah menggunakan <a href="http://serverless.com/">Serverless Framework</a>. Pada bulan Oktober kemarin, Postflight merilis “Mercury Web Parser”, dan hasilnya mencengangkan. <strong>Biaya operasional mereka seketika turun</strong>, dan hari ini, Mercury Web Parser membutuhkan biaya sekitar <strong>$400</strong> untuk beroperasi. Dengan detail biaya :</p><h4>API Gateway:</h4><ul><li>Biaya Request: $137 ($3.50 per 1 juta API calls)</li><li>Cache: $14</li><li>Data Transfer Out: $42 ($0.09/GB)</li></ul><h4>Lambda</h4><ul><li>Biaya Request: $3</li><li>Biaya Compute: $174 (rata-rata 2.37 detik/invocation dengan alokasi memory sebesar 256MB)</li></ul><h4>Total Biaya: $370</h4><blockquote><em>Kunjungi artikel lengkapnya </em><a href="https://trackchanges.postlight.com/serving-39-million-requests-for-370-month-or-how-we-reduced-our-hosting-costs-by-two-orders-of-edc30a9a88cd"><em>Serving 39 Million Requests for $370/Month, or: How We Reduced Our Hosting Costs by Two Orders of Magnitude</em></a></blockquote><h3>Kesimpulan</h3><blockquote><em>Dengan 5 kisah sukses tersebut, sudah siapkah anda dipermudah oleh arsitektur serverless?</em></blockquote><p>Artikel ini adalah publikasi ulang dari <a href="https://blog.serverless.id/post/arsitektur-serverless-siapa-yang-sudah-sukses/">https://blog.serverless.id/post/arsitektur-serverless-siapa-yang-sudah-sukses/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85fd7eb6076e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/serverless-indonesia/arsitektur-serverless-siapa-yang-sudah-sukses-85fd7eb6076e">Arsitektur Serverless? Siapa yang sudah sukses?</a> was originally published in <a href="https://medium.com/serverless-indonesia">Serverless Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Arsitektur Serverless? Siapa yang harus beralih?]]></title>
            <link>https://medium.com/serverless-indonesia/arsitektur-serverless-siapa-yang-harus-beralih-a53948f56e0f?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a53948f56e0f</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Thu, 29 Jun 2017 08:31:31 GMT</pubDate>
            <atom:updated>2017-06-29T08:38:45.175Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Pastikan anda sudah memahami apa itu serverless, dan </em><a href="https://blog.serverless.id/post/arsitektur-serverless-kenapa-harus-beralih/"><em>kenapa harus beralih</em></a><em>.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/0*S3GUUtQSW5ai8fJY.jpg" /></figure><h3>Startup yang mengejar MVP</h3><p>Startup secara mudah bisa Kita artikan perusahaan yang sedang merintis, dengan jumlah pegawai yang masih sedikit. Sedikit ini relatif, ada yang bilang satu orang bisa membangun startup seperti <a href="https://www.indiehackers.com/businesses/apex-ping">Apex Ping</a> (FYI, TJ Holowaychuk juga membuat salah satu framework serverless yaitu <a href="http://apex.run/">Apex</a>), atau dua orang seperti <a href="https://www.indiehackers.com/blog/acquired-by-stripe">Indie Hackers</a>, atau yang cukup banyak dengan 15 orang seperti <a href="https://www.indiehackers.com/businesses/less-annoying-crm">Less anoyying CRM</a>.</p><p>MVP yang dimaksud adalah <a href="https://en.wikipedia.org/wiki/Minimum_viable_product">Minimun Viable Product</a>, bukan MVP dalam Ragnarok Online, bukan juga MVP seekor monyet dalam film Most Valuable Primate. Produk yang akan dibuat, sudah bisa memuaskan pengguna, memudahkan hidup pengguna dan bisa menyelesaikan masalah dengan adanya produk ini.</p><p>Arsitektur serverless akan sangat memangkas waktu yang dibutuhkan untuk membuat sebuah produk, jika produk yang ditawarkan adalah sebuah aplikasi berbasis web, dengan kombinasi :</p><ul><li><a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html">S3 untuk hosting</a></li><li><a href="http://docs.aws.amazon.com/apigateway/latest/developerguide/create-api-using-restapi.html">AWS Api gateway sebagai REST API</a>, <a href="http://docs.aws.amazon.com/lambda/latest/dg/with-on-demand-https-example.html">AWS Lambda sebagai handler Api Gateway</a></li><li><a href="http://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html">AWS DynamoDB sebagai data layer</a></li><li><a href="http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html">AWS Route 53 untuk manajemen DNS</a></li></ul><p>Bisnis Anda sudah bisa dimulai, <strong>Anda bisa fokus pada produk dan pengguna, bukan infrastruktur</strong>.</p><blockquote><em>Mahmoud Matouk memberikan tutorial gratis di Udemy dengan judul “</em><a href="https://www.udemy.com/serverless-architecture-on-aws/"><strong><em>Serverless Architecture on AWS</em></strong></a><em>” yang membahas 4 stack tersebut.</em></blockquote><h3>Startup yang kekurangan / tidak ada tim Devops</h3><p><a href="https://en.wikipedia.org/wiki/DevOps">Devops</a> secara mudah bisa Kita artikan seseorang yang bertanggung jawab atas berjalannya sebuah produk digital. Jika produk yang ditawarkan adalah aplikasi berbasis web, minimal akan berurusan dengan :</p><ul><li><strong>Web Server</strong> seperti Nginx, Apache Web Server, Lighttpd</li><li><strong>Web Framework</strong> seperti Laravel, Django, Ruby on Rails, Sails / Meteor</li><li><strong>Platform Application</strong> seperti PHP-FPM, gunicorn, PM2</li><li><strong>RDBMS</strong> seperti MySQL, PostgreSQL</li><li><strong>NoSQL</strong> seperti Redis, MongoDB, Elasticsearch, Cassandra</li><li>Permintaan server atau instance seperti pada provider AWS, GCP, Azure</li></ul><p>Lebih dalam pada peran pentingnya tim Devops adalah <strong>mereka harus memastikan produk yang dikembangkan tidak ada masalah</strong>, khususnya saat terjadi traffic yang tinggi, jika terjadi service down seperti “I/O server aplikasi tinggi” sehingga tidak bisa diakses, sudah pasti tim Devops yang akan berada di garis depan. Lalu ketika weekend, dan terjadi “Server Database Down”, sudah pasti tim Devops yang menangani.</p><p>Arsitektur serverless akan memudahkan tim Devops. Dengan tidak perlunya memikirkan Web Server yang tidak pernah down (AWS Api Gateway + AWS Lambda sebagai layer Web Server), tim Devops bisa <strong>fokus pada pekerjaan lain</strong>.</p><p>Kita ambil contoh produk NoSQL <a href="http://www.aerospike.com/">Aerospike</a>. Produk ini tidak tersedia sebagai layanan “Managed” di AWS, dan kita sendiri yang harus melakukan manajemen. Tim Devops bisa fokus pada proses Scaling Up &amp; Scaling Horizontal dengan menambahkan node yang akan membuat aerospike melakukan “<a href="http://www.aerospike.com/docs/architecture/data-distribution.html">Automatic Rebalance</a>”.</p><p>Kita ambil contoh lain, yaitu Duolingo menggunakan DynamoDB dan menyimpan <a href="https://www.youtube.com/watch?v=fhnAvn2YxZA">31 Miliar item</a>, Duolingo hanya membutuhkan 2 orang tim Devops untuk menghandle <a href="https://www.techinasia.com/how-duolingo-got-110-million-users">110 Juta Pengguna</a> (2 orang Devops disebutkan dalam video).</p><h3>Perusahaan yang memiliki server fisik</h3><p>Banyak perusahaan IT yang memulai bisnis dengan memiliki server, baik itu di data center maupun membuat ruang server khusus di perusahaan mereka. Ada beberapa hal yang perlu diperhatikan jika memiliki server fisik, diantaranya :</p><ul><li>Ketersediaan Part seperti HDD, Processor, Ram, dan Power Supply</li><li>Akses Internet inbound / outbound</li><li>Suhu Ruangan</li><li>Kestabilan Listrik</li><li>Tim Server yang selalu standby di data center</li><li>Biaya yang tidak murah</li></ul><p>Perusahaan banyak beranggapan dengan memiliki / mengelola server sendiri, bisnis akan tetap berjalan. Betul, cara ini dilakukan oleh Stackoverflow, <a href="https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/">Arsitektur</a> dan <a href="https://nickcraver.com/blog/2016/03/29/stack-overflow-the-hardware-2016-edition/">Hardware</a>.</p><p>Anda bisa mengikuti langkah yang diambil stackoverflow jika :</p><ul><li>Memiliki Tim Devops yang sangat berpengalaman</li><li>Sudah <strong>menguasai 100% produk</strong> yang sedang dibuat</li><li>Sudah mengerti ketika harus melakukan scaling</li><li>Sudah mengerti ketika terjadi masalah pada server</li><li>Sudah melakukan tuning server sampai maksimal</li></ul><h3>Lalu, mengapa masih menawarkan arsitektur serverless?</h3><h4><strong>Processor dari tahun ke tahun semakin baik dan semakin murah</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*jExPhvMQbSRgZWe0.png" /></figure><p>Dengan melihat tabel diatas yang bersumber dari <a href="http://www.techspot.com/article/1039-ten-years-intel-cpu-compared/">techspot</a>, processor dari tahun ke tahun itu semakin baik dan semakin murah. Dengan memiliki $316 pada tahun 2006 kita bisa memiliki high end processor. Namun dengan harga yang tidak berbeda jauh, dengan memiliki $339 pada tahun 2013 kita bisa memiliki high end processor terbaik di tahun tersebut.</p><p>Kita bandingkan dengan processor Server, antara <a href="https://ark.intel.com/compare/91770,64596">Intel Xeon E5–2690 Generasi Pertama dengan Generasi Keempat</a>. Hasil benchmark menunjukkan, <a href="https://ark.intel.com/products/64596/Intel-Xeon-Processor-E5-2690-20M-Cache-2_90-GHz-8_00-GTs-Intel-QPI">generasi pertama</a> mendapatkan score <strong>503|681</strong>, dan <a href="https://ark.intel.com/products/91770/Intel-Xeon-Processor-E5-2690-v4-35M-Cache-2_60-GHz">generasi keempat</a> mendapatkan score <strong>943|1300</strong>.</p><h4><strong>Effort yang tidak sedikit jika ingin melakukan upgrade server</strong></h4><p>Merujuk pada point nomor 1, dengan melihat processor yang semakin murah tiap tahun, pastinya perusahaan juga ingin melakukan upgrade pada server yang mereka miliki. Dengan biaya yang semakin murah, akan didapatkan “compute power” yang lebih baik. Namun proses upgrade server itu tidak mudah, ada beberapa hal yang harus anda pertimbangkan, diantaranya :</p><ul><li>Anda harus memikirkan tentang socket processor, beruntung jika socket masih sama, jika socketnya berbeda?</li><li>Processor lama yang Anda miliki, mau dipakai untuk apa?</li><li>Jika Processor lama yang Anda miliki ingin dijual saja, berapa banyak jatuhnya?</li><li>Jika Socket processor cocok, apakah dengan motherboard yang lama, sistem bisa berjalan dengan maksimal?</li><li>Jika Socket processor tidak cocok, berikut juga komponen lainnya tidak cocok, Anda lebih baik <strong>membeli Server baru</strong></li><li>Server lama yang Anda miliki, mau dipakai untuk apa?</li><li>Jika Server lama yang Anda miliki ingin dijual saja, berapa banyak jatuhnya?</li><li>Jika server lama yang Anda miliki tetap ingin digunakan, Anda harus <strong>membayar biaya maintenance ke Data Center</strong></li></ul><h4><strong>Tidak bisa melakukan proses Autoscaling</strong></h4><p>Saya rasa point ini cukup jelas. Kalaupun ada sebuah kasus, ambil contoh ecommerce yang melakukan flash sale, effort untuk melewati event ini sangat besar sekali. Sudah adakah provider di Indonesia yang menyediakan sewa server fisik per jam?</p><blockquote><em>Anda tidak perlu memikirkan hal-hal diatas dengan arsitektur serverless.</em></blockquote><h3>Kesimpulan</h3><blockquote><em>Arsitektur Serverless akan mempermudah hidup Kita dan membuat Kita lebih efisien terhadap waktu dengan tidak perlu memikirkan tentang infrastruktur atau server.</em></blockquote><p>Artikel ini adalah publikasi ulang dari <a href="https://blog.serverless.id/post/arsitektur-serverless-siapa-yang-harus-beralih/">https://blog.serverless.id/post/arsitektur-serverless-siapa-yang-harus-beralih/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a53948f56e0f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/serverless-indonesia/arsitektur-serverless-siapa-yang-harus-beralih-a53948f56e0f">Arsitektur Serverless? Siapa yang harus beralih?</a> was originally published in <a href="https://medium.com/serverless-indonesia">Serverless Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Arsitektur Serverless? Kenapa harus beralih?]]></title>
            <link>https://medium.com/serverless-indonesia/arsitektur-serverless-kenapa-harus-beralih-252701efa268?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/252701efa268</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Thu, 29 Jun 2017 08:19:26 GMT</pubDate>
            <atom:updated>2017-06-29T08:19:26.502Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Serverless? Tidak ada server? Mana mungkin.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*yFL7wLA2oivd1BUB.jpg" /></figure><p>Betul, teknologi serverless, dibelakangnya memang masih ada server, tapi hanya memiliki waktu sekian detik / mili detik, istilah ini memang sangat awam bagi para Developer.</p><p>Lalu apa sebetulnya serverless itu sendiri? Sudah sangat banyak yang membahas mengenai serverless ini, beberapa diantaranya :</p><ul><li><a href="https://martinfowler.com/articles/serverless.html">Serverless Architectures</a> <em>(Paling jelas, disertai dengan beberapa perbandingan)</em></li><li><a href="https://medium.com/@PaulDJohnston/what-is-serverless-architecture-43b9ea4babca">What <em>is</em> Serverless Architecture?</a></li><li><a href="https://medium.com/the-node-js-collection/serverless-javascript-5d2528ac46b5">Answering All Your Questions on Serverless Javascript</a></li><li><a href="https://read.acloud.guru/whats-this-serverless-thing-anyway-b101cb72c7e6">What’s This Serverless Thing Anyway?</a></li></ul><h3>Beri saya alasan untuk beralih</h3><p>Umumnya untuk membuat sebuah sistem, mayoritas menggunakan pola seperti ini :</p><h4>Web Server + Aplikasi Server dan Database Server dalam satu server.</h4><p>Ini langkah yang sederhana, dan efisien terhadap biaya. Kita bisa memulai dari satu server untuk melihat traffic yang terjadi didalam sistem yang dibangun, selama semua muat dalam satu server, kenapa tidak?</p><p>“Provision Tools” seperti <a href="https://www.ansible.com/how-ansible-works">Ansible</a>, <a href="https://www.chef.io/automate/">Chef</a> dan <a href="https://puppet.com/product/capabilities/automated-provisioning">Puppet</a> dapat menangani deployment seperti ini. Cukup ketik satu baris perintah di command line kita, tunggu, dan sistem siap digunakan. Limitasi pada sistem seperti ini adalah ketika traffic meningkat, imbasnya yaitu pada CPU, Memory dan Storage yang sudah terkunci pada server kita.</p><p>Jika server yang digunakan adalah server fisik, limitasinya pada jumlah slot CPU, slot Memory, dan slot Storage. Jika server yang digunakan berbasis cloud, cukup dengan sekian klik, server kita semakin kuat untuk menerima traffic. Harus diingat, deployment seperti ini pasti ada downtime ketika melakukan upgrade. Lalu muncul pertanyaan</p><blockquote>Kalau trafficnya cuma rame waktu weekend, gimana ya?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/0*d3KUI1ihkbQn8hV0.png" /></figure><h4>Web Server + Aplikasi Server dan Database Server dalam server yang berbeda dan banyak.</h4><p>Ini langkah yang sangat baik. Ketika traffic meningkat, dan yang membutuhkan penanganan lebih lanjut hanya pada sisi Web Server, Web Server bisa kita upgrade atau tambah node, begitu juga jika server Database membutuhkan penanganan, bisa dilakukan scaling dengan menambahkan node untuk replikasi / sharding.</p><p>Deployment jenis ini juga bisa ditangani oleh “Provision Tools” sama seperti diatas. Cukup ketik satu baris perintah di command line kita, tungggu, dan sistem siap digunakan. Atau jika menggunakan service cloud seperti <a href="https://aws.amazon.com/elasticbeanstalk/"><strong>AWS Elastic Beanstalk</strong></a> atau <a href="https://aws.amazon.com/opsworks/"><strong>AWS Opswork</strong></a>, kita sudah sangat dimudahkan.</p><p>Apakah dengan deployment seperti ini bisa menjawab pertanyaan diatas? Saya pikir ini bisa. Lalu mengapa masih menawarkan serverless?</p><p>Kita tengok sebuah cerita di suatu tim. Pada suatu waktu tim melihat halaman monitoring, tampak jelas grafik yang sangat menanjak, dan seketika turun dengan curam. Dan di bagian report <strong>http request 500</strong>, naik sangat tajam. Lalu muncul pertanyaan</p><blockquote>Server Aplikasi kita ngga kuat, berapa banyak pengunjung yang dibuat kecewa dengan error ini?</blockquote><p>Lalu dilakukan meeting dan muncullah sebuah formula, “Kita tambah 1 instance, ketika rata — rata CPU sudah <strong>mencapai 85% selama 5 menit</strong>. Dan kurangi 1 instance ketika rata — rata CPU <strong>sudah 25% selama 5 menit</strong>”. Dengan cara ini, kita bisa melakukan antisipasi awal ketika traffic sudah mulai tinggi. Lalu muncul lagi pertanyaan</p><blockquote>Berapa waktu yang dibutuhkan untuk menambah instance baru?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/0*9cjlgnqKt9E2Rt0f.png" /></figure><h3>Selamat Datang di Dunia Serverless</h3><p>Dengan merujuk pertanyaan sebelumnya, maka pertanyaan pada point nomor 1 sudah diselesaikan oleh serverless. Dengan otomatis serverless akan melakukan scaling sejumlah traffic. Berita baiknya, jika menggunakan framework zappa, Kita bisa memanfaatkan fungsi <a href="https://github.com/Miserlou/Zappa#asynchronous-task-execution"><strong>async</strong></a>. Mengenai async &amp; zappa akan dibahas dalam post yang berbeda.</p><p>Lalu mengenai perbandingan antara arsitektur serverless dan arsitektur berbasis server seperti elasticbeanstalk, perbedaan yang sangat jelas yaitu saat proses <a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/GettingStartedTutorial.html">autoscaling</a> terjadi. Provisioning instance baru, itu <a href="https://stackoverflow.com/questions/14115502/elastic-beanstalk-auto-scaling-how-long-to-bring-up-a-new-instance"><strong>dibutuhkan proses</strong></a>, seperti install dependency (contoh : Nginx + PHP-FPM), register ke Elastic Load Balancer, Health Check, sehingga instance bisa menerima traffic.</p><p>Pada arsitektur serverless, tidak ada lagi proses install dependency dan lainnya seperti pada elastic beanstalk. Karena kode kita sudah dibungkus menjadi sebuah fungsi, proses menambah instance dalam arsitektur serverless tersebut berjalan sangat cepat dan otomatis.</p><p>Dalam dunia serverless, Kita bisa fokus pada bisnis dan kepuasan user. Tidak perlu banyak effort dalam scaling, semua sudah di handle oleh provider.</p><p>Dalam dunia serverless, Kita berbicara mengenai pemanfaatan service. Service seperti <a href="https://auth0.com/">auth0</a> yang memberikan fitur identitas tanpa harus memanage server. Login menggunakan akun social dipermudah dengan service ini.</p><p>Jika kita ingin membuat aplikasi data streaming, kita bisa memanfaatkan service seperti <a href="https://aws.amazon.com/kinesis/firehose/">aws kinesis firehose</a> untuk mengirim data ke <a href="https://aws.amazon.com/s3/">S3</a>, atau <a href="https://aws.amazon.com/kinesis/streams/">aws kinesis stream</a> untuk mengirim data ke RDBMS seperti <a href="https://aws.amazon.com/rds/">AWS RDS</a> / <a href="https://aws.amazon.com/rds/aurora/">Aurora</a> dan NoSQL seperti <a href="https://aws.amazon.com/dynamodb/">AWS DynamoDB</a>, <a href="https://aws.amazon.com/elasticache/">Redis</a>. Mengenai teknologi streaming akan dibahas dalam post yang berbeda.</p><h3>Kesimpulan</h3><blockquote><em>Serverless adalah tentang tidak diperlukannya manage server, semua tentang pemanfaatan service, Kita tidak lagi dipusingkan oleh limitasi server, sehingga proses bisnis berjalan dengan baik berapapun traffic yang datang.</em></blockquote><p>Artikel ini adalah publikasi ulang dari <a href="https://blog.serverless.id/post/arsitektur-serverless-kenapa-harus-beralih/">https://blog.serverless.id/post/arsitektur-serverless-kenapa-harus-beralih/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=252701efa268" width="1" height="1" alt=""><hr><p><a href="https://medium.com/serverless-indonesia/arsitektur-serverless-kenapa-harus-beralih-252701efa268">Arsitektur Serverless? Kenapa harus beralih?</a> was originally published in <a href="https://medium.com/serverless-indonesia">Serverless Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Scaling AWS Kinesis Firehose]]></title>
            <link>https://medium.com/@clasense4/how-to-scaling-aws-kinesis-firehose-4ef4bea590e0?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ef4bea590e0</guid>
            <category><![CDATA[amazon-s3]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kinesis-firehose]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Fri, 17 Mar 2017 07:26:20 GMT</pubDate>
            <atom:updated>2017-03-17T12:58:20.313Z</atom:updated>
            <content:encoded><![CDATA[<p>I explore how to scale aws kinesis firehose.</p><h3>Background</h3><p>I was given task to create unlimited log pipeline that can scale easily. So the plan is using aws kinesis firehose and S3 as the destination.</p><p>Here is overview what we are going to built.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MxmW4pjHq5du10sk.png" /></figure><p>The producer, can be any application. But remember, kinesis firehose is not available yet in every region. So, if your application is in Asia, there will be latency to US / EU around 180–250ms.</p><p>One big question in my mind is, “How can I scale this firehose after see the limitations?”. Technically, 5000 records/second is not a small number. But right now, data can increasing without notice to us first. Especially when we want to save log of application. So, my choice is using S3 to save the log. I skipped on using object lifecycle, you can read more about object lifecycle <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html">aws version</a> and <a href="https://www.terraform.io/docs/providers/aws/r/s3_bucket.html">terraform version</a>.</p><h3>Aws CLI Up &amp; Running</h3><pre>$ aws configure<br>AWS Access Key ID [<strong>********************</strong>]:<br>AWS Secret Access Key [<strong>********************</strong>]:<br>Default region name [us-east-1]: us-west-2<br>Default output format [None]:</pre><h3>Terraform Up &amp; Running</h3><p>I use mac os version, so to install it just type :</p><pre>brew install terraform</pre><p>And my terraform version is terraform-0.8.8.</p><p>For another OS, you can refer <a href="https://www.terraform.io/downloads.html">here</a>.</p><p>This is list of variables that we want to use on our terraform project. Here I name it, terraform.tfvars. More information on this, <a href="https://www.terraform.io/docs/configuration/variables.html">https://www.terraform.io/docs/configuration/variables.html</a>.</p><pre><em>#--------------------------------------------------------------</em><br><em># General</em><br><em>#--------------------------------------------------------------</em><br>region            = &quot;us-west-2&quot;<br>firehose_count    = 2<br>log_group_name    = &quot;scaling-firehose&quot;<br>log_stream_name   = &quot;scaling-firehose&quot;</pre><p>This is list of resource that we want to use on our terraform project. We plan to add 1 S3 buckets, 1 new rule and 2 kinesis firehose delivey stream with same S3 destination. Here I name it, scaling_firehose.tf. More information on this, <a href="https://www.terraform.io/docs/configuration/resources.html">https://www.terraform.io/docs/configuration/resources.html</a></p><pre>variable &quot;region&quot;            { }<br>variable &quot;firehose_count&quot;    { }<br>variable &quot;log_group_name&quot;    { }<br>variable &quot;log_stream_name&quot;   { }</pre><pre>provider &quot;aws&quot; {<br>  region = &quot;${var.region}&quot;<br>}</pre><pre>resource &quot;aws_s3_bucket&quot; &quot;bucket&quot; {<br>    bucket = &quot;scaling-firehose&quot;<br>}</pre><pre>resource &quot;aws_iam_role&quot; &quot;firehose_role&quot; {<br>   name = &quot;firehose_example_role&quot;<br>   assume_role_policy = &lt;&lt;EOF<br>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Service&quot;: &quot;firehose.amazonaws.com&quot;<br>      },<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Sid&quot;: &quot;&quot;<br>    }<br>  ]<br>}<br>EOF<br>}</pre><pre>resource &quot;aws_iam_role_policy&quot; &quot;firehose_role_policy&quot; {<br>   name = &quot;firehose_example_role_policy&quot;<br>   role = &quot;${aws_iam_role.firehose_role.id}&quot;<br>   policy = &lt;&lt;EOF<br>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Sid&quot;: &quot;&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;s3:AbortMultipartUpload&quot;,<br>        &quot;s3:GetBucketLocation&quot;,<br>        &quot;s3:GetObject&quot;,<br>        &quot;s3:ListBucket&quot;,<br>        &quot;s3:ListBucketMultipartUploads&quot;,<br>        &quot;s3:PutObject&quot;<br>      ],<br>      &quot;Resource&quot;: [<br>        &quot;arn:aws:s3:::scaling-firehose&quot;,<br>        &quot;arn:aws:s3:::scaling-firehose/*&quot;,<br>        &quot;arn:aws:s3:::%FIREHOSE_BUCKET_NAME%&quot;,<br>        &quot;arn:aws:s3:::%FIREHOSE_BUCKET_NAME%/*&quot;<br>      ]<br>    }<br>  ]<br>}<br>EOF<br>}</pre><pre>resource &quot;aws_cloudwatch_log_group&quot; &quot;log_group&quot; {<br>  name = &quot;${var.log_group_name}&quot;<br>}</pre><pre>resource &quot;aws_cloudwatch_log_stream&quot; &quot;log_stream&quot; {<br>  name           = &quot;${var.log_stream_name}&quot;<br>  log_group_name = &quot;${aws_cloudwatch_log_group.log_group.name}&quot;<br>}</pre><pre>resource &quot;aws_kinesis_firehose_delivery_stream&quot; &quot;firehose_stream&quot; {<br>  count = &quot;${var.firehose_count}&quot;<br>  name = &quot;scaling-firehose-${count.index + 1}&quot;<br>  destination = &quot;s3&quot;<br>  s3_configuration {<br>    role_arn = &quot;${aws_iam_role.firehose_role.arn}&quot;<br>    bucket_arn = &quot;${aws_s3_bucket.bucket.arn}&quot;<br>    buffer_size = 5<br>    buffer_interval = 60</pre><pre>    cloudwatch_logging_options {<br>      enabled = &quot;true&quot;<br>      log_group_name = &quot;${aws_cloudwatch_log_group.log_group.name}&quot;<br>      log_stream_name = &quot;${aws_cloudwatch_log_stream.log_stream.name}&quot;<br>    }<br>  }<br>}</pre><p>And after we are done with the configuration, we can try terraform plan.</p><pre>$ terraform plan<br>...</pre><pre>+ aws_cloudwatch_log_group.log_group<br>...</pre><pre>+ aws_cloudwatch_log_stream.log_stream<br>...</pre><pre>+ aws_iam_role.firehose_role<br>...</pre><pre>+ aws_iam_role_policy.firehose_role_policy<br>...</pre><pre>+ aws_kinesis_firehose_delivery_stream.firehose_stream.0<br>...</pre><pre>+ aws_kinesis_firehose_delivery_stream.firehose_stream.1<br>...</pre><pre>+ aws_s3_bucket.bucket<br>...<br></pre><pre>Plan: 7 to add, 0 to change, 0 to destroy.</pre><p>Then if we are confident with the plan, we can try to apply the plan with command terraform apply.</p><pre>$ terraform apply<br>aws_cloudwatch_log_group.log_group: Creating...<br>...</pre><pre>aws_iam_role.firehose_role: Creating...<br>...</pre><pre>aws_s3_bucket.bucket: Creating...<br>...</pre><pre>aws_cloudwatch_log_group.log_group: Creation complete<br>aws_cloudwatch_log_stream.log_stream: Creating...<br>...<br>aws_iam_role.firehose_role: Creation complete<br>aws_iam_role_policy.firehose_role_policy: Creating...<br>...<br>aws_cloudwatch_log_stream.log_stream: Creation complete<br>aws_iam_role_policy.firehose_role_policy: Creation complete<br>aws_s3_bucket.bucket: Still creating... (10s elapsed)<br>aws_s3_bucket.bucket: Creation complete<br>aws_kinesis_firehose_delivery_stream.firehose_stream.0: Creating...<br>aws_kinesis_firehose_delivery_stream.firehose_stream.1: Creating...<br>aws_kinesis_firehose_delivery_stream.firehose_stream.0: Still creating... (3m0s elapsed)<br>aws_kinesis_firehose_delivery_stream.firehose_stream.0: Creation complete</pre><pre>Apply complete! Resources: 7 added, 0 changed, 0 destroyed.</pre><pre>The state of your infrastructure has been saved to the path<br>below. This state is required to modify and destroy your<br>infrastructure, so keep it safe. To inspect the complete state<br>use the `terraform show` command.</pre><pre>State path: terraform.tfstate</pre><p>Go to aws console, and this is what we get.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E68Qy_bCL2LzuB17.png" /></figure><h3>Test using AWS CLI</h3><pre>$ aws firehose put-record --delivery-stream-name scaling-firehose-1 --record &#39;{&quot;Data&quot;:&quot;{\&quot;foo\&quot;:\&quot;bar\&quot;}&quot;}&#39;<br>{<br>    &quot;RecordId&quot;: &quot;PoKUUQ4805aidLc3CUH...&quot;<br>}<br> $ aws firehose put-record --delivery-stream-name scaling-firehose-2 --record &#39;{&quot;Data&quot;:&quot;{\&quot;foo2\&quot;:\&quot;bar2\&quot;}&quot;}&#39;<br>{<br>    &quot;RecordId&quot;: &quot;kjoYUT2YJt3VNcxCrSH...&quot;<br>}</pre><p>And this is what we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/0*dBIyhnUeffAKlOUZ.png" /></figure><h3>Test using AWS Python SDK</h3><p>This is the python code, we will test using 1 delivery stream first.</p><pre>import boto3</pre><pre>client = boto3.client(&#39;firehose&#39;, region_name=&#39;us-west-2&#39;)</pre><pre>message = &#39;{&quot;foo&quot;:&quot;bar&quot;}&#39; + &quot;\n&quot;<br>response = client.put_record(<br>    DeliveryStreamName=&#39;scaling-firehose-1&#39;,<br>    Record={<br>        &#39;Data&#39;: message<br>    }<br>)</pre><p>And this is what we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*2As_c2CpWiNvX0MW.png" /></figure><p>After we are confident with the code, we can try this</p><pre>import boto3<br>import random</pre><pre>client = boto3.client(&#39;firehose&#39;, region_name=&#39;us-west-2&#39;)</pre><pre>for x in xrange(1,100):<br>    delivery_streams = [&#39;scaling-firehose-1&#39;,&#39;scaling-firehose-2&#39;]<br>    delivery_stream = random.choice(delivery_streams)</pre><pre>    print delivery_stream</pre><pre>    message = &#39;{&quot;foo&quot;:&quot;bar&quot;}&#39; + &quot;\n&quot;<br>    response = client.put_record(<br>        DeliveryStreamName=delivery_stream,<br>        Record={<br>            &#39;Data&#39;: message<br>        }<br>    )</pre><pre>$ python firehose.py<br>scaling-firehose-2<br>scaling-firehose-2<br>scaling-firehose-2<br>scaling-firehose-1<br>scaling-firehose-1<br>scaling-firehose-1<br>...</pre><p>As you can see, we are randomizing the delivery stream, let’s checkout the S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*z13RNKZyR16QJstE.png" /></figure><h3>Destroying Infra</h3><p>You need to empty the S3 bucket first.</p><pre>aws s3 rm s3:<em>//scaling-firehose/ --recursive</em></pre><p>And after the bucket is empty, you can use terraform command.</p><pre>terraform plan -destroy<br>terraform destroy</pre><h3>Conclusion</h3><p>This is just a proof of concept, and if this is going well, when I want to add new delivery stream, what I need to do is edit terraform.tfvars and change the value of firehose_count into desired value. Type terraform plan check if it return error, and terraform apply. Then add list of current delivery stream to firehose.py. Creating delivery stream is taking time, so need to be careful about deploying your new code.</p><p>Originally posted on <a href="https://blog.fajri.my.id/post/how-to-scaling-aws-kinesis-firehose/">https://blog.fajri.my.id/post/how-to-scaling-aws-kinesis-firehose/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ef4bea590e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Install latest laravel using nginx, php-fpm on centos 7]]></title>
            <link>https://medium.com/@clasense4/install-latest-laravel-using-nginx-php-fpm-on-centos-7-8ef12c149f87?source=rss-2f6060739de2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ef12c149f87</guid>
            <category><![CDATA[php]]></category>
            <category><![CDATA[laravel]]></category>
            <category><![CDATA[nginx]]></category>
            <dc:creator><![CDATA[Fajri Abdillah]]></dc:creator>
            <pubDate>Fri, 17 Feb 2017 23:02:40 GMT</pubDate>
            <atom:updated>2017-02-17T23:02:40.611Z</atom:updated>
            <content:encoded><![CDATA[<p>Laravel 5.4 has been released. And here is step by step how to installing it on centos 7 using nginx and php-fpm.</p><h3>Requirement :</h3><ol><li><a href="https://blog.fajri.my.id/post/install-nginx-stable-mainline-on-centos7">Install Nginx</a></li><li><a href="https://blog.fajri.my.id/post/install-php71-on-centos7">Install PHP-FPM</a></li></ol><h3>1. Composer</h3><p><em>Execute on linux shell</em></p><pre>php -r &quot;copy(&#39;https://getcomposer.org/installer&#39;, &#39;composer-setup.php&#39;);&quot;<br>php -r &quot;if (hash_file(&#39;SHA384&#39;, &#39;composer-setup.php&#39;) === &#39;55d6ead61b29c7bdee5cccfb50076874187bd9f21f65d8991d46ec5cc90518f447387fb9f76ebae1fbbacf329e583e30&#39;) { echo &#39;Installer verified&#39;; } else { echo &#39;Installer corrupt&#39;; unlink(&#39;composer-setup.php&#39;); } echo PHP_EOL;&quot;<br>php composer-setup.php<br>php -r &quot;unlink(&#39;composer-setup.php&#39;);&quot;<br>mv composer.phar /usr/local/bin/composer</pre><h3>2. Write phpinfo</h3><p>Let’s add some basic info about our server, located at /var/www/blog/routes/web.php.</p><pre>&lt;?php<br>Route::get(&#39;/&#39;, function () {<br>    return view(&#39;welcome&#39;);<br>});</pre><pre>if (App::environment(&#39;local&#39;, &#39;staging&#39;)) {<br>    Route::get(&#39;/info&#39;, function() {<br>        return phpinfo();<br>    });<br>}</pre><h3>3. Install Laravel and try using builtin webserver</h3><p><em>Execute on linux shell</em></p><pre>composer global require &quot;hirak/prestissimo:^0.3&quot;<br>composer create-project <em>--prefer-dist laravel/laravel blog</em><br>mv blog /var/www/blog<br>chown -R nginx:nginx /var/www/blog<br>cd /var/www/blog<br>php artisan serve <em>--host=0.0.0.0</em></pre><blockquote><em>Go to </em><a href="http://128.199.127.234:8000/info"><em>http://128.199.127.234:8000/info</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*Z7QkX6ayAwkFwK7_aFnkFg.png" /></figure><h3>4. Nginx conf</h3><p><em>Execute on linux shell</em></p><pre>sudo echo &quot;server {<br>    listen     8000;<br>    root /var/www/blog/public;<br>    index index.php;</pre><pre>    access_log     /var/log/nginx/nginx.vhost.access.log;<br>    error_log      /var/log/nginx/nginx.vhost.error.log;</pre><pre>    location / {<br>        try_files \$uri \$uri/ /index.php?\$query_string;<br>    }</pre><pre>    location ~ \.php$ {<br>        fastcgi_pass 127.0.0.1:9000;<br>        fastcgi_index index.php;<br>        fastcgi_param  SCRIPT_FILENAME   \$document_root\$fastcgi_script_name;<br>        fastcgi_param PATH_INFO \$fastcgi_script_name;<br>        include        fastcgi_params;<br>    }<br>}&quot; &gt; /etc/nginx/conf.d/laravel-sample.conf<br>nginx -t<br>service nginx restart</pre><h3>5. Make sure our laravel app running on exact port</h3><p><em>Execute on linux shell</em></p><pre>[root@centos-512mb-sgp1-01-1483019484734-scrapy-512mb-sgp1-01 conf.d]<em># netstat -ntpl</em><br>Active Internet connections (only servers)<br>Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name<br>tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      23176/nginx: master<br>tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      23176/nginx: master<br>tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      23176/nginx: master<br>tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      2614/php-fpm: maste</pre><blockquote><em>Go to </em><a href="http://128.199.127.234:8000/info"><em>http://128.199.127.234:8000/info</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*ls-tLaFFhDgKeqcr31M65Q.png" /></figure><h3>6. Conclusion</h3><p>It’s quite simple to setup laravel 5.4 projects on nginx and php-fpm. And notice the difference on Server API when open up /info. For convenience, remove /info routes when production ready.</p><p>Originally posted on <a href="https://blog.fajri.my.id/post/install-latest-laravel-nginx-php-fpm-on-centos-7/">https://blog.fajri.my.id/post/install-latest-laravel-nginx-php-fpm-on-centos-7/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ef12c149f87" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>