<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prabhu Rajendran on Medium]]></title>
        <description><![CDATA[Stories by Prabhu Rajendran on Medium]]></description>
        <link>https://medium.com/@prabhur24?source=rss-13d9e742dd73------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*61-bg70lKRoMUBBFWI3rYg.jpeg</url>
            <title>Stories by Prabhu Rajendran on Medium</title>
            <link>https://medium.com/@prabhur24?source=rss-13d9e742dd73------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:20:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prabhur24/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ Getting started with Aerospike on Docker pointing host drive]]></title>
            <link>https://medium.com/@prabhur24/getting-started-with-aerospike-on-docker-pointing-host-drive-a0f0f58adebc?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0f0f58adebc</guid>
            <category><![CDATA[aerospike]]></category>
            <category><![CDATA[aerospike-ce]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Sat, 13 Apr 2024 01:03:09 GMT</pubDate>
            <atom:updated>2024-04-13T01:03:09.903Z</atom:updated>
            <content:encoded><![CDATA[<p>This article discusses utilising a host drive, specifically AWS EBS, for storing Aerospike data within a namespace. Since there are limited resources on setting up Aerospike using Docker, this <strong>article serves as a reference</strong>. For further understanding of Aerospike, basic Docker setups can be found on <a href="https://aerospike.com/docs/deploy_guides/docker">aerospike.com</a>.</p><p>Aerospike offers various <strong>hybrid storage options</strong> including Dynamic Random Access Memory (DRAM), Non-volatile Memory extended (NVMe) Flash or Solid State Drive (SSD), and Persistent Memory. In scenarios <strong>where data isn’t frequently accessed or when opting out of RAM storage, choosing SSD becomes a viable alternative</strong>. The decision of when and which option to choose is subjective and depends on specific requirements, making it an arguable question.</p><p><strong>docker image</strong> : aerospike/aerospike-server:4.9.0.37 (tags can be latest)</p><p>Basic step to run docker : docker run -d --name aerospike -p 3000-3002:3000-3002 aerospike/aerospike-server:4.9.0.37</p><p>By default, Aerospike <strong>operates within the RAM in the default container setup, supplemented by a file backup</strong>. Examining the Dockerfile for Aerospike, it’s evident that the image utilises a default configuration file specifying this mode within the namespace configuration.</p><p>Note: ( ${###} : this can be passed as environment variable).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5b3c258499d656a3b00cb009a4ddb8a/href">https://medium.com/media/f5b3c258499d656a3b00cb009a4ddb8a/href</a></iframe><p>Before delving into the Docker setup with a host drive, it’s essential to understand the advantages offered by Docker.</p><p><em>“Docker provides </em><strong><em>an ability to expose a host’s block devices to a running container.</em></strong><em> The </em><em>--device option can be used to map a host block device within a container.”</em></p><p>Now, its time explore setting up Docker with a host drive device!</p><p>This is most effectively accomplished on a public cloud infrastructure like AWS, as <strong>obtaining a new SSD volume and utilising it as a raw block device is straightforward</strong>. Once a new device is attached to an EC2 Linux VM (with Docker installed), we can map that volume for use inside the container.</p><p>Another advantage of setting up Aerospike is the ability to pass a custom configuration file (using --config-file) containing modified namespace settings and its must when using device configurations.</p><p>below gist is only specific example of namespace modified with device.</p><p>Note : test.conf is copy of template aerospike.conf</p><pre>docker run -d --name aerospike --device &#39;/dev/sdg:/dev/xvdg&#39; -v /opt/aerospike/data:/opt/aerospike/data -v /etc/aerospike/:/etc/aerospike/ -v /var/log/aerospike:/var/log/aerospike/ -p 3000-3002:3000-3002 aerospike/aerospike-server:4.9.0.37 --config-file /etc/aerospike/test.conf</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35ee117f3a6d4232541bf180a110725f/href">https://medium.com/media/35ee117f3a6d4232541bf180a110725f/href</a></iframe><p>Where /dev/xvdg is the new device volume, /etc/aerospike is the host location where the config file is stored. This is mapped to /etc/aerospike/ inside the container.</p><p>With that, Aerospike is now running inside a Docker container in hybrid mode.</p><p>However, Aerospike currently <strong>operates on a single node</strong>. This raises the question: what value does this hold, given that the true potential of Aerospike lies in horizontal scaling to hundreds of nodes? The term “scaling” naturally brings to mind Kubernetes, Docker Swarm, and others. Although these are optimal solutions for scaling software, they come with a steep learning curve.</p><p>Note : while to trying to write data in aerospike, couple of issues might be faced , Have a look over the aerospike forum &amp; stackoverflow.</p><p>Thanks for reading, if any suggestions, please feel free to comment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0f0f58adebc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[101 on Kafka (beginners)]]></title>
            <link>https://medium.com/everythingatonce/101-on-kafka-beginners-fdf1b36eb059?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdf1b36eb059</guid>
            <category><![CDATA[kafka-beginner]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Wed, 23 Mar 2022 14:35:26 GMT</pubDate>
            <atom:updated>2022-03-24T09:10:30.442Z</atom:updated>
            <content:encoded><![CDATA[<ol><li><strong>What is Kafka &amp; Why do need Kafka ?</strong></li></ol><p>Before diving into Kafka , lets know - how Kafka/Streaming platform come into picture:-</p><pre><strong>*</strong> Companies initially have “Source System &amp; Target System” - (kind-off <strong>Monolithic architecture</strong>).</pre><pre><strong>*</strong> Later <strong>multiple Source system (S1,S2,S3...) &amp; Multiple Target System (T1,T2,T3...)</strong> <br>     1. S1 can be user service<br>     2. S2 can be order service<br>     3. T1 can be notification service<br>     4. T2 can be order delivery service<br>     5. T3 can be order tracking service ....</pre><pre><strong>*</strong> If Source <strong>S1 has to use T1 , T3 and T1 has to use S2 and S3</strong> .. so on - which requires multiple integration and complicated </pre><pre>* Each integration comes with <strong>different protocol </strong>(HTTP,TCP,REST,JDBC,,,) and <strong>different formats</strong> (JSON,BINARY,CSV,,,) - which <strong>increase Load on connected systems.</strong></pre><pre>* [PROBLEM Solution] - Here it comes, we &quot;<strong>bring decoupling to system&quot; using KAFKA </strong>, how it applies now ? </pre><pre>* Source &amp; Target Service be independent and <strong>KAFKA will placed middle to Source &amp; Target.</strong></pre><pre>* what happens now , Source Service are responsible for Sending data (<strong>Producing</strong>) to Kafka , Target Service are responsible for Receiving Data(<strong>Consuming</strong>). </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-k13Tnfu_DJLKK3xQDI0mg.png" /></figure><pre>* Kafka is created by LinkedIn ,now it become OpenSource and maintained by Confluent, IBM, Cloudera.</pre><pre>* [<strong>Features</strong>] :- <strong>Distributed, Fault Tolerant, Resilient Architecture , Horizontal Scalability , High Performance (less than 10ms)</strong> - used by LinkedIn (prevent spam , collect user interactions) , AirBnb, Uber (pricing real time), Netflix(for recommendations), Walmart ... - will see why &amp; how it below.</pre><pre>* Kafka is highly available and resilient to node failures and supports automatic recovery (key point to choose Kafka..).</pre><h3>History of Apache Kafka</h3><p>Previously, LinkedIn was facing “<strong>issue of low latency ingestion of huge data from the website into a lambda architecture</strong>” which could be able to process real-time events. As a solution, Apache Kafka was developed in the year 2010, since none of the solutions was available to deal with this drawback, before.</p><p>However, there were technologies available for batch processing, but the deployment details of those technologies were shared with the downstream users. Hence, it comes to <strong>Real-time Processing, those technologies were not enough suitable</strong>. Then, in the year 2011 Kafka was made public.</p><p><strong>2. Use Cases of Kafka (Real Time)</strong></p><pre>     - Activity Tracking <br>     - Messaging System (data transfer from one system to another system asynchronous) - <strong>point to point , publish-subscribe messaging system</strong>  <br>     - Metrics Gathering (logs.)<br>     - Decoupling system<br>     - Stream processing <br>     - Micro Services Pub-Sub (above example S1,S2,T1..T3)<br>     - Integration with BigData Technologies(spark, flink, hadoop..) </pre><p><strong>3. Kafka Core &amp; Architecture</strong> <strong>Offers</strong></p><pre>    1. <strong>Kafka Broker</strong> - <br>          node on kafka cluster, used to persist and replicate the data. (by default data is available for a week and its configurable)</pre><pre>    2. <strong>Kafka Producer API - <br>          </strong>permits application to publish(push) stream of records to one or more topics. (<strong>sequence of message called data stream</strong>) via Producer API </pre><pre>    3. <strong>Kafka Consumer API - <br>          </strong>subscribe to one or more topics(pulls the message from topic) and process the events via Consumer API.</pre><pre>    4. <strong>Kafka Streams</strong> <strong>API</strong> - <br>          acts as stream processor, consuming one or more topics and producing output stream to one or more topics &amp; transforming input streams to output streams - Kafka Stream API gives permission to application.<br> <br>    5. <strong>Kafka Connector</strong> <strong>API</strong> - allows building and running reusable producers and consumers that connects Kafka topics to existing application.</pre><p><strong>4. Why Should we use Apache Kafka ?</strong></p><p>As we all know, there is an enormous volume of data in Big Data. And, when it comes to big data, there are two main challenges. One is to <strong>collect the large volume of data, while another one is to analyse the collected data</strong>. Hence, in order to overcome those challenges, we need a messaging system. Then Apache Kafka has proved its utility. There are numerous <strong>benefits of Apache Kafka </strong>such as:</p><ul><li>Tracking web activities by storing/sending the events for real-time processes.</li><li>Alerting and reporting the operational metrics.</li><li>Transforming data into the standard format.</li><li>Continuous processing of streaming data to the topics.</li></ul><p>5. <strong>Kafka Components :-</strong></p><pre>1. <strong>Topic :- (Once Written , can&#39;t be changed - Immutable)</strong></pre><pre>      - Particular stream of data (in relational database we call it as table)<br>      - Identified by name , can have N&#39; number of topics <br>      -  any kind of formats (JSON,CSV,..)<br>      - topics cant be queried directly , send data to producer , use consumer to read data. <br>      - can have many partitions - topic split into partitions<br>      - each partition have incremental id (ordered) called offset.</pre><pre><br>2. <strong>Kafka Producer</strong> - publish message to topics</pre><pre>3. <strong>Kafka Consumer</strong> - Subscribe message from topics </pre><pre>4. <strong>Kafka Broker</strong> - manages the storage of messages in topics (persist and replicate) , more than one broker called cluster.</pre><pre>5. <strong>Kafka Zookeeper -  </strong>offers the brokers with metadata about the process running system, <strong>health checks and helps in broker leadership election.</strong></pre><p>6. <strong>Alternatives to Kafka :-</strong></p><pre>1. <strong>Flume - <br>     </strong>-<strong> </strong>special purpose tool for specific applications<br>     - does not replicate events.<br> <br>2. <strong>RabbitMQ - </strong>offers relatively less support features<br>     - performance rate is 80 to 90K messages <strong>per second per</strong> connection (it varies which different client...) where Kafka is 1 Million message per second</pre><p>These are basics things to know about Kafka.</p><p>Refer for more detailed : <a href="https://www.conduktor.io/kafka/what-is-apache-kafka">https://www.conduktor.io/kafka/what-is-apache-kafka</a></p><p>Thanks for Reading!.</p><p>Resources :</p><ol><li><a href="https://www.cloudkarafka.com/blog/part1-kafka-for-beginners-what-is-apache-kafka.html">https://www.cloudkarafka.com/blog/part1-kafka-for-beginners-what-is-apache-kafka.html</a></li><li><a href="https://www.confluent.io/blog/kafka-fastest-messaging-system/">https://www.confluent.io/blog/kafka-fastest-messaging-system/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdf1b36eb059" width="1" height="1" alt=""><hr><p><a href="https://medium.com/everythingatonce/101-on-kafka-beginners-fdf1b36eb059">101 on Kafka (beginners)</a> was originally published in <a href="https://medium.com/everythingatonce">Everything at Once</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Elastic Block Store and What we should know about?]]></title>
            <link>https://medium.com/everythingatonce/what-is-elastic-block-store-and-what-we-should-know-about-5ab3c4199c3a?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ab3c4199c3a</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[ebs-volume]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Mon, 17 Feb 2020 18:22:49 GMT</pubDate>
            <atom:updated>2022-03-23T14:33:43.635Z</atom:updated>
            <content:encoded><![CDATA[<p>Previous Part To know about what is Amazon Storage devices . <a href="https://medium.com/everythingatonce/everything-that-the-aws-storage-devices-can-do-day-3-cd92ab4f1ac7">Click here!</a></p><p><strong>What is Block Store ? — </strong>Concept of <strong>cloud hosted data persistence </strong>(such as physical hard drive, form of network-attached-storage[NAS]).</p><ol><li>Storage in such is organized as blocks (physical record, is sequence of bytes,bits).</li><li>This Behavior are seen in Traditional disk or tape storage.(Blocks are identified by which they may be stored and retrieved).</li></ol><p>Example : Amazon Elastic Block Store (EBS) — Which offers replication of reliability or backup services.</p><p><strong>Amazon EBS : — provides block level storage volumes as a service that works in conjunction with EC2 Instance </strong>(were more number of volumes can be attached to ec2 instance in the same availability region).</p><ol><li>Unlike EC2 Instances which stores are ephemeral or lose any data when they terminated but <strong>EBS volumes maintains their state when stopped or detached from an instance</strong>.(Persist independently from the life of instance).</li></ol><p>Features of Amazon EBS :</p><ol><li>EBS volumes are created in a specific availability zone (can be attached to any instance in same availability zone) —<strong><em>“ To make available outside of availability zone — were we can create snapshot and restore the snapshot to a new volume anywhere in the region”</em></strong>.</li><li><strong>Volume Types : Offer 2 Types SSD (transactional workloads)&amp; HDD (large streaming workloads)</strong></li></ol><ul><li><strong>General Purpose SSD (gp2) </strong>— offers base performance of 3 IOPS/GiB , with ability to burst to 3000IOPS for extended period of time. These volumes are ideal for broad ranges such as test,boot and development volumes.</li><li><strong>Provisioned IOPS SSD (io1) — </strong>volume supports to 64,000 IOPS and 1000 MiB/s of throughput.This allows you to predictably scale to ten of thousand of IOPS per EC2 Instance.</li><li><strong>Throughput Optimized HDD (st1) — </strong>Provides low cost magnetic storage defines performance in terms of throughput rather than IOPS. These volumes are ideal for large ,sequential workloads.</li><li><strong>Cold HDD (sc1) — </strong>Same as Throughput Optimized HDD — Infrequent access of date this kind of HDD can be used.</li></ul><p>3. <strong>Benefits of EBS Volumes :</strong></p><ul><li><strong>Data availability</strong> — when we create an EBS volume in an availability zone,it is automatically replicated in same zone to Prevent data loss due to failure of any hardware component.</li><li><strong>Data Persistence — </strong>off-instance storage that can persist independently from the life of an instance.By default , root EBS volumes that created and attached to instance is deleted when the instance in terminated.</li><li><strong>Data Encryption — </strong>we can create EBS volumes with encryption feature to EBS volumes to meet a wide range of data-at-rest encryption requirements for regulated/audited data and applications.</li><li><strong>Snapshots — </strong>Amazon EBS provides the ability to create snapshots (backups) of any EBS volume and write a copy of the data in the volume to Amazon S3, where it is stored redundantly in multiple Availability Zones.</li><li><strong>Flexibility — </strong>EBS volumes support live configuration changes while in production. You can modify volume type, volume size, and IOPS capacity without service interruptions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ivhcj_YJX-qyMSGf8jccsA.png" /></figure><p><strong>EBS Performance : I/O Characteristics and Monitoring</strong></p><ol><li><strong>IOPS : </strong>IOPS are unit of measure representing i/o operations per second.(<strong>Operations measured in KiB</strong>).The maximum amount of data that a volume type counts as a single I/O. <strong>I/O size is capped at 256 KiB for SSD volumes and 1,024 KiB for HDD volumes </strong>because SSD volumes handle small or random I/O much more efficiently than HDD volumes.</li><li><strong>Volume Queue Length and Latency : — Number of Pending I/O </strong>requests for a device.Latency is the true end-to-end client time of an I/O operation, in other words, the time elapsed between sending an I/O to EBS and receiving an acknowledgement from EBS that the I/O read or write is complete. Queue length must be correctly calibrated with I/O size and latency to avoid creating bottlenecks either on the guest operating system or on the network link to EBS.</li></ol><p><strong>Monitor I/O Characteristics with CloudWatch :</strong></p><ol><li><strong>BurstBalance — displays the burst bucket balance</strong> for gp2, st1, and sc1 volumes as a percentage of the remaining balance. When your burst bucket is depleted, volume I/O (for gp2 volumes) or volume throughput (for st1 and sc1 volumes) is throttled to the baseline</li><li><strong>VolumeReadBytes/VolumeWriteBytes</strong>— Provides information on the read/write operations in a specified period of time. (Units: Bytes)</li><li><strong>VolumeReadOps /VoumeWriteOps— </strong>The total number of read/write operations in a specified period of time.(Units: Count)</li><li><strong>VoulmeTotalReadTime/VoulmeTotalWriteTime</strong> — The total number of seconds spent by all read/write operations that completed in a specified period of time.(Unit : Seconds)</li><li><strong>VolumeIdleTime</strong>- The total number of seconds in a specified period of time <strong>when no read or write operations</strong> were submitted.(Units: Seconds)</li><li><strong>VolumeQueueLength </strong>— The number of <strong>read and write operation requests waiting</strong> to be completed in a specified period of time.(Units: Count)</li><li><strong>VolumeThroughputPercentage</strong> - Used with Provisioned IOPS SSD volumes only. The percentage of I/O operations per second (IOPS) delivered of the total IOPS provisioned for an Amazon EBS volume.(Units: Percent)</li><li><strong>VolumeConsumedReadWriteOps — </strong>Used with Provisioned IOPS SSD volumes only. The total amount of read and write operations (normalized to 256K capacity units) consumed in a specified period of time.(Units: Count)</li></ol><p>These are Basics things to know about EBS.</p><p>Thanks for Reading!.</p><p>Resources:</p><p><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html">https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html</a></p><p><a href="https://www.datadoghq.com/blog/amazon-ebs-monitoring/">https://www.datadoghq.com/blog/amazon-ebs-monitoring/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ab3c4199c3a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/everythingatonce/what-is-elastic-block-store-and-what-we-should-know-about-5ab3c4199c3a">What is Elastic Block Store and What we should know about?</a> was originally published in <a href="https://medium.com/everythingatonce">Everything at Once</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[101 On Aerospike Data Base for Beginners]]></title>
            <link>https://medium.com/everythingatonce/101-on-aerospike-data-base-for-beginners-ea0408e0493e?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea0408e0493e</guid>
            <category><![CDATA[aerospike]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[dbms]]></category>
            <category><![CDATA[nosql-database]]></category>
            <category><![CDATA[beginner]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Sun, 16 Feb 2020 17:31:36 GMT</pubDate>
            <atom:updated>2020-02-16T17:31:36.036Z</atom:updated>
            <content:encoded><![CDATA[<p>Before diving in to Aerospike, we should know about what is RDBMS vs NoSql.</p><p><strong>RDBMS</strong> (Relational Database) — <strong>Structured Schema</strong> dealing with Relational databases , which is effectively used in insert,search,update and delete database records.</p><ol><li>RDBMS is right option for ACID problems.</li><li>Support Dynamic Queries.(Great choice for complex queries).</li><li>Predefined schema to determine the structure of data.</li><li>SQL Databases are Vertically scalable.(Example : MySQL,PostgreSQL,Oracle…)</li></ol><p>No SQL (Non SQL/Relational) — Database that provides a mechanism for storage and retrieval of data, which includes <strong>simplicity</strong> of design , simpler <strong>horizontal scaling </strong>to <strong>clusters of machines </strong>and control over availability <strong>(in favor of speed and partition tolerance).</strong></p><ol><li><strong>High</strong> <strong>Scalability : </strong>No SQL Databases use sharding for horizontal scaling (Partitioning of data and placing it on multiple machines). — Example : MongoDB,Cassandra, Aerospike..etc..</li><li><strong>High Availability : Auto Replication feature</strong> in No SQL databases makes it high available in case of any failure data replicates itself to the previous consistent state.</li><li>Types of No SQL Db :- <strong>Key Value</strong> store, <strong>Tabular</strong>, Graph databases or wide column and <strong>document</strong> based.</li></ol><p>Yess! , Now we got basic difference of RDBMS and No SQL Databases , It time to jump on Aerospike.</p><p><strong>What is Aerospike ?- </strong>is a distributed database supporting <strong>key-value store</strong> and <strong>document oriented data models</strong>. — Providing <strong>robustness and strong consistency with no downtime</strong>.</p><ol><li><strong>Scalability</strong> :- <strong>Flash and Hybrid Memory Architectures</strong> allows the aerospike database to scale Petabytes of data.</li><li><strong>Speed</strong> :- Low Latency is maintained at high scale (which makes better decision in real time).</li><li><strong>Ease</strong> of Deployment and Management.</li><li><strong>Low</strong> <strong>T</strong>otal <strong>C</strong>ost Of <strong>O</strong>wnership :- Fueled by a hybrid memory architecture and compression, Aerospike provides significantly lower TCO than first generation No SQL and relational databases.</li></ol><ul><li>In this modern times of transformation , organizations are now required to make<strong> lightning-fast decisions which powers applications like recommendation engines,digital payments, fraud detection&#39;s…</strong></li></ul><p>Technology Behind Aerospike Database :</p><ol><li>Real Time Transaction Engine</li><li>Data Distribution</li><li>Smart Cluster Management</li><li>Dynamic Data Rebalancer</li><li>High Performance Storage Engine</li><li>Cross Data Center Replication.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s_NNeAgqJNgQiJiAYNGpXQ.png" /></figure><ol><li><a href="https://www.aerospike.com/technology/#0e687fb8-5bab-95246-ac1d"><strong>Real Time Transaction Engine</strong></a><strong> : — </strong>fully capabilities of available technology , Aerospike’ s real time engine <strong>delivers the maximum performance possible </strong>and can <strong>scale millions of transactions</strong> per second at sub-millisecond latency.</li></ol><p>i. Responsible for Reading and Writing data upon request while providing <strong>consistency and isolation</strong> (which involves synchronous and asynchronous replication).</p><p>ii. Requests to alternate node if a nodes becomes unavailable as well as conflict/duplicate resolution after node rejoins the cluster.</p><p>iii. <strong>Multiple Core System</strong> — Improves latency by reducing data across multiple regions by grouping multiple threads per CPU socket.</p><p>iv. <strong>Context Switch</strong> — Some Operations are run in network thread itself without giving to CPU.</p><p>v. <strong>Data Structure Design</strong> — Safe and concurrent read ,write and delete access to index tree without holding multiple locks (does not involve acquiring multiple locks at each level by having its both reference count and its own lock).</p><p>vi. <strong>Scheduling and Prioritization</strong>— In addition to key value store operations, Aerospike supports <strong>batch queries,scans and secondary index queries.</strong>Prioritization via throughput is achieved via job partitioning based on type (CPU effort required and controlling the load generated).</p><p>vii. <strong>Memory Allocation</strong> — while minimizing the need for memory de-fragmentation, System resources are leveraged by keeping the index packed RAM.Fragmentation is minimized by grouping data objects by same namespace — so the long term creation, access ,modification and deletion are optimized.</p><p><strong>2. </strong><a href="https://www.aerospike.com/technology/#0e687fb8-5bab-95246-ac1d"><strong>Data Distribution</strong></a><strong> :- </strong>Data Partitioning that has uniform distribution of keys in the digest space ,”Avoiding the creation of hots pots during data access” — which helps in achieving high level scale and fault tolerance.</p><p>i. Application work load is uniformly distributed</p><p>ii.Performance of database operations is predictable.</p><p>3. <a href="https://www.aerospike.com/technology/#0e687fb8-5bab-95246-ac1d"><strong>Cross-data center Replication</strong></a><strong>: </strong>supports different replication topologies, including active-active, active-passive, chain, star, and multi-hop configurations.</p><ol><li>load sharing</li><li>Remote cluster management</li><li>Data shipping</li><li>Pipe-lining</li></ol><p><a href="https://www.aerospike.com/technology/#1545435078323-0-85246-ac1d"><strong>4. Storage Engine</strong></a>: — It is not just the throughput and latency characteristic, but also the ability to store and process large swaths of data that defines the ability of a DBMS to scale up. Aerospike has been designed from the ground up to leverage SSD technology. This allows Aerospike to <strong>manage dozens of terabytes of data on a single machine with sub-millisecond record access times</strong>. Aerospike supports three kinds of storage structures:<strong> Hybrid-Memory, All-Flash, and In-Memory.</strong></p><p><a href="https://www.aerospike.com/technology/#1545435076582-0-15246-ac1d">5<strong>. Dynamic Data Rebalancer</strong></a><strong> : — </strong>The data re-balancing mechanism ensures that the transaction volume is distributed evenly across all nodes and is robust in the event of node failure happening during re-balancing itself. The system is designed to be continuously available, so data re-balancing doesn’t impact cluster behavior.</p><p>6. <a href="https://www.aerospike.com/technology/#1545435074921-0-15246-ac1d"><strong>Smart Cluster Management</strong></a><strong> (Self Healing Cluster Management) — </strong>adding and removing nodes seamlessly to the cluster.</p><p>i. Hear Beat subsystem.</p><p>ii. Clustering subsystem.</p><p>iii. Exchange Subsystem.</p><p>These are overview of aerospike key points which makes more powerful distributed database engine, let see development and different memory architecture in next parts.</p><p>Thanks for reading!</p><p><strong>Resources</strong> :</p><ol><li><a href="https://www.aerospike.com/technology/#1545435074921-0-15246-ac1d">https://www.aerospike.com/technology/#1545435074921-0-15246-ac1d</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea0408e0493e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/everythingatonce/101-on-aerospike-data-base-for-beginners-ea0408e0493e">101 On Aerospike Data Base for Beginners</a> was originally published in <a href="https://medium.com/everythingatonce">Everything at Once</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Elastic Cloud Compute and What we should know about?]]></title>
            <link>https://medium.com/everythingatonce/what-is-elastic-cloud-compute-and-what-we-should-know-about-6e68b119a20c?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e68b119a20c</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Mon, 14 Oct 2019 12:24:17 GMT</pubDate>
            <atom:updated>2022-03-01T05:46:18.794Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*XreGlafEQBZqQyjDdv9jog.jpeg" /></figure><p>Amazon Web Services (AWS) is one of the most “<strong><em>popular cloud infrastructure service provider</em></strong>” (like other Microsoft Azure, Google Cloud Platform,Digital Ocean,Heroku). The AWS EC2 Cloud server is absolutely unique and helped many resources in intensive business websites scale seamlessly across the globe.</p><p><strong>What is AWS ? </strong>— is comprehensive <em>cloud computing platform that allows entrepreneurs to </em><strong><em>power their business infrastructure</em></strong> and become more agile.It is an off-shoot of Amazon’s Internal Infrastructure, launched in 2006 to enhance their management of online retail business.</p><p><strong>What is AWS EC2 and Its features ? —</strong></p><ol><li><strong>Elastic Compute Cloud or EC2 </strong>is a <em>virtual server </em>that assists users to run numerous applications (called instances) on the AWS cloud infrastructure.</li><li>With AWS EC2,we will get <strong><em>instances with different resource configurations of CPU, memory, storage and networking</em></strong>.Each type is available in different sizes so that it can cater to the work load as required.</li><li>AWS EC2 is core component of Amazon Web services platform that provides <strong>scalable cloud based computing platform.</strong></li><li>EC2 offers wide variety of <strong>standard &amp; custom templates<em> </em></strong><em>(preconfigured) </em><strong>stores</strong> as (AMI- Amazon Machine Images — that have various Linux based or windows based operating system and software configurations).</li><li>EC2 instances allows users to <strong>increase or decrease resource configuration</strong> within minutes and ability to launch instances in specific parts of the world (means on any regional demand).</li><li>EC2 can be <strong>integrated into other AWS components and features</strong> (including <strong>Auto Scaling</strong> which automatically launches and stop instances to meet the demand on your application). Instances running applications in Docker container can be grouped into clusters and managed by Amazon EC2 Container Service (ECS or EKS -Kubernetes).</li><li>Were we can <strong>easily attach Amazon Elastic Block Store (EBS) volumes</strong> to new instances for <strong>persistent block-level storage</strong> and many instance types can be <strong>optimized for EBS to provide additional performance</strong>.</li><li>A <strong>firewall</strong> that enables us to specify the protocols,ports and source IP ranges that can reach your instances <strong>using security groups.</strong></li><li>we can provide some metadata (known as tags) when we assign to resources.</li></ol><h4><strong>Things That to start Amazon EC2 :-</strong></h4><ol><li><strong>Basics</strong>: — Instances, AMI’ s ,regions &amp; availability zones, instance types,tags</li><li><strong>Networking &amp; Security </strong>— EC2 Key Pairs,Security groups , Elastic IP Address , EC2 &amp; VPC</li><li><strong>Storage </strong>— EBS &amp; Instance Store</li><li><strong>Working with Linux instances — </strong>System Manager, Run Command</li><li><strong>Related Services</strong>.- Auto Scaling, Cloud Formation, Elastic Bean Stalk, Ops Work</li><li>How to <strong>Access Amazon EC2</strong> — Command Line Interface , Power shell</li><li>EC2 <strong>Pricing Model</strong></li><li>PCI DSS Compliance.</li></ol><p>lets look about above something in detail:</p><ol><li><strong>AMI </strong>— is a template that contains a software configuration (Example : Operating system,application server and applications). From an AMI , we can launch an N number of instances — which is “<strong><em>copy of the AMI running as a virtual server in the cloud”</em></strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/0*--QmKl1x2TE9MmyT.png" /><figcaption>Instances Running from Copy of AMI until stop or terminate them.</figcaption></figure><p>2. <strong>Instances :- </strong>is a virtual server in the cloud.</p><p>i.we can <strong>launch different type of instances</strong> from single AMI.<br>ii.<strong>Instance type specially speaks about hardware of the computer</strong> used for the instance.(each instance type offers different compute and memory capabilities)<br>iii. Amazon <strong>Account has a limit number of instances</strong> that we have for running.<br>iv. Storage for Your instance <br> a.<strong> Root device</strong> for your instance contains the image used boot the instance <br> b. instance may include local storage volumes , which can configure at lanch time with block device mapping.<br> c. to keep important data safe , we should use replication strategy across multiple instances, or store your persistent data in Amazon S3 or Amazon EBS volumes.</p><h4><strong>What to look for EC2 monitoring ?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P5W25dBjjXe2vcivbDeOnw.png" /></figure><p>Regardless of the configuration of our individual instances, we want to monitor basic system level metrics to<em> </em><strong><em>keep an eye on the health of our core infrastructure (well how our resources capacity matches demand).</em></strong></p><p>Available <strong>EC2<em> </em>Metrics falls into three categories</strong>:</p><ol><li>Disk I/O</li><li>Network</li><li>CPU</li></ol><p>In addition to this ec2 metrics , we also have access to <strong>binary status checks</strong> ( which report health of our instances and AWS system they are hosted on). And we can track scheduled events (stoping, retiring, shutdown instances) that might affect our instances status or availability.</p><p>Amazon <strong><em>Cloud Watch monitoring system is the easiest way to see most resource metrics of our EC2 instances and other AWS Services</em></strong>.</p><ol><li>First By Default cloud watch uses basic monitoring (which only publishes metrics at five-minute intervals). we <strong>can enable detail monitoring when available that resolution to one minute at additional cost.</strong></li><li>Some metrics have <strong>nuances specific to EC2 instances</strong>.</li><li>AWS separates most resource by region , so we can generally view cloud watch metrics within a single region at a time. And finally cloud watch does not expose metrics related to instance memory.</li></ol><p>Now, lets jump to EC2 metric categories:-</p><h4>1. Disk I/O metrics :-</h4><p>a. <strong>2 primary kind</strong> of block level storage volumes attached to ec2-instances (<strong><em>EBS volumes and instance store</em></strong> (ephemeral) volumes). Instance store volumes are physically attached to host computer the instance runs on (this means their performance are more predictable than EBS volumes , which might be splitting hardware resources among multiple tenants.But, <strong><em>all data on instance store volumes are lost when instance is stopped or if the disk fails</em></strong>(hence “ephemeral”). on Other hand EBS volumes provides persistent storage.(many instance types not support instance store).</p><p>b. Both EBS and instance store volumes can be in <strong>solid-state drive (SSD) or Hard-Disk-Drive (HDD) </strong>Format. The number,capacity and performance of these disks differ based on the instance type and volume configuration.</p><p>So, Monitoring EC2 Disk I/O can help you ensure that our chosen instance type’s disk IOPS and throughput matches our application’s needs.</p><p>CloudWatch ’s main <strong>EC2 disk I/O metrics only collect data from instance store volumes</strong>.(means it does not offer EBS disk I/O metrics within EC2 namespace, but these only available for C5 &amp; M5 instance types). For Other instance types , <strong><em>disk I/O EBS volumes must be monitored via cloudwatch EBS‘s metric.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mXeZ1nS6OXSU5wcagGIJmA.png" /></figure><h4>Disk read/write operations :-</h4><p>Because any data stored on instance volumes is lost if the instance stops or fails, these type of volumes are best suited for the I/O intensive uses such as buffer,caches and other cases where data is stored temporarily and changes frequently.This metric pair can help determine if degraded performance is the result of consistently high IOPS, causing bottlenecks as disk requests become queued. If your instance volumes are HDD, you can consider a move to faster SSDs. Or you can upgrade the VM to an instance with more volumes attached to it.</p><h4>Disk read/write bytes:-</h4><p>Monitoring the <strong>amount of data being written to and read from disk can help reveal application-level problems. </strong>Too much data being read from disk might indicate that your application would benefit from adding a caching layer. Prolonged higher-than-anticipated disk read or write levels could also mean request queuing and slowdowns if your disk speed is not fast enough to match your use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tq9I-6rfWVqV7haQJTOzug.png" /></figure><h4>2. Network metrics:-</h4><p>Network metrics are particularly important for cloud based services like ec2 that rely on consistently strong network connections,and that might be dispersed across various availability zones.This is especially true if you have attached EBS volumes to your instances,as they are networked drives.</p><p>Instance type provide <strong>different limits both for network bandwidth and maximum transmission unit (MTU) or the largest amount of data that can be sent in a single packet.</strong></p><p>Bandwidth limit range from 5 to 25Gbps.Network MTU is a standard 1,500 bytes for most instance types but many allow jumbo frames of as much as 9,001 bytes , increasing efficiency and reducing overhead applications that transmit large amounts of data.</p><p>Selecting the right type and availability zone for your instances can improve network performance, as can configuration options such as <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html">placement groups</a> and <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html">enhanced networking</a>.</p><p>In addition to measuring network throughput in bytes, CloudWatch provides metrics for packets sent and received. Note, though, that packet metrics are only available in basic monitoring, at five-minute resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fx1e5x4ZjL5MkGVVAv0L4A.png" /></figure><h4>Network in/out:-</h4><p>These metrics report the network throughput, in bytes, of your instances. Drops or fluctuations can be correlated with other application-level metrics to pinpoint possible issues. It’s unlikely that your instances will approach their network throughput limits unless they are severely mismatched with your application’s needs, but it can still be helpful to keep an eye on possible network saturation to make sure your instances meet demand — for example, if you want to restore or backup large amounts of data quickly. Or, if certain instances are receiving considerably more network traffic than others, you may wish to use a load balancer to distribute traffic more evenly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKfCy3MVmUu5aGVtUiAyNg.png" /></figure><h3>3. CPU metrics:-</h3><p>EC2 instance type have wide range of vCPU configurations, so tracking of CPU Usage can help ensure that our instances are appropriately sized for workload.</p><ol><li>Note that cloudwatch measures the percent utilization of virtualized processing capacity of the instance (which AWS labels as <strong>compute units</strong>)</li><li>It does not report the CPU usage of the underlying hardware being used.</li><li>T2 instances are capable of <strong>bursting or providing processing power above a standard baseline level for short period of time. This is ideal for applications that are not generally CPU Intensive but may benefit from higher CPU capacity.</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0terpNrapv8Z-pAaiZ0eTg.png" /></figure><h4>CPU utilization :-</h4><p>CPU usage is one of the <strong>prime host-level metrics to monitor</strong>. Depending on the application, consistently high utilization levels may be normal. But if performance is degraded, and the application is not constrained by disk I/O, memory, or network resources, then maxed-out CPU may indicate a resource bottleneck or application performance problems. You can dive into application-level metrics or <a href="https://docs.datadoghq.com/tracing/">request traces</a> to diagnose the cause of CPU saturation, or switch to an instance type with more vCPUs.</p><p>For T2 instances with bursting, the increased processing power comes at the cost of CPU credits. EC2’s CPU credit metrics help keep track of your available balance and usage so that you are aware of possible charges as a result of extended bursting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E0qSLxrula6mw6q6oeZqQg.png" /></figure><h4>CPU credit balance:-</h4><p>For standard T2 instances with bursting, a burst can continue only as long as there are available CPU credits, so it’s important to monitor your instance’s balance. Credits are earned any time the instance is running below its baseline CPU performance level. The initial balance, accrual rate, and maximum possible balance are all dependent on the instance level.</p><h4>CPU credit usage:-</h4><p>One CPU credit is equivalent to <strong>one minute of 100 percent CPU utilization</strong> (or two minutes at 50 percent, etc.). Whenever an instance requires CPU performance above that instance type’s baseline, it will burst, consuming CPU credits until the demand lessens or the credit balance runs out. Keeping an eye on your instances’ credit usage will help you identify whether you might need to switch to an instance type that is optimized for CPU-intensive workloads. Or, you can create an alert for when your credit balance drops below a threshold while CPU usage remains above baseline.</p><h4>CPU surplus credit balance</h4><p>In the case of T2 Unlimited instances, if the CPU credit balance is exhausted but burst performance is still required, the instance will consume additional credits to maintain greater CPU usage. This metric tracks the accumulated balance.</p><h4>CPU surplus credits charged</h4><p>This metric tracks the difference between the number of credits accumulated and the current credit balance that can be used to pay down the surplus balance. In other words, it is a measure of extra credits that will result in additional charges.</p><h3>Status checks</h3><p>EC2 status checks are, simply, checks on the status of an individual instance and of the AWS systems hosting it. Status checks are available at one-minute intervals. They provide a clear, high-level indication of an instance’s health and whether there is a problem with either the larger AWS infrastructure or with the software or network configuration of the instance itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BsLYEbtdIMYI4ZUNfLxO3g.png" /></figure><h4>Metric to watch: Status check failed — system</h4><p>This status check reports whether there are problems detected with the system hosting the instance. Generally these are problems with the Amazon-administered computer on which your instance is hosted and are outside of your control — as an example, power loss. Possible resolutions include stopping and restarting an instance to switch it to a new host computer. (Keep in mind that instance store–backed volumes will be lost if the instance is stopped.) This check returns False (0) if an instance passes the system status check, and True (1) if it fails.</p><h4>Metric to watch: Status check failed — instance</h4><p>This check reports whether there are any problems detected with the instance itself and returns False (0) if an instance passes the status check and True (1) if it fails. Problems that might cause this check to fail include software or network configuration issues, a corrupted file system, etc. <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html">Amazon’s troubleshooting tips</a> offer causes and possible solutions for common errors that result in a failed status check.</p><h3>Events</h3><p><a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-instances-status-check_sched.html">Events</a> are scheduled changes in an <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html">instance’s lifecycle</a>. AWS may initiate events if problems are detected or if standard maintenance is required on an instance’s host computer.</p><p>Events include:</p><ul><li>Stopping an instance. This is only applicable to EBS-backed instances, which retain their data and can be restarted. If restarted, the instance will be hosted on a new computer.</li><li>Retiring an instance. This will terminate the instance and delete any attached volumes.</li><li>Rebooting either the instance (again, only applicable to EBS-backed instances) or the host computer.</li><li>System maintenance, possibly affecting the instance’s performance or availability.</li></ul><p>AWS will inform users if an event has been scheduled for their instances. But you can also use CloudWatch’s events stream to track events and monitor upcoming changes to your EC2 infrastructure that might degrade performance or affect data availability. This is particularly important for any instance store volumes — even if they are connected to an EBS-backed instance — as all data stored on those volumes is lost. Keeping an eye on your EC2 events will help you determine if you need to migrate data to a new instance before the current one is terminated or stopped.</p><p>Hope we learned something about EC2 and how to monitor.</p><p>Thanks for reading!.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e68b119a20c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/everythingatonce/what-is-elastic-cloud-compute-and-what-we-should-know-about-6e68b119a20c">What is Elastic Cloud Compute and What we should know about?</a> was originally published in <a href="https://medium.com/everythingatonce">Everything at Once</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Runlevels , rc.d Scripts and Copy System Logs to S3]]></title>
            <link>https://medium.com/everythingatonce/runlevels-rc-d-scripts-and-copy-system-logs-to-s3-885578e3bc26?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/885578e3bc26</guid>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[shell]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Wed, 09 Oct 2019 13:23:40 GMT</pubDate>
            <atom:updated>2019-10-09T13:23:40.878Z</atom:updated>
            <content:encoded><![CDATA[<h3>Runlevels , rc.d Scripts and Copy System Logs to S3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Aem51Izd4AW4qf7HM65jg.png" /></figure><p>As you might have read about How Linux Boots it uses runlevels to determine what to do.Lets study runlevels and rc scripts in detail.</p><h4>Runlevels:</h4><p>Runlevels are the state to which computer intends to go or it intends to reach.</p><p>There are total 7 runlevels.</p><ol><li>Runlevel 0: Halt or Shutdown</li><li>Runlevel 1: Single-user mode</li><li>Runlevel 2: Basic Multi-user mode but without networking</li><li>Runlevel 3: Full text-based Multi-user mode (includes networking)</li><li>Runlevel 4: X11 with session managers (Generally not used by most of the operating systems and is customizable)</li><li>Runlevel 5: Full GUI based Multi-user mode</li><li>Runlevel 6: Reboot</li></ol><p>These are basic definitions what each runlevel is responsible for.</p><p>For Example : if an operating system is required to shutdown , then it go for runlevel 0. we can switch between runlevels using init command.(init 0).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lLZCeXxJxSFivWjmQgT2EA.png" /><figcaption>init 0 system to shutdown</figcaption></figure><p>init 0 will shutdown the system. And Each System has a default runlevel to which it boots. To see which runlevel you are in type runlevel in terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KrTPK0LhUyw3xpLZH9-2Xw.png" /></figure><p>Runlevel and target file have the following relation:</p><ol><li>runlevel 0: poweroff.target</li><li>runlevel 1: rescue.target</li><li>runlevel 2: multi-user.target</li><li>runlevel 3: multi-user.target</li><li>runlevel 4: multi-user.target</li><li>runlevel 5: graphical.target</li><li>runlevel 6: reboot.target</li></ol><p>We should <strong><em>never mess with the default runlevels unless you are absolutely sure with what you’re doing</em></strong>.Like, <strong><em>if you manage to set runlevel 0 or 6 as default you will end up in a loop kind of condition</em></strong>. Your PC will shut down( or reboot depending on what runlevel you set) as soon as it has booted.</p><h4>rc Scripts :</h4><p>When a computer entered into specific runlevel,it runs scripts according to the runlevel it’s in.The rc here comes from use in the older computers which had rc files to contain commands that run at start up.</p><p>These files are called <strong>runcom (RunCommand) files.</strong></p><p>Our Script Must contain the line before any command executes.</p><pre>. /etc/init.d/functions</pre><p>This is because , our script might be executed at a time when nothing is loaded, not even file system (though it can be specified in boot facilities in “Required-Start” in LSB styled stanzas). So we cannot say that each command we write will already be present — Remember that each command is executed from a SCRIPT present in path variable from our system). hence in order for those commands to work, we need the functions file to be loaded as the first step.</p><p>This function file contains the function that a shell script might use.(files are located in /etc folder) The folders containing these scripts are:</p><p>/etc/init.d<br>/etc/rc.d/rc0.d<br>/etc/rc.d/rc1.d<br>/etc/rc.d/rc2.d<br>/etc/rc.d/rc3.d<br>/etc/rc.d/rc4.d<br>/etc/rc.d/rc5.d<br>/etc/rc.d/rc6.d</p><p>And, folders with the same name also exist in /etc but they are actually symbolic links of there corresponding folders in /etc/rc.d For example:</p><p>/etc/rc0.d (Symbolic link of /etc/rc.d/rc0.d)</p><p>Each of these folders contain symbolic links of scripts that are actually stored in init.d for the runlevel they have in their name. Like /etc/rc.d/rc0.d contains a symbolic link for scripts in /etc/init.d that are required to be run at shutdown.</p><p>Now if you open any of these folders you’d find out weird names with the structure like</p><p>S|K[01-99]filename where filename is the name of the script in /etc/int.d</p><p>Like K50netconsole. Now let’s see what these names mean.</p><h3>Demystifying the symlink names:</h3><p>The terms have their meaning as follows:</p><ol><li><strong>S: </strong>start the service, if it’s not already started.</li><li><strong>K: </strong>Kill the service, if it’s running.</li></ol><p><strong>01–99 is the priority</strong> for that start or kill task.</p><p>filename is the exact same name of the file in /etc/init.d folder.(Remember that it’s actually /etc/init.d that actually houses those scripts. Rest of the folders just have symlinks to these.)</p><p>So, If a file named shutdown-script is to be run at shutdown all times with a priority of 90 then the file shutdown-script should be present in init.d folder. And a symlink to it must be created in /etc/rc.d/rc0.d with the name S90shutdown-script, rest of the folders rc[1-6] can have K90shutdown-script. If 90 is the priority required to shut down.</p><p><strong>Priority Of these scripts:</strong></p><p>Essentially, it can be better thought of as a position, and think of them being executed <strong>downwards from 01 to 99.</strong> For example, if you have a script that requires the network to be switched on then the network script should be at a higher position or priority than your script. Like the network, a script can have a priority of 01 and your script can have any number greater than it. So that it’s executed <em>after the network script.</em></p><h3>The chkconfig:</h3><p>the chkconfig command here deserves the ‘The’ in the heading for how easy it makes to manage these scripts and runlevels! Yes, this command just needs your source script file and manages creation deletion, the setting of S or D and also the priority level!</p><p>chkconfig has an amazing man page and here are the basic commands that will get you going.</p><pre>chkconfig <em>--list</em></pre><p>This prints the System V services, and that will do for us.</p><pre>chkconfig <em>--add</em></pre><p>Is used to add your file for management by chkconfig. BUT wait. This command won’t run unless you have some things set with your script.</p><p>First and foremost, your script should exist in the home of all scripts, and that is:</p><pre>/etc/init.d</pre><p>Paste your script here.</p><p>Secondly, your file must have certain comment lines so that it can be managed by chkconfig.</p><p>It must have these comment lines, though the description is not absolutely necessary and the script would work without it, it’s recommended. So that anyone who intends to clean up the unnecessary services knows if this script is essential or not:</p><pre># chkconfig: 235 29 88<br># description: The description <strong>of</strong> the script <strong>like</strong> what it does</pre><p>The first line here has some numbers ahead chkconfig: like 235, 29 and 88.</p><p>here the First number is the runlevels it’s required to run on combined into a string. Like 235 here means it’s supposed to run in runlevel 2,3 and 5.</p><p>The next two numbers are the priority for start and stop respectively. So in this example, start priority is 29 and stop priority is 88.</p><p>you can use backslash for a multi line comment though you still require a #, in the beginning, to identify it as a comment.</p><p>This is how it creates the symlinks with required priorities.</p><p>Another way you can do this is enter LSB(Linux Standard Base) style init stanzas instead of these two comments. For it you can do it like:</p><pre>### BEGIN INIT INFO<br> # Provides: scriptA<br> # Required-Start: scriptB<br> # Default-Start: 2 3 5<br> # Default-Stop: 0 1 6<br> # Description: Description of what the script does<br> ### <strong>END</strong> INIT INFO</pre><p>This one gives you more ease though it just looks a bit more complicated.</p><p>In this, Required-Start is the script that is required to start this script. That is, it should be running before the current script gets up. Provides: It tells about the boot facilities that this script provides so that those will be counted as already present when another script requires it. Hence priorities will be set accordingly to what is required to be already present and what it&#39;ll provide. Default-Start or Stop tells which runlevel should this start on and stopped(if running) respectively.</p><p>And don’t forget a shebang at the beginning of the file.</p><p>Now we are all set to go with these things done. Let’s execute this command to add it to the management of chkconfig:</p><pre>chkconfig <em>--add $filename</em></pre><p>like for a file named <em>ec2-copylog</em> in init.d it would be:</p><pre>chkconfig <em>--add ec2-copylog</em></pre><p>Now most probably your file will be visible with a status of on under the respective runlevels when you run</p><pre>chkconfig <em>--list</em></pre><p>And if it’s not or you want to change runlevels you can go like:</p><pre>chkconfig <em>ec2-copylog</em> <em>--level 0 on</em></pre><p>Will make it run on runlevel 0 that is at shutdown.</p><p>Similarly off argument will kill it if present in that level. If no level is specified then these act on level 2,3,4 and 5. You can also specify multiple levels like --level 26 for level 2 and 6.</p><pre>chkconfig <em>--del ec2-copylog</em></pre><p>will delete its symlinks from all rc folders but not from the init.d folder. That is required to be done manually. Ad chkconfig generally doesn’t modify the content of init.d but rc folders.</p><p><strong>Example ec2-copy log : (under /etc/init.d/ec2-copylog)</strong></p><pre><strong>#!/bin/sh<br></strong><em>#<br># ec2-copylog<br>#<br># chkconfig: 3 11 01<br># Description: This script is responsible for copying logs S3<br>#                   as a daemon<br>#<br>### BEGIN INIT INFO<br># Provides:          ec2-shutdown-hook<br># Required-Start:    network<br># Required-Stop:     network<br># Default-Start:     3<br># Default-Stop:<br># Description:       This script is responsible for copying logs S3<br>#                    as a daemon.<br>### END INIT INFO<br><br></em>. /etc/init.d/functions<br><br><strong>copylog</strong>() {<br>    echo -n <strong>$&quot;Starting Copying logs: &quot;<br>    </strong>HOSTIP=<strong>$</strong>{HOSTNAME:-default}<br>    SOURCEFOLDER=<strong>&quot;$directory from where we need to copy&quot;<br>    </strong>BUCKETTOCOPY=<strong>&quot;$bukcetname&quot;<br>    </strong>CURRENTDATE=<strong>$</strong>(date +<strong>&quot;%d-%m-%Y&quot;</strong>)<br>    DIRECTORY=<strong>&quot;${HOSTIP}-${CURRENTDATE}&quot;<br>    </strong>echo Copying from $SOURCEFOLDER to Bucket $BUCKETTOCOPY Directory $DIRECTORY<br>    aws s3 cp $SOURCEFOLDER s3://$BUCKETTOCOPY/$DIRECTORY --recursive<br>}<br><br><strong>case &quot;$1&quot; in<br>  </strong>start)<br>        touch /var/lock/subsys/ec2-shutdown-log<br>        <strong>;;<br>  </strong>restart|force-reload|stop)<br>        <em>copylog<br>        </em>rm -f /var/lock/subsys/ec2-shutdown-log<br>        <strong>;;<br>  </strong>*)<br>        <em>copylog<br>        </em>rm -f /var/lock/subsys/ec2-shutdown-log<br>        exit 0<br><strong>esac</strong></pre><p>Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=885578e3bc26" width="1" height="1" alt=""><hr><p><a href="https://medium.com/everythingatonce/runlevels-rc-d-scripts-and-copy-system-logs-to-s3-885578e3bc26">Runlevels , rc.d Scripts and Copy System Logs to S3</a> was originally published in <a href="https://medium.com/everythingatonce">Everything at Once</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Spring ? & 101 On Spring Start!]]></title>
            <link>https://medium.com/@prabhur24/why-spring-101-on-spring-start-3be672f9d75?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/3be672f9d75</guid>
            <category><![CDATA[spring-framework]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[spring]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Sun, 29 Sep 2019 13:24:21 GMT</pubDate>
            <atom:updated>2019-09-29T13:27:31.430Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Spring ? &amp; <strong>101</strong> On Spring!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*_9PeCmUm1wKuEShk.png" /></figure><p>First step towards the most popular application development framework for enterprise Java applications. When Spring came out it was <strong>“simpler,light-weight alternative to J2EE, to make J2EE development easier”.</strong></p><p>Now hold on! <strong><em>what enterprise java refers too? </em></strong>— Its computer software that <strong>satisfy the needs of an organization rather than single/individual user</strong>. (Examples are Accounting System, ERP , CMS, CRM , Billing…so on).</p><p>So, Now <strong><em>what is the Benefit of Using Spring?</em></strong></p><ol><li><strong>Uses POJO </strong>(Plain Old Java Object) Makes it simpler 🔥 to build enterprise class applications as compared to the heavy weight EJB.</li><li><strong>POJO is java object without any restrictions</strong> (no need to extend any predefined specific classes or implement specified interfaces in framework).</li><li>It Promotes loose coupling by making <strong>“Dependency of Injection”</strong>, instead of hard coding the dependencies of an object, simply specify the dependencies <strong>via configuration file</strong> (annotations or java code).</li><li>Uses <strong>Aspects of Programming (AOP) — </strong>allows separation of cross cutting concerns like logging,auditing,declarative transactions, security,caching..etc.</li><li><strong>Minimize the boilerplate Java code — </strong>which has collection of helper packages and classes to make it easier and avoid boilerplate code and “we have to worry about classes we need and rest”.</li><li>Uses <strong>Existing Technologies </strong>like ORM,logging , JEE , Quartz , JDK Time Framework and others..</li><li><strong>Core Features</strong> of the Spring Framework can be used in developing any java application, <strong>but there are extensions for building web applications</strong> on the top of Java EE Platform.</li></ol><p>Now, lets explore some concepts that lie as the ❤️ of Spring.</p><p><strong>Dependency Injection (DI) :</strong></p><ol><li>The <strong>Inversion of Control (IOC) </strong>is a design principle in which flow of control of a system is invented. General concept , where the dependency injection is one concrete example of it.</li></ol><blockquote>💡 The Spring IoC Container “<strong>creates, manages and injects Object’s <em>dependencies</em></strong>”</blockquote><p>2. Dependency Injection “Separates an object from dependencies in which dependencies to be injected during run time”.</p><blockquote>💡 This decision is done by Spring in <strong>Bean Configuration file</strong> (or using annotation or java code) instead of modifying the code.</blockquote><pre><strong>public class </strong>App {<br><br>   <strong>private </strong>Service <strong>service</strong>;<br><br>   <strong>public </strong>App(Service service) {<br>      <strong>this</strong>.<strong>service </strong>= service;<br>   }<br>}</pre><p>And the benefit is we are able to replace and extend the functionality of dependencies without changing the dependent App class at all.</p><p>3. Decouple our application objects from its dependencies.</p><p><strong>Aspects of Programming (AOP) :</strong></p><ol><li>It allows the separation of <strong>cross cutting concerns from the business logic</strong>.</li></ol><blockquote>💡 The cross cutting concerns are features that <strong>span across the different part of application and affect the entire application. </strong>Examples are logging, caching, security, declarative transactions..etc.</blockquote><p>But why we should use it?</p><p>What happens when we need to change , example “Specific security consider across all application , such change would require more effort , if especially security consideration is used across multiple methods”.</p><p>So, <strong>AOP attempts to solve this problem be allowing us to express “Cross-cutting concerns in stand alone modules called aspect”</strong>.</p><p>Example : Security Module “aspect” can advise that performs security check (<strong>actual piece of code is executed</strong>) in corresponding program. which is <strong>done by either XML Syntax or AspectJ.</strong></p><p>2. Decouple cross cutting concerns from the objects.</p><p>Yass!! Now lets dive to <strong>Spring Architecture </strong>🏡!!</p><p>Spring is organized in a <strong>modular fashion which allows us to only worry about the modules we need and rest ignore</strong>!.</p><blockquote>💡 <strong>Modular Programming</strong> — is a software design technique<strong><em> separates the functionality of a program into independent modules</em></strong>. each contains one specific functionality.</blockquote><p>Now <strong><em>Modularity</em></strong> Vs <strong>AOP : </strong>key unit of modularity in OOP’s is the class, wheres in AOP the unit of modularity is the aspect.</p><p>Spring Provides 20 Modules, which built on top of its core container. These modules provide everything that a developer may need for use in enterprise application development.</p><p>💚 of Spring Framework :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*V_fxOdNtgDxDiRC_.png" /><figcaption>Spring Architecture — <a href="https://docs.spring.io/spring/docs/5.0.0.RC2/spring-framework-reference/overview.html">spring.io</a></figcaption></figure><p>Let see more information about each 💚 modules:</p><h4><strong>Core Container </strong>📦<strong>:</strong></h4><ol><li>The <strong>Core and Beans modules provides the fundamental part of framework</strong> and includes the Inversion of control &amp; Dependency Injection.</li><li>Bean Module Provides the <strong>Bean Factory</strong> which is sophisticated implementation of factory pattern. It <strong>decouples the configuration and specification of dependencies from the actual program logic</strong>.</li><li><strong>Context Module</strong> build on the core and beans,It is mean to access any objects defined &amp; configured. Where context module extends the BeanFactory and add supports for application life cycle events and validation.Enables many enterprise services such as JNDI access,EJB integration, Remoting and Scheduling and also facilitates easy integration with other framework.</li><li><strong>SpEL</strong> : provides powerful language of expression and manipulate the object at run time.</li></ol><h4>Data Access/Integration Modules 📄📑:</h4><p>Provides abstraction layer and used to communicate with Databases.</p><ol><li><strong>JDBC</strong> : provides the JDBC abstraction layer that removes the need of JDBC related coding.</li><li><strong>ORM</strong> : provides an integration layer doe popular object relational mapping api including JPA, Hibernate, JDO &amp; iBatis. [Object Relational Mapping]</li><li><strong>OXM</strong> : provides an abstraction layer that supports Object/XML Mapping implementation.</li><li><strong>JMS</strong> : contains features for producing and consuming messages(emails).[Java Messaging Services]</li><li><strong>Transactions</strong> : supports transaction management for classes.</li></ol><h4>Web Modules 🏠:</h4><p>It is home for Spring MVC Framework and we can integrate other technologies like JSF.</p><ol><li><strong>Web</strong> : provides basic web oriented integration such as multipart file upload and initialization of the IoC container through servlet listeners and a web oriented application context.</li><li><strong>Web-MVC</strong> : contains Spring Model-View Controller and Rest web services implementation for web applications.</li><li><strong>Web-Socket : </strong>provides support for web socket based two way communication between client and server web applications.</li><li><strong>Web-Portlet</strong> : provides the MVC implementation to be used in portlet environment and mirror’s the functionality of Web-MVC.</li></ol><h4><strong>AOP &amp; Instrumentation:</strong></h4><ol><li><strong>AOP</strong> : Aspect oriented programming implementation allows us to separate the cross cutting concerns from the business logic (as explained above).</li><li><strong>AspectJ</strong> : separate module provides integration with AspectJ.</li><li><strong>Instrumentation</strong> : provides support to class instrumentation support and class loader implementations.</li></ol><h4>Messaging 📥📤:</h4><p>Serves as a foundation for messaging based applications.</p><ol><li>Brings the features of asynchronous messaging system and provides support for STOMP to be used over web socket for user’s to exchange messages.</li></ol><p><strong>Test</strong>:</p><p>It supports for Unit Testing &amp; Integration Testing of spring components with Junit &amp; TestNG. It also provides mock objects that we can use to test our code in isolation.</p><h4>Setting Up The Environment 💪</h4><p>Before digging deeper, and write out first code in Spring, we need to prepare our development environment.</p><ol><li>JDK</li><li>Install Java Application Server (Most widely used tomcat)</li><li>Java IDE</li><li>Spring Jar Files using Gradle.</li></ol><p><strong>Gradle</strong> (Recommended) : tool used to describe how software is built, and its dependencies. It then downloads all the jar files from the internet automatically. So, instead of manually downloading the jar files, we can use gradle instead.</p><p>More about <a href="https://gradle.org/">Gradle!</a>.</p><p>In Next Part let see about creating first Spring Project using Gradle! .</p><p>In case of queries please feel free to comment!.</p><p>Thanks for the time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3be672f9d75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Terraform]]></title>
            <link>https://medium.com/everythingatonce/an-introduction-to-terraform-17135670029b?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/17135670029b</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Sun, 08 Sep 2019 18:13:30 GMT</pubDate>
            <atom:updated>2019-09-08T18:13:30.875Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ubtqEZA1SbYxGyRFDtA98A.png" /></figure><p><em>About Terraform :</em><strong><em> With Terminal a few bytes of code and no SSH logins we can spin up in minute , a production ready cluster and a capable of heavy work loads and here it comes “Terraform” — </em></strong><em>an extremely powerful tool written in “Go” that helps us “</em><strong><em>writing</em></strong><em> </em><strong><em>,planing and creating </em>stateful <em>reproducible infrastructure as code”.</em></strong></p><p>Terraform by HashiCorp an orchestration tool for “<strong>building,changing and versioning infrastructure safely and efficiently</strong>” that allows to represent instances and other resources with declarative code inside configuration files.(instead of manually creating those resources).</p><ol><li>Write configuration files (.tf) in which we declare the elements or piece of <em>infrastructure that we want to create.</em></li><li>Tell terraform to analyze our created configuration file and then create corresponding infrastructure.</li></ol><p>Primary Job of terraform is to create,modify and destroy servers and other resources,can manage existing or popular service providers like custom in house solutions.</p><ol><li>Infrastructure as <strong>code</strong> : Instead of going to provider console and doing manually -Terraform allows something in code. — <strong>Minimization of human error,Better collaboration among team members.</strong></li><li><strong>Automation</strong> of our infrastructure.</li><li>Keep our infrastructure in <strong>certain state.</strong> ( ex : 2 web instances with 2 volumes and 1 load balancer)</li><li>Make our infrastructure <strong>auditable</strong>. (We can keep infrastructure in a version control like Git)</li><li>Ansible,Chef,Puppet,Saltstack focus on automating installation the installation and configuration of software (will see why we choose terraform then these software in one part)</li><li>Terraform can automate provisioning of infrastructure itself.</li></ol><p><strong>Installation Terraform </strong>: <a href="https://www.terraform.io/downloads.html">Download from here</a>! according to machine and unzip downloaded file. That’s it ready for Go!</p><p><strong>Step : 1 : Terraform State : </strong>Terraform<strong> </strong>keeps<strong> </strong>the <strong>State </strong>of the managed infrastructure and its configuration.</p><ol><li>By Default state is stored locally in a JSON formatted file called terraform.state.</li><li>What it keeps? from first lift to latest change of our infrastructure’s plan and every chunk of information , to keep track of meta data is saved in the state file.</li><li>Bring Benefits to scenarios like versioning,performance monitoring,debugging,rollbacks,rolling updates, immutable deployments,traceability and so on..</li></ol><p><strong>Now Let’s see how it works :</strong></p><ol><li>Terraform uses the <strong>state</strong> to create plans , which are <strong>representation</strong> of the resources which we are <strong>deploying and the changes we are applying.</strong></li><li><strong>Prior to any operation , a refresh is performed </strong>to synchronise the state with the running resources.</li><li><strong>state</strong> file (terraform.state) contains sensitive information and plays a crucial role in entire management process of our infrastructure.</li><li>Now Next question arise? How to store the state? when to store state? and How to secure it ?</li><li>What determines how the state is stored and loaded? How do we modify the default behavior? The answer is “<strong>Backend</strong>”.</li></ol><p><strong>Step : 2: Terraform Backend: is an abstraction that determines how the state is handled and way certain operations are executed and enable a number of important features.</strong></p><ol><li><strong>Backend </strong>— store the state remotely and protects it with lock to prevent corruption.</li><li><strong>Better Protection for sensitive data</strong> : when the state is retrieved from backend it get stored in memory and never persisted on a local drive.</li><li><strong>Remote Operations : </strong>For large infrastructure or when pushing specific changes , terraform apply can take quite long time.Normally for the entire duration of the deployment,it is not possible to disconnect the client without compromising its execution.Along with remote state storage and locking, remote operations are a big help for teams and more structured environments.</li></ol><p><strong>Step : 3 : Configuration: </strong>Backends are configured in Terraform files with the HCL syntax, inside the terraform section. The following simplified snippet shows how a remote backend can be enabled leveraging an AWS s3 bucket, where the <em>terraform.tfstate</em> will persist.</p><pre><strong>terraform </strong>{<br>  <strong>backend &quot;s3&quot; </strong>{<br>    <strong>bucket </strong>= <strong>&quot;</strong>&lt;your_bucket_name&gt;<strong>&quot;<br>    key </strong>= <strong>&quot;terraform.state&quot;<br>    region </strong>= <strong>&quot;</strong>&lt;your_aws_region&gt;<strong>&quot;<br>  </strong>}<br>}</pre><p>when configuring a backend rather than default for first time, terraform will provide the option to migrate the current state to new backend , in order to transfer existing data and not losing any information.</p><p>It is recommended,though not mandatory, to manually backup the state before initializing any new backend by running “terraform init”.To do that, it is enough to copy the state file outside the scope of the project, in a different folder. The initialisation process should create a backup as well.</p><p>Time for coding and demonstrate how to set up a remote backend, with a real life example.</p><p><strong>Step : 4: The AWS s3 Backend : </strong>Setting up a Terraform backend it is relatively easy. Let’s see how to implement one with AWS s3.<br>To run the code of the example, be sure to have available AWS IAM credentials with enough permissions to create/ delete s3 buckets and put bucket policies.</p><pre># backendstate-store.tf</pre><pre><strong>terraform </strong>{<br>  <strong>backend &quot;s3&quot; </strong>{<br>    <strong>bucket </strong>= <strong>&quot;everythingatonce&quot;<br>    key </strong>= <strong>&quot;terraform.state&quot;<br>    region </strong>= <strong>&quot;ap-south-1&quot;<br>    encrypt </strong>= <strong>true<br>  </strong>}<br>}</pre><p>Once in place, terraform must be initialised <em>(again)</em>.terraform init.</p><p><strong>What about locking?</strong></p><p>Storing the state remotely brings a pitfall, especially when working in scenarios where several tasks, jobs and team members will have access to it. Under these circumstances, the risk of multiple concurrent attempts to make changes to the state, is high. Here comes to help the <strong>lock</strong>, a feature that prevents the state file to be opened while already in use.</p><p>Back to our example, the lock can be implemented by creating an <strong>AWS DynamoDB Table</strong>, that will be used by terraform to set and unset the locks.</p><p>We can create the table resource using terraform itself:</p><pre><strong>resource &quot;aws_dynamodb_table&quot; &quot;dynamodb-terraform-state-lock&quot; </strong>{<br>  <strong>name </strong>= <strong>&quot;terraform-state-lock-dynamo&quot;<br>  hash_key </strong>= <strong>&quot;LockID&quot;<br>  read_capacity </strong>= 20<br>  <strong>write_capacity </strong>= 20<br>  <strong>attribute </strong>{<br>    <strong>name </strong>= <strong>&quot;LockID&quot;<br>    type </strong>= <strong>&quot;S&quot;<br>  </strong>}<br>  <strong>tags </strong>= {<br>    <strong>Name </strong>= <strong>&quot;DynamoDB Terraform State Lock Table&quot;<br>  </strong>}<br>}</pre><p>and deploy it (terraform apply).</p><p>Once the command execution is completed, the locking mechanism must be added to our backend configuration as follow:</p><pre><strong>terraform </strong>{<br>  <strong>backend &quot;s3&quot; </strong>{<br>    <strong>bucket </strong>= <strong>&quot;everythingatonce&quot;<br>    key </strong>= <strong>&quot;terraform.state&quot;<br>    region </strong>= <strong>&quot;ap-south-1&quot;<br>    encrypt </strong>= <strong>true<br>    dynamodb_table </strong>= <strong>&quot;terraform-state-lock-dynamo&quot;<br>  </strong>}<br>}</pre><p>That’s it we are done now.</p><p>In case of queries please feel free to comment!.</p><p>Thanks for the time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17135670029b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/everythingatonce/an-introduction-to-terraform-17135670029b">An Introduction to Terraform</a> was originally published in <a href="https://medium.com/everythingatonce">Everything at Once</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Auto Scaling Amazon EC2 and How ? — Day 3]]></title>
            <link>https://medium.com/everythingatonce/auto-scaling-amazon-ec2-and-how-day-3-cfda66c7b979?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfda66c7b979</guid>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Wed, 28 Aug 2019 17:40:55 GMT</pubDate>
            <atom:updated>2019-08-28T17:40:55.506Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Auto Scaling Amazon EC2 and How ? — Day 3</strong></h3><p>In Tradition IT World, there are limited number of servers to handle the application load.when the <strong><em>number of requests increases load on the server also increases</em></strong>, which cause failures and latency in system.</p><p>Amazon web service provide <strong>Amazon EC2 auto scaling services to overcome this failure</strong>.Auto Scaling ensures that Amazon EC2 instances are sufficient to run our application.</p><p><strong>What is Amazon EC2 Auto Scaling ? — <em>helps you to ensure that we have the correct number of ec2 instances available to handle the load of you application.</em></strong></p><ol><li><strong><em>Collection of EC2 instances</em></strong><em> called Auto Scaling Groups.</em></li><li><em>Specify the </em><strong><em>minimum number of instances</em></strong><em> in each auto scaling groups and auto scaling groups ensures that </em><strong><em>never goes below the size.</em></strong></li><li><em>Specify the</em> <strong><em>maximum number of instance </em></strong><em>in each auto scaling groups and auto scaling groups ensures that </em><strong><em>never goes above the size.</em></strong></li><li><em>Specify the</em><strong><em> desired capacity</em></strong><em> either when create or after any time — ensures that group has this many instances.</em></li><li><em>Specify the </em><strong><em>Scaling Policies- Amazon EC2 Auto scaling can launch or terminate instances as demand on our application increases or decreases.</em></strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*8T5n-pIiVSHZ7TAZnEjNuQ.png" /></figure><p><strong>Auto Scaling Components:</strong></p><ol><li><strong>Groups</strong> — Each EC2 instances are organized into groups so that they can be treated as logical units for the purpose of scaling and management.(When you create a group we can specify minimum, maximum, desired number of EC2 instances).</li><li><strong>Configuration Templates- </strong>Our Group uses launch configuration or launch template for its ec2 instances,we can specify information such as AMI ID,instance type, security groups , block device mapping…</li><li><strong>Scaling Options — </strong>Amazon EC2 Auto scaling provides several ways to scale our auto scaling groups.we can configure scale based on occurrence of specified conditions (dynamic scaling) or on a schedule.</li></ol><p><strong>Auto Scaling Life Cycle :- </strong>EC2 Instance in Auto Scaling Group has a <strong>path or life cycle (that differ from other ec2 instances)</strong></p><ol><li>Life cycle <strong>starts when Auto Scaling group launches an instance </strong>and puts into service.</li><li>Life cycle ends when <strong>Auto Scaling group terminate the instance, or Auto Scaling group takes the instance out of service and terminates it.</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eXWyj3_9jxhU6k16GjfP9A.png" /><figcaption>life cycle of auto scaling group ec2 instance</figcaption></figure><p>Let see detail about each state whats happening in life cycle.</p><p><strong>1.Scale Out: — </strong>Directs the Auto Scaling group to <strong>launch EC2 instance and attach then to group.</strong></p><p>How to attach ec2 instance to group ?</p><ol><li>Manually increase the size of group.</li><li>create a scaling policy to automatically increase the size of group based on increase in demand.</li><li>set up scaling by schedule to increase the size of group.</li></ol><p>When a scale out event occurs, the auto scaling group launches the required number (one or more) of ec2 instances, using it launch configuration.</p><p>These instances start in <strong>PENDING STATE</strong> , if we added life cycle hook we can perform custom action, when each instance is fully configured and passes EC2 health check ,it is attached to Auto Scaling group and it enters the <strong>In Service State.</strong>(The instance is counted against the desired capacity of the auto scaling group).</p><p><strong>2. Instance In Service :- </strong>Instance remains In service state until one of following state occurs.</p><p>a. Scale In event occurs and Amazon Auto Scaling chooses to terminate the instance in order to reduce the size of the Auto Scaling group.</p><p>b. put the instance into a<strong> Stand By state.</strong></p><p>c. Detach the instance from Auto scaling group.</p><p>d. Instance fails a required number of health checks, so it is removed from the auto scaling group.</p><p><strong>3. Scale In: — </strong>Directs the Auto Scaling group to <strong>detach EC2 instance from the group and terminate them.</strong></p><p>How to detach ec2 instance to group ?</p><ol><li>Manually decrease the size of group.</li><li>create a scaling policy to automatically decrease the size of group based on decrease in demand.</li><li>set up scaling by schedule to decrease the size of group.</li></ol><p>It is important that we create a corresponding scale in event for each scale out event that you create.</p><p>when a scale in event occurs, the Auto Scaling group detaches one or more instances. The Auto Scaling group uses its termination policy to determine which instances to terminate.Instance that are in process of detaching from auto scaling group and shutting down enter the <strong>TERMINATING STATE</strong> and cant put back into the service.if we added life cycle hook we can perform custom action, Finally instances are completely terminated and enter the <strong>TERMINATED STATE.</strong></p><p><strong>4. Attach an Instance —</strong>we can attach a running EC2 instance that meets certain criteria to our Auto Scaling group. After the instance is attached, it is managed as part of the Auto Scaling group.</p><p><strong>5. Detach an Instance — </strong>we can detach an instance from your Auto Scaling group. After the instance is detached, you can manage it separately from the Auto Scaling group or attach it to a different Auto Scaling group.</p><p><strong>6. Life cycle Hooks — </strong>we can add life cycle hook to our auto scaling group so that we can perform custom actions when instances launch or terminate.</p><p>a. <strong>Scale Out</strong> : (it launches one or more instances) -instances start in the <strong>PENDING STATE</strong> if we added “autoscaling:EC2_INSTANCE_LAUNCHING” hook to our auto scaling group, instance move from <strong>PENDING WAIT STATE, </strong>after completion of life cycle action, the instances enter the<strong> PENDING: PROCEED state.</strong>When the instances are fully configured, they are attached to the Auto Scaling group and they enter the Inservice state.</p><p>PENDING -&gt; {PENDING WAIT &amp; PENDING PROCEED STATE} lifecycle action added -&gt; INSERVICE STATE.</p><p>b. <strong>Scale In</strong> : (it detaches one or more instances) -instances are detached in <strong>TERMINATING STATE</strong> , if we added “autoscaling:EC2_INSTANCE_TERMINATING” hook to our auto scaling group, instance move from <strong>TERMINATING WAIT STATE, </strong>after completion of life cycle action, the instances enter the<strong> TERMINATING: PROCEED state.</strong>When the instances are fully terminated, they are detached from Auto Scaling group and they enter the <strong>Terminated</strong> state.</p><p>TERMINATING -&gt; {TERMINATING WAIT &amp; TERMINATING PROCEED STATE }life cycle action added -&gt; TERMINATED .</p><p>7. <strong>Enter and Exit Stand By :- </strong>put any instance that is in an <strong>INSERVICE</strong> state into a <strong>STANDBY</strong> state. This enables you to remove the instance from service, troubleshoot or make changes to it, and then <strong>put it back into service.</strong></p><p>Instances in a <strong>STANDBY</strong> state continue to be managed by the Auto Scaling group. However, they are <strong>not an active part of your application until you put them back into service.</strong></p><p><strong>Auto Scaling Limits:</strong></p><p><strong>Default Limits</strong></p><ul><li>Launch configurations per Region: 200</li><li>Auto Scaling groups per Region: 200</li></ul><p><strong>Auto Scaling Group Limits</strong></p><ul><li>Scaling policies per Auto Scaling group: 50</li><li>Scheduled actions per Auto Scaling group: 125</li><li>Lifecycle hooks per Auto Scaling group: 50</li><li>SNS topics per Auto Scaling group: 10</li><li>Classic Load Balancers per Auto Scaling group: 50</li><li>Target groups per Auto Scaling group: 50</li></ul><p><strong>Scaling Policy Limits</strong></p><ul><li>Step adjustments per scaling policy: 20</li></ul><p>Yes, we have learnt something about basics in autoscaling and needs of Auto scaling, let see how to do set up , templates configuration in next part.</p><p>In case of queries please feel free to comment!.</p><p>Thanks for the time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfda66c7b979" width="1" height="1" alt=""><hr><p><a href="https://medium.com/everythingatonce/auto-scaling-amazon-ec2-and-how-day-3-cfda66c7b979">Auto Scaling Amazon EC2 and How ? — Day 3</a> was originally published in <a href="https://medium.com/everythingatonce">Everything at Once</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything that the AWS storage devices can do! — Day 3]]></title>
            <link>https://medium.com/everythingatonce/everything-that-the-aws-storage-devices-can-do-day-3-cd92ab4f1ac7?source=rss-13d9e742dd73------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd92ab4f1ac7</guid>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws-storage]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[Prabhu Rajendran]]></dc:creator>
            <pubDate>Tue, 27 Aug 2019 20:22:58 GMT</pubDate>
            <atom:updated>2019-08-28T09:58:25.433Z</atom:updated>
            <content:encoded><![CDATA[<h3>Everything that the AWS storage devices can do! — Day 2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bAoSLSEtRsQa_1nylama6w.png" /></figure><p>Storage Devices that available</p><ol><li>Simple Storage Service (S3):</li><li>Glacier</li><li>Elastic Block Store</li><li>Elastic File System</li><li>Storage Gateway</li><li>Snowball.</li></ol><blockquote><strong>Simple Storage Service :</strong> ?</blockquote><ol><li>Designed to <strong>store and access any kind of data</strong> over internet.</li><li>Server-less Service</li><li>Yes! Obviously file server ,an operating system, a hard disk but we don’t need to be concerned about that at all. We just <strong>simply need to create this thing called a bucket and then we upload objects to that bucket.</strong></li><li>Bucket grows as we add objects to it and the <strong>size of bucket is theoretically unlimited.</strong></li></ol><blockquote><strong>Glacier :?</strong></blockquote><ol><li>Mainly used for long term archiving data.</li><li>Server-less Service</li><li>Same like S3 but it not as readily accessible.(so it should only be used for content that is to be archived).</li></ol><p>we can also set up automatically migrate old date to glacier from s3 forlong term archiving data.</p><blockquote><strong>Elastic Block Store (EBS):?</strong></blockquote><ol><li>low latency block storage and it’s <strong>specifically for attaching to servers</strong> that are launched with the Amazon ec2 service.(similar to attaching hard drive to our computer)</li></ol><blockquote><strong>Elastic File System (EFS):?</strong></blockquote><p><strong>Network attached storage and it’s specifically for Amazon EC2 servers. </strong>Because it is network attached store, this allows multiple servers to access one data source in a similar way to Network access storage(NAS) on your network at home can be accessed by multiple computers on that network.</p><blockquote><strong>AWS Storage Gateway :?</strong></blockquote><p>Enables <strong>hybrid storage between on-premise environments and the AWS cloud</strong>. It provides a low latency performance by caching frequently used data on-premises while storing the less frequently data in Amazon Cloud storage services.</p><blockquote><strong>Snow Ball :?</strong></blockquote><ol><li>Device is portable, Petabyte scalable ,Data Storage device that can be used to migrate data and large amount data from <strong>on-premise environments over the AWS cloud.</strong></li><li>Simply download data to Snowball device and send it off to AWS who will then upload that data to an AWS storage services.</li></ol><p>…. Now we can <strong>create a VPC inside</strong> (VPC — own private space within AWS cloud and that is an impenetrable fortress against attack and no one will enter to own private space without until allowing that to happen.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CtAuPZucyuoSr_e-RZ0cQQ.png" /></figure><p>Now , just say we launched two servers in our VPC now we want these servers to have data and to access data (in normal environment we add hard drive to that server , so in the same way we can attach an AWS EBS storage device to server).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gZZXVFBOD_vcoFTt86u_Iw.png" /></figure><p>Now , that data to be available to both of those servers? So here we’ve only got two volumes , what if we want to have only one volume data that to available both servers? — (Purchase a <strong>NAS network</strong> attached storage device. You would attach it to your network you would set up your operating system) — Same way we can do in Amazon —<strong> <em>elastic file system is network attached storage and so that with a mount target can enable multiple servers.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*topbi4Wa-_cfsatBjEGevw.png" /></figure><p>Now what if we don’t want to worry about mount targets and block devices and all this sort of stuff? We just want somewhere we can upload objects to in a similar way do we do with Google Drive or something like that and we also want to have an automated solution that over time migrates that data over to something more low cost and more long term for archiving now — That is where <strong>Amazon S3 comes in and so we can use Amazon s3 to create a bucket.</strong></p><p><strong><em>Amazon S3 </em></strong>: ?</p><ol><li>Create a Bucket (unique name).</li><li>Store objects to that bucket, delete objects do what ever we need.(simple key value store)</li><li>And also we can set up a life cycle rule on that bucket so that over a period time as object age , they can be migrated to Glacier Vault for long term archiving.</li><li>But the advantages of s3 will be located in the AWS cloud not in VPC (our private space) — were vpc endpoint comes in.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XnM5VJLZYXjw_V1oWxjfsw.png" /></figure><p>5. Unique namespace (name will be unique globally) — bucket contain DNS address and bucket must contain unique name to generate unique DNS address.</p><p>Ex of S3 URL : <a href="https://everythingatonce.s3.ap-south-1.amazonaws.com/Screen+Shot+2019-08-28+at+1.21.44+AM.png">https://ap-south-1.amazonaws.com/</a><a href="https://everythingatonce.s3.ap-south-1.amazonaws.com/Screen+Shot+2019-08-28+at+1.21.44+AM.png">everythingatonce</a></p><p>Advantages of S3 Buckets ?</p><ol><li>Buckets (Unique Name)</li><li>Storing data in buckets — Each object can be <strong>stored and retrieved by using a unique developer assigned-key</strong> — Each <strong>object can contain upto 5 TB of data.</strong></li><li>Download data — at any time where you want.</li><li>Permissions — also grant or deny access to others who want to access S3 Bucket.</li><li>Standard Interfaces : SOAP/Rest can work with any development tool kit.</li><li>Security : By Protecting unauthorized users from accessing data.</li></ol><p><strong>Why S3 is simple key value store ?</strong></p><ol><li><strong>Key — </strong>name of the object .For example, Availability.xml, airport.xlsx, etc. You can use the key to retrieve the object.</li><li><strong>Value</strong> — Data which made up of sequence of bytes.</li><li><strong>Version ID </strong>— Unique identifies the object (generated by s3 when we upload to s3 bucket).</li><li><strong>Metadata: </strong>Meta info about data that we are storing.</li><li><strong>Sub resources</strong>: mechanism is used to store object-specific information.</li><li><strong>Access Control Information:</strong> we can put the permissions individually on objects we added in bucket.</li></ol><h3>S3 Concepts:</h3><ul><li><strong>Buckets </strong>— (container for storing the objects)</li><li><strong>Objects</strong> -(entities which are stored in an S3 bucket)</li><li><strong>Keys</strong> (object in a bucket is associated with one key — unique identifier for an object)</li><li><strong>Regions </strong>— geographical region in which you want to store the buckets that you have created ,latency, minimize costs or address regulatory requirements.</li><li>Data Consistency Model (2 types)</li></ul><p>a. <strong>Read-after-write consistency for PUTS of new objects.</strong> (Across multiple servers to achieve high availability for immediately available to read the object) — never take time to update.(list keys immediately within bucket)</p><p><strong>b.Eventual consistency for overwrite PUTS and DELETES (</strong>the changes are reflected eventually, and they are not available immediately.Until the change is fully propagated, the S3 might return the list of the deleted key or data).</p><p>Yes!, we have know about little in S3 bucket let see another part how to create S3 Buckets.</p><p>In case of queries please feel free to comment!.</p><p>Thanks for the time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd92ab4f1ac7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/everythingatonce/everything-that-the-aws-storage-devices-can-do-day-3-cd92ab4f1ac7">Everything that the AWS storage devices can do! — Day 3</a> was originally published in <a href="https://medium.com/everythingatonce">Everything at Once</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>