<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abdelrahman Ahmed on Medium]]></title>
        <description><![CDATA[Stories by Abdelrahman Ahmed on Medium]]></description>
        <link>https://medium.com/@abdelrhmanahmed131?source=rss-3227207ccd6b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TEkbUlDirql9CbB5NH6NUQ.jpeg</url>
            <title>Stories by Abdelrahman Ahmed on Medium</title>
            <link>https://medium.com/@abdelrhmanahmed131?source=rss-3227207ccd6b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 07:55:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abdelrhmanahmed131/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From AWS Kinesis to Apache Kafka: Building a Real-Time Streaming Bridge]]></title>
            <link>https://medium.com/@abdelrhmanahmed131/from-aws-kinesis-to-apache-kafka-building-a-real-time-streaming-bridge-a2ca8bd4e991?source=rss-3227207ccd6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2ca8bd4e991</guid>
            <category><![CDATA[amazon-kinesis]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Abdelrahman Ahmed]]></dc:creator>
            <pubDate>Tue, 25 Nov 2025 13:36:11 GMT</pubDate>
            <atom:updated>2025-11-26T09:43:34.912Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQ4Ar3PcF0rzEnSYgxt3Vw.png" /></figure><h4>Introduction:</h4><p>In one of my recent projects, I needed to replicate data from an AWS Kinesis stream into an Apache Kafka topic. The goal was to allow downstream systems to consume the same events from Kafka that were originally being produced to Kinesis.</p><p>The core question here is: <strong>How can we replicate data from AWS Kinesis stream to a Kafka topic in real-time and with minimal latency, and with the highest reliability?</strong></p><p>I explored different solutions, including <strong>Kafka Connect</strong>, <strong>MirrorMaker</strong>, and other integration tools. However, most of them came with complex configurations and limitations that didn’t fit our infrastructure setup.</p><p>So, I decided to build a custom Kinesis-to-Kafka <strong>bridge</strong> that will be flexible, lightweight, highly configurable, and perfectly plug into our infrastructure setup.</p><p>If you’re new to<strong> Amazon Kinesis</strong>, here’s a quick overview of it (feel free to skip this section if you’re already familiar):</p><p><strong>Amazon Kinesis </strong>is a group of streaming services within Amazon Web Services (AWS) created to stream real-time data at scale. It contains different services like:</p><ul><li>Amazon Kinesis Data Streams</li><li>Amazon Kinesis Data Firehose</li><li>Amazon Kinesis Video Streams</li><li>Amazon Managed Service for Apache Flink</li></ul><blockquote>In this article, we will focus on <strong>Kinesis Data Streams</strong>, which is a serverless (fully managed by AWS) streaming service designed to handle high-throughput, low-latency streaming use-cases like events, logs, and clickstreams.</blockquote><p>If you’re also new to<strong> Apache Kafka</strong>, here’s a quick overview(feel free to skip this section if you’re already familiar):</p><p><strong>Apache Kafka</strong> is a distributed event streaming platform that handles large volumes of real-time data. It provides durability, scalability, and strong ordering guarantees. It is usually used to build data pipelines and streaming applications at different scales, and it is very dominant in event-driven architectures.</p><h4>The Replicator Schema</h4><p>The replicator consists of three core stages, as shown in the following diagram:</p><pre>┌──────────────────────────┐<br>│     Kinesis Consumer     │  ← Consumes records from AWS Kinesis stream<br>└────────────┬─────────────┘<br>             │<br>             ▼<br>┌──────────────────────────┐<br>│     Data Processing      │  ← Applies business logic to transform/validate/parse the records<br>└────────────┬─────────────┘<br>             │<br>             ▼<br>┌──────────────────────────┐<br>│    Producing to Kafka    │  ← Produces processed events to Kafka topic/topics<br>└──────────────────────────┘<br></pre><p><strong>Kinesis Consumer:</strong> Consuming data from a Kinesis stream requires more than just reading records; it requires manual management for offsets since Kinesis doesn’t provide a built-in offset tracking mechanism for consumers. Instead, consumers are responsible for tracking their own progress, typically by storing checkpoint data in a DynamoDB table, often referred to as a <strong>lease table</strong>.</p><p>This approach, called <strong>leasing</strong>, allows multiple consumers to coordinate shard (the basic unit of Kinesis stream) access and avoid processing the same records. You can implement this logic manually (very complex) or use a ready-to-use library such as Amazon <a href="https://github.com/awslabs/amazon-kinesis-client"><strong>Kinesis Client Library (KCL)</strong></a>, which handles the lease management, DynamoDB tables creation, and offset tracking out of the box.</p><p>In my case, I used KCL for Python <a href="https://github.com/awslabs/amazon-kinesis-client-python">(<strong>KCLPY</strong>)</a> to consume from Kinesis so that I can avoid re-implementing low-level coordination and state management logic.</p><p><strong>Data Processing: </strong>This stage is where your core business logic lives. After data is fetched from Kinesis but before it’s forwarded to Kafka, you may need to:</p><ul><li>Adding any required metadata to records</li><li>Validate or filter the records</li><li>Transform the records to meet Kafka consumer expectations</li></ul><p>If no transformation is needed, the data can be streamed directly to Kafka.</p><p><strong>Producing to Kafka: </strong>This is the final stage, in which records are being published to the designated Kafka topic.</p><p>At the implementation level, this is typically achieved by using one of the Kafka SDKs. In my case, I used the <strong>confluent_kafka‎</strong> client library for Python.</p><p>After this stage, the records are serialized and published to Kafka, which allows the downstream systems to consume the data in real-time.</p><h4>Tech Stack and Project Structure</h4><p>This project was developed using <strong>Python</strong>, <strong>kclpy</strong>, <strong>confluent_kafka</strong>, and <strong>Docker</strong>.</p><p>Below is an overview of the project structure:</p><pre><br>kinesis-to-kafka-replicator/<br>├── kinesis_replicator/<br>│   ├── __init__.py<br>│   └── record_processor.py<br>│<br>├── config/<br>│   ├── kcl.properties.template<br>│   └── replicator_configs.template<br>│<br>├── amazon_kclpy/<br>│   └── jars/<br>│<br>├── run.sh<br>├── Dockerfile<br>├── requirements.txt<br>└── README.md</pre><p><strong>Configuration Breakdown: </strong>The configuration is separated into two parts:</p><ul><li><strong>KCLPY Configuration: </strong>Located in<strong> </strong><em>kcl.properties.template</em> and contains all the required settings for the Kinesis client library (KCL), such as <strong>streamName</strong>, <strong>initialPositionInStream</strong>, and the AWS authentication parameters. A more detailed list can be found <a href="https://github.com/awslabs/amazon-kinesis-client-python/blob/master/samples/sample.properties">here</a>.</li><li><strong>Replicator Configuration:</strong> Located in<strong> </strong><em>replicator_configs.template</em> and includes the kafka_specific settings, such as <strong>bootstrap_servers</strong>, <strong>kafka_topic</strong>, and <strong>client_id, </strong>along with any global parameters required for tuning the replicator, such as <strong>sleep_seconds</strong>.</li></ul><blockquote>All Python dependencies, such as <strong>boto3</strong>, <strong>confluent_kafka</strong>, and <strong>amazon_kclpy</strong>, are listed in <strong>requirements.txt</strong>.</blockquote><p><strong>Core Logic: <em>record_processor.py</em></strong></p><p>This is the heart code base of the replicator.</p><p>The <em>RecordProcessor</em><strong> </strong>class is responsible for:</p><ul><li>Consuming/Receiving the records from the Kinesis stream</li><li>Applying any business logic or transformations</li><li>Publishing the resulting data to Kafka</li><li>Handling retries, checkpointing, and logging</li></ul><p>A more detailed sample can be found <a href="https://github.com/awslabs/amazon-kinesis-client-python/blob/master/samples/sample_kclpy_app.py"><strong>here</strong></a></p><p>Following a simplified breakdown of what’s happening:</p><p>➤ Processing Kinesis Records</p><pre># Example for KCL process_records method with leasing functionality (checkpointing)<br><br>    def process_records(self, process_records_input):<br>        &quot;&quot;&quot;<br>        Called by a KCLProcess with a list of records to be processed and a checkpointer which accepts sequence numbers<br>        from the records to indicate where in the stream to checkpoint.<br><br>        :param amazon_kclpy.messages.ProcessRecordsInput process_records_input: the records, and metadata about the<br>            records.<br>        &quot;&quot;&quot;<br>        try:<br>            for record in process_records_input.records:<br>                data = record.binary_data<br>                seq = int(record.sequence_number)<br>                sub_seq = record.sub_sequence_number<br>                key = record.partition_key<br>                self.process_record(data, key, seq, sub_seq) # Business logic is being imlemented in this methos<br>                if self.should_update_sequence(seq, sub_seq):<br>                    self._largest_seq = (seq, sub_seq)<br><br>            # Checkpoints every N seconds<br>            if time.time() - self._last_checkpoint_time &gt; self._CHECKPOINT_FREQ_SECONDS:<br>                self.checkpoint(process_records_input.checkpointer, str(self._largest_seq[0]), self._largest_seq[1])<br>                self._last_checkpoint_time = time.time()<br><br>        except Exception as e:<br>            self.log(&quot;Encountered an exception while processing records. Exception was {e}\n&quot;.format(e=e))</pre><p>➤ Producing to Kafka</p><pre># A code snippet from the process_record method mentioned above that shows how to produce to Kafka<br><br>            try:<br>                self.kafka_producer.produce(<br>                    topic=self.kafka_topic,<br>                    value=data,<br>                    key=partition_key,<br>                    callback=callback<br>                )<br>            except BufferError:<br>                logger.warning(f&quot;Kafka producer queue full, draining and retrying for record {sequence_number}&quot;)<br>                self.kafka_producer.poll(1.0)<br>                self.kafka_producer.produce(<br>                    topic=self.kafka_topic,<br>                    value=data,<br>                    key=partition_key,<br>                    callback=callback<br>                )</pre><p>This ensures the record is safely published to Kafka, with retry logic in case of temporary backpressure.</p><blockquote><strong>amazon_kclpy/jars/: </strong>This folder contains the Java dependencies used by KCLPY. These are required to enable the bridge between Python and the underlying KCL lease coordination. <br>Jar files can be generated by following the instructions written <a href="https://github.com/awslabs/amazon-kinesis-client-python">here</a>.</blockquote><h4><strong>Deployment and Startup</strong></h4><p>To ensure portability and seamless deployment, the replicator is contranized using Docker. Here is a breakdown of how to build and run it.</p><p><strong>Dockerfile:</strong> Defines how to build a container image for the replicator that includes the required Python environment, libraries, and the Kinesis Client Library (KCL) dependencies.</p><p>➤ Sample Dockerfile</p><pre>FROM python:3.9-slim<br><br># Install system dependencies<br>RUN apt-get update &amp;&amp; apt-get install -y openjdk-11-jre curl unzip &amp;&amp; \<br>    rm -rf /var/lib/apt/lists/*<br><br># Set workdir and copy code<br>WORKDIR /app<br>COPY . /app<br><br># Install Python requirements<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Set entrypoint<br>ENTRYPOINT [&quot;./run.sh&quot;]</pre><p><strong>run.sh: </strong>The <strong><em>run.sh</em></strong> script acts as the <strong><em>entrypoint</em></strong> to launch the replicator. First, the script renders the configuration template files into typical configuration files, then it bootstraps the KCL MultiDaemon with that configuration.</p><p>The <strong><em>run.sh</em></strong> script includes:</p><ul><li>Template rendering for the config files using the <strong><em>envsubst</em></strong> command</li><li>Launching the KCL Python daemon with the provided configuration</li></ul><p>➤ Sample run.sh</p><pre>#!/bin/bash<br>set -e<br><br># Rendering the configuration templates<br>envsubst &lt; config/replicator_configs.template &gt; config/replicator_configs.json<br>envsubst &lt; config/kcl.properties.template &gt; config/kcl.properties<br><br># Start KCL daemon<br>java -cp &quot;amazon_kclpy/jars/*&quot; software.amazon.kinesis.multilang.MultiLangDaemon config/kcl.properties</pre><h4>Build and Run</h4><p><strong>Pre-requisites:</strong></p><ul><li>Make sure that your Kinesis stream has data</li><li>Make sure that your Kafka topic exists and is ready to receive data</li><li>The number of shards in the stream must equal the number of partitions in the Kafka topic to help in maintaining global ordering</li><li>The KCL requires some DynamoDB tables to handle Kinesis offsets; you can pre-create them or give the KCL permissions to do that based on the AWS IAM role used.</li></ul><p><strong>1) Build the Docker image</strong></p><pre># Go to the Dockerfile path and run the following command<br><br>docker build -t kinesis-to-kafka-replicator .</pre><p><strong>2) Prepare the required environment variables</strong></p><pre># Example for KCL configuration<br>KCL_STREAM_NAME=kafka-kinesis-replicator-test<br>KCL_APPLICATION_NAME=replicator-app-name<br>KCL_REGION=eu-west-1<br>KCL_INITIAL_POSITION=LATEST<br>KCL_LOG_LEVEL=DEBUG<br>KCL_AWS_CREDENTIALS_PROVIDER=DefaultCredentialsProvider<br><br># Example for Kafka and global configuration<br>SLEEP_SECONDS=5<br>CHECKPOINT_RETRIES=5<br>CHECKPOINT_FREQ_SECONDS=30<br>KAFKA_TOPIC=kinesis_kafka_replication_test<br>KAFKA_BOOTSTRAP_SERVERS=bootstrap-server:9094<br>KAFKA_SASL_USERNAME=kafka-user<br>KAFKA_SASL_PASSWORD=kafka-password<br>KAFKA_SASL_MECHANISM=SCRAM-SHA-512<br>KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT<br>KAFKA_CLIENT_ID=replicator-client<br>NUMBER_OF_SHARDS=4<br>AWS_ACCESS_KEY_ID=example-key-id<br>AWS_SECRET_ACCESS_KEY=example-secret-key<br>AWS_SESSION_TOKEN=example-token</pre><p><strong>Note:</strong> The above environment variables are saved into a <em>kinesis_kafka_replicator.env</em> file and will be used in the following Docker run command to be passed to the container.</p><p><strong>3) Run the container</strong></p><pre>docker run --env-file kinesis_kafka_replicator.env -it kinesis-to-kafka-replicator</pre><h4>Verification and Observability</h4><p>Once the replicator container is up and running, we can observe some logs to make sure that the app inside the container is working properly.</p><p><strong>1) The application started successfully: </strong>The highlighted log indicates that the replicator started successfully, and it is processing the correct stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWM4TUHNiuG1T93kt-20DA.png" /></figure><p><strong>2) Monitor the DDB tables creation: </strong>The following images confirm that the replicator can recognize the pre-created tables and will not create them again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*37P15U4tXA1QmdqUxsqJSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E8XqgLl7MmVwJQ4p5VBJug.png" /></figure><p><strong>3) KCL leader election: </strong>The replicator elected a worker leader and assigned it to lead the consumption from a shard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kzhvru3Z5lc0ahkBjXfV3g.png" /></figure><p><strong>4) KCL Kinesis consumer: </strong>The replicator started to consume the records from the Kinesis stream and push them to Kafka.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zV77fOTi84prWAH-qcKdGg.png" /></figure><p><strong>5) Observing the Kafka topic: </strong>By observing the Kafka topic on Kafka Ui, it can be easily noticed that the messages started to show up there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F60zGlLcWJHDZtSL6PXUdA.png" /></figure><p>All of this confirms that our replicator is working properly and can mirror the data between the two platforms successfully.</p><h4>Conclusion</h4><p>This approach helped in real-time streaming between Kinesis and Kafka without over-engineering. In this guide, we built a lightweight replicator using Python, KCL, and Kafka libraries to bridge data between the two systems in real-time, ensuring reliability and configurability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2ca8bd4e991" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Longhorn — Distributed Block Storage for K8s]]></title>
            <link>https://medium.com/@abdelrhmanahmed131/longhorn-distributed-block-storage-for-k8s-2ea11df400d1?source=rss-3227207ccd6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ea11df400d1</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[longhorns]]></category>
            <dc:creator><![CDATA[Abdelrahman Ahmed]]></dc:creator>
            <pubDate>Fri, 17 May 2024 23:34:56 GMT</pubDate>
            <atom:updated>2024-05-17T23:34:56.480Z</atom:updated>
            <content:encoded><![CDATA[<h3>Longhorn — Distributed Block Storage for K8s</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LqJ_QQamY_gc7cmOK0wvoA.jpeg" /><figcaption>Figure 1.1: Kubernetes pods using Loghorn Volumes</figcaption></figure><h3>What is Longhorn?</h3><blockquote><strong><em>Longhorn</em></strong><em> </em>is a cloud-native distributed block storage system for Kubernetes that provides persistent storage for stateful applications.</blockquote><blockquote><strong><em>Longhorn</em></strong><em> </em>is designed to be easy to use, scalable, reliable, and efficient.</blockquote><blockquote><strong><em>It </em></strong>uses container-native storage and runs as a container on each worker node in Kubernetes cluster.</blockquote><blockquote><strong><em>Longhorn</em></strong><em> </em>uses replication to make copies of data to ensure high availability and data redundancy. When a Longhorn volume is created, the user specifies the number of replicas that should be created for the volume. Each replica is a complete copy of the volume data and is stored on a different physical disk on a different node within the Kubernetes cluster.</blockquote><blockquote><strong><em>Longhorn</em></strong><em> </em>is open source and is maintained by Rancher Labs, the same company that developed the RKE2 cluster for Kubernetes.</blockquote><h4>Why do we need persistent storage in our K8s cluster?</h4><p><strong><em>In </em></strong>containerized applications, data is usually stored in a storage system that is separate from the container itself. This allows the container to be easily destroyed or replaced without losing any important data. Storing persistent data means that the data is saved in a way that it is not lost when the container is stopped, restarted, or deleted. This is achieved by using persistent storage solutions, such as cloud-native distributed block storage systems, to store the data in a way that it can be accessed by the container when needed.</p><p><strong><em>Data</em></strong><em> </em>could be application data, configuration files, user data, databases, and other types of data that need to be preserved and accessed over time.</p><h3>How does Longhorn work? (Design &amp; Architecture)</h3><p>There are three main components in Longhorn:</p><ul><li>Longhorn Manager</li><li>Longhorn Engine</li><li>Longhorn UI</li></ul><h4><strong><em>Longhorn Manager</em></strong></h4><p>The Longhorn Manager Pod runs on each node in the Longhorn cluster as a Kubernetes DaemonSet. It is responsible for creating and managing volumes in the Kubernetes cluster and handles the API calls from the UI or the volume plugins for Kubernetes.</p><p>The Longhorn Manager communicates with the Kubernetes API server to create a new Longhorn volume CRD. Then the Longhorn Manager watches the API server response, and when it sees that the Kubernetes API server created a new Longhorn volume CRD, the Longhorn Manager creates a new volume.</p><h4><strong><em>Longhorn Engine</em></strong></h4><p><em>When</em> the Longhorn Manager is asked to create a volume, it creates a Longhorn Engine instance on the node the volume is attached to, and it creates a replica on each node where a replica will be placed. Replicas should be placed on separate hosts to ensure maximum availability.</p><p><em>Longhorn </em>creates a dedicated engine for each volume on each node in the Longhorn cluster.</p><p><em>An </em>engine is a component responsible for handling the I/O operations for a Longhorn volume. When a volume is created, Longhorn creates a dedicated engine on each node where the volume is scheduled to run. This ensures that the data for the volume is stored locally on the node where the volume is running.</p><p><em>If </em>the node that hosts the Longhorn Engine for a volume crashes or becomes unavailable for any reason, Longhorn will detect the failure and automatically reschedule the Engine on another available node. This process is known as<em> </em><strong><em>automatic node recovery</em></strong>.</p><p><em>The </em>Engine is not a Kubernetes object like a Pod or a Deployment. Rather, it is a Longhorn-specific process that runs as a binary executable on the host operating system.</p><p>The following figure illustrates how Longhorn works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VyRVeac2fVFipBX4A3ctMg.png" /><figcaption>Figure 1.2: How Longhorn works — image source (<a href="https://longhorn.io/img/diagrams/architecture/how-longhorn-works.svg">longhorn.io</a>)</figcaption></figure><ul><li>There are three instances of Longhorn volumes.</li><li>Each volume has a dedicated controller, which is called the Longhorn Engine, and runs as a Linux process.</li><li>Each Longhorn volume has two replicas, and each replica is a Linux process.</li><li>The arrows in the figure indicate the read/write data flow between the volume, controller instance, replica instances, and disks.</li><li>By creating a separate Longhorn Engine for each volume, if one controller fails, the function of other volumes is not impacted.</li></ul><h4><strong><em>Longhorn UI</em></strong></h4><p><em>The </em>Longhorn UI is deployed as a Kubernetes deployment by Longhorn Manager with a single replica by default, which means that it runs as a pod on a single node.</p><p><em>The </em>Longhorn UI interacts with the Longhorn Manager through the Longhorn API. Through the Longhorn UI, you can manage snapshots, backups, nodes, and disks.</p><p><em>Besides</em>, the space usage of the cluster worker nodes is collected and illustrated by the Longhorn UI.</p><p>The following figure illustrates a view from the Longhorn UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4HkXi3qtoTq1p602MnQ5tQ.png" /><figcaption>Figure 1.3: Longhorn UI</figcaption></figure><p>Let’s delve into the meaning of each terminology in the view:</p><ul><li><strong>Schedulable</strong>: The actual space that can be used for Longhorn volume scheduling.</li><li><strong>Reserved</strong>: The space reserved for other applications and systems.</li><li><strong>Used</strong>: The actual space that has been used by Longhorn, system, and other applications.</li><li><strong>Disabled</strong>: The total space of the disks/nodes on which Longhorn volumes are not allowed for scheduling.</li></ul><h3>Longhorn installation (pre-requisites and requirements)</h3><h4>Number of nodes</h4><p>Longhorn can be installed on a Kubernetes cluster with as few as one node, but it is recommended to have at least <strong>three</strong> nodes for data redundancy and high availability.</p><h4>Each node in K8s cluster must fulfill the following requirements:</h4><ul><li>A container runtime compatible with Kubernetes <strong>(Docker v1.13+, containerd v1.3.7)</strong></li><li>Kubernetes <strong>version</strong> &gt; v1.21</li><li><strong>open-iscsi</strong> is installed, and the iscsid daemon is running on all the nodes.</li></ul><blockquote><strong>Explanation:<br></strong>Longhorn relies on <strong>iscsiadm</strong> which is the host to provide persistent volumes to Kubernetes. refer to this link for installing <a href="https://longhorn.io/docs/1.6.1/deploy/install/#installing-open-iscsi"><strong>open-iscsi</strong></a><strong>.</strong></blockquote><blockquote>Open-iSCSI is an implementation of the iSCSI (Internet Small Computer System Interface) protocol for Linux.</blockquote><blockquote>It enables communication between iSCSI initiators and targets over an IP network.</blockquote><blockquote>It allows remote block-level access to storage devices such as hard drives, SSDs, or storage arrays as if they were directly attached to the host system.</blockquote><ul><li><strong>RWX</strong> support requires that each node has an <strong>NFSv4</strong> client installed.</li></ul><blockquote><strong>Explanation:</strong></blockquote><blockquote>“<strong>RWX”</strong> stands for <strong>“Read-Write-Many”</strong>, which refers to a mode of access to a persistent volume where multiple pods can simultaneously read and write data.</blockquote><blockquote>To support <strong>RWX</strong> access mode for a persistent volume, Longhorn uses <strong>NFS</strong> (Network File System) protocol, specifically <strong>NFSv4</strong>.</blockquote><blockquote><strong>NFSv4</strong> is a distributed file system protocol that allows clients to access and share data over a network.</blockquote><blockquote>For each node in the Kubernetes cluster that will access the <strong>RWX</strong> persistent volume, an <strong>NFSv4</strong> client must be installed. This is because the client software is responsible for communicating with the <strong>NFS</strong> server to access the shared data.</blockquote><blockquote>Refer to this link for installing <a href="https://longhorn.io/docs/1.6.1/deploy/install/#installing-nfsv4-client"><strong>NFSv4</strong></a>.</blockquote><ul><li>The host filesystem supports the <strong>file extents</strong> feature to store the data. Currently, we support: <strong><em>ext4</em></strong> and <strong><em>XFS</em></strong>.</li></ul><blockquote><strong>Explanation:</strong></blockquote><blockquote><strong>“file extents</strong>” refers to a feature of these file systems where file data can be stored in contiguous blocks on disk instead of separate files, allowing for more efficient use of disk space and faster access to data.</blockquote><ul><li>These Linux command line utilities must be installed: bash, curl, findmnt, grep, awk, blkid, lsblk</li></ul><blockquote><strong>bash:</strong> It is a shell that provides a command-line interface for interacting with the operating system. It is the default shell in most Linux distributions.</blockquote><blockquote><strong>curl:</strong> It is a command-line tool for transferring data from or to a server, using one of the supported protocols such as HTTP, FTP, SCP, etc.</blockquote><blockquote><strong>findmnt:</strong> It is a utility to search and list all mounted filesystems or search for a specific mount point.</blockquote><blockquote><strong>grep:</strong> It is a command-line tool for searching patterns in text files.</blockquote><blockquote><strong>awk:</strong> It is a programming language and utility for processing and manipulating text files, mainly used for data extraction and reporting.</blockquote><blockquote><strong>blkid:</strong> It is a utility for displaying information about the available block devices on a system, including their UUID, filesystem type, and label.</blockquote><blockquote><strong>lsblk: </strong>It is a utility for listing information about all available or the specified block devices on a system, including their size, mount point, and other details.</blockquote><ul><li>Mount propagation must be enabled.</li></ul><blockquote><strong>Explanation:</strong></blockquote><blockquote><strong>“Mount Propagation</strong>” is a feature that allows the sharing of volumes mounted by one container with other containers in the same pod or even in the other pods in the same node.</blockquote><ul><li>The Longhorn workloads must be able to run as root for Longhorn to be deployed and operated properly.</li></ul><blockquote><strong>Explanation:</strong></blockquote><blockquote>For Longhorn to function properly, the Longhorn Manager and Longhorn instances running on each node need to be able to perform certain operations that require elevated privileges. These operations include managing block devices, creating and mounting file systems, and managing storage pools. By default, Kubernetes pods run as a non-root user for security reasons. However, Longhorn requires that the Longhorn workloads be able to run as root to perform these operations.</blockquote><h3>Min recommended hardware</h3><ul><li>3 nodes</li><li>4 vCPUs per node</li><li>4 GiB per node</li><li>SSD/NVMe or similar performance block device on the node for storage (recommended)</li><li>HDD/Spinning Disk or similar performance block device on the node for storage (verified)</li><li>500/250 max IOPS per volume (1 MiB I/O)</li><li>500/250 max throughput per volume (MiB/s)</li></ul><h3>OS</h3><ul><li><strong>Ubuntu:</strong> Versions (20.04, 22.04)</li><li><strong>SLES:</strong> Version (15 SP4)</li><li><strong>SLE Micro:</strong> Version (5.3)</li><li><strong>CentOS:</strong> Version (8.4)</li><li><strong>RHEL:</strong> Version (8.6)</li><li><strong>Oracle Linux:</strong> Version (8.6)</li><li><strong>Rocky Linux:</strong> Version (8.6)</li></ul><h3>Longhorn Capabilities, Limitations, and Disadvantages</h3><h4>Capabilities</h4><ol><li><strong>High Availability</strong>: Longhorn is designed to provide high availability of your persistent data by replicating volumes across multiple nodes in the cluster.</li><li><strong>Snapshots and Backups</strong>: Every time Longhorn detects a replica is down, it will automatically take a (system) snapshot and start rebuilding it on another node.</li><li><strong>Disaster Recovery</strong>: Longhorn is designed to recover from disasters by rebuilding volumes and restoring data from backups.</li><li><strong>Replication</strong>: Longhorn provides replication of volumes across different clusters, enabling you to create disaster recovery solutions across multiple locations.</li><li><strong>Dynamic Provisioning</strong>: Longhorn supports dynamic provisioning of volumes in Kubernetes, allowing you to create and manage volumes using Kubernetes APIs.</li><li><strong>Volume Expansion</strong>: Longhorn supports online expansion of volumes, enabling you to increase the size of your volumes without downtime.</li><li><strong>Monitoring and Metrics</strong>: Longhorn provides detailed metrics and monitoring information about the health and performance of your volumes, making it easy to troubleshoot and optimize your storage infrastructure.</li><li><strong>Web-based UI</strong>: Longhorn provides a web-based user interface that makes it easy to manage and monitor your volumes and storage infrastructure.</li></ol><h4>Limitations &amp; Disadvantages</h4><ol><li>Version-to-version upgrades are a complex process.</li><li>Longhorn doesn’t support downgrading from a higher version to a lower version.</li><li>Longhorn Block Storage speed is slightly slower than the local disk storage speed.</li></ol><h3>Longhorn Installation</h3><p>Longhorn can be installed with<strong> Helm, kubectl, </strong>and Rancher Apps &amp; Marketplace</p><p>Longhorn is 100% open-source software. Project source code is spread across a number of repositories:</p><ol><li><a href="https://github.com/longhorn/longhorn-engine"><strong>Longhorn Engine</strong></a> — Core controller/replica logic.</li><li><a href="https://github.com/longhorn/longhorn-instance-manager"><strong>Longhorn Instance Manager</strong></a> — Controller/replica instance lifecycle management.</li><li><a href="https://github.com/longhorn/longhorn-share-manager"><strong>Longhorn Share Manager</strong></a> — NFS provisioner that exposes Longhorn volumes as ReadWriteMany volumes.</li><li><a href="https://github.com/longhorn/backing-image-manager"><strong>Backing Image Manager</strong></a> — Backing image file lifecycle management.</li><li><a href="https://github.com/longhorn/longhorn-manager"><strong>Longhorn Manager</strong></a> — Longhorn orchestration, includes CSI(Container Storage Interface) driver for Kubernetes.</li><li><a href="https://github.com/longhorn/%20longhorn-ui"><strong>Longhorn UI</strong></a> — Dashboard.</li></ol><h4>Install with Helm</h4><p><strong><em>Requirements</em></strong></p><ul><li><em>Helm v2.0+ must be installed on your workstation.</em></li></ul><p>You can check the Longhorn helm charts <a href="https://github.com/longhorn/charts">here</a>.</p><p><strong><em>The content of the helm charts:</em></strong></p><p><strong><em>templates: A </em></strong><em>folder contains the Kubernetes manifests that define the resources to be created by the chart.</em></p><p><strong><em>values.yaml:</em></strong><em> A file defines the default values for the configuration parameters used in the chart.</em></p><p><strong><em>Chart.yaml:</em></strong><em> A file contains metadata about the chart, including its name, version, description, and maintainers.</em></p><p><strong><em>Installation</em></strong></p><p><em>Add Longhorn chart repository:</em></p><pre>$ helm repo add longhorn https://charts.longhorn.io</pre><p><em>Update local Longhorn chart information from the chart repository:</em></p><pre>$ helm repo update</pre><p><em>Install Longhorn chart:</em></p><p><em>With Helm 2, the following command will create the </em><strong><em>longhorn-system </em></strong><em>namespace and install the Longhorn chart together:</em></p><pre>$ helm install longhorn longhorn/longhorn --namespace longhorn-system</pre><p><em>With Helm 3, the following commands will create the </em><strong><em>longhorn-system</em></strong><em> namespace first, then install the Longhorn chart:</em></p><pre>$ kubectl create namespace longhorn-system<br>$ helm install longhorn longhorn/longhorn --namespace longhorn-system</pre><p><em>Validate if the installation was done successfully by getting the running pods after installation, you should see Longhorn pods after running the following command:</em></p><pre>$ kubectl get pods --namespace longhorn-system</pre><h3>Creating Longhorn Volumes</h3><p><strong><em>Step_1</em></strong><em>: Create a Longhorn StorageClass</em></p><p><em>Here is how the </em><strong><em>storageclass. yaml </em></strong><em>looks like this:</em></p><pre>apiVersion: storage.k8s.io/v1<br>kind: StorageClass<br>metadata:<br>  name: longhorn<br>provisioner: driver.longhorn.io<br>allowVolumeExpansion: true</pre><pre>$ kubectl apply -f storageclass.yaml</pre><p><strong><em>Step_2</em></strong><em>: </em>Create a PersistentVolumeClaim (PVC) that references the Longhorn StorageClass</p><p><em>Here is how the </em><strong><em>longhorn-pvc. yaml </em></strong><em>looks like this:</em></p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: longhorn-vol-pvc<br>spec:<br>  storageClassName: longhorn<br>  accessModes:<br>    - ReadWriteMany<br>  resources:<br>    requests:<br>      storage: 1Gi</pre><pre>$ kubectl apply -f longhorn-pvc.yaml</pre><p><strong><em>Step_3</em></strong><em>: Create a Pod that uses Longhorn volume</em></p><p><em>Here is how the </em><strong><em>pod-with-longhorn. yaml</em></strong><em> looks like this:</em></p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: longhorn-test<br>spec:<br>  containers:<br>  - name: longhorn-test-container<br>    image: busybox<br>    command: [&quot;/bin/sh&quot;]<br>    args: [&quot;-c&quot;, &quot;while true; do echo $(date) &gt;&gt; /mnt/data/date.txt; sleep 1; done&quot;]<br>    volumeMounts:<br>    - mountPath: /mnt/data<br>      name: longhorn-vol<br>  volumes:<br>  - name: longhorn-vol<br>    persistentVolumeClaim:<br>      claimName: longhorn-vol-pvc</pre><p><em>Subsequently, a pod named </em><em>longhorn-test is launched, along with a PersistentVolumeClaim named </em><em>longhorn-volv-pvc. The </em><strong><em>PersistentVolumeClaim</em></strong><em> references the </em><strong><em>Longhorn</em></strong><em> </em><strong><em>StorageClass.</em></strong></p><p><em>The </em><strong><em>persistentVolumeClaim</em></strong><em> is mounted in the Pod as a volume.</em></p><h3><strong><em>Uninstallation</em></strong></h3><p><em>With Helm 2, to uninstall Longhorn:</em></p><pre># Patch the Longhorn settings to enable confirmation for deletion<br>$ kubectl -n longhorn-system patch -p &#39;{&quot;value&quot;: &quot;true&quot;}&#39; --type=merge lhs deleting-confirmation-flag<br><br># Uninstall Longhorn<br>$ helm delete longhorn --purge</pre><p><em>With Helm 3 to uninstall Longhorn:</em></p><pre># Patch the Longhorn settings to enable confirmation for deletion<br>kubectl -n longhorn-system patch -p &#39;{&quot;value&quot;: &quot;true&quot;}&#39; --type=merge lhs deleting-confirmation-flag<br><br># Uninstall Longhorn<br>helm uninstall longhorn -n longhorn-system<br><br># Delete the Longhorn namespace<br>kubectl delete namespace longhorn-system</pre><h3>Conclusion</h3><p>In conclusion, Longhorn is a solution for creating and managing persistent storage in Kubernetes. Its architecture, featuring components like the manager, controller, and replica instances, ensures data durability and high availability. Longhorn’s key features include distributed block storage, snapshots, backups, and volume replication, making it ideal for running stateful applications. With easy installation using Helm 2 or 3, Longhorn simplifies storage provisioning and configuration, making it a valuable tool for developers and system administrators in Kubernetes environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ea11df400d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Readiness & Liveness probes]]></title>
            <link>https://medium.com/@abdelrhmanahmed131/kubernetes-readiness-liveness-probes-9c8eead045f6?source=rss-3227207ccd6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c8eead045f6</guid>
            <category><![CDATA[kubernetes-probes]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Abdelrahman Ahmed]]></dc:creator>
            <pubDate>Fri, 26 Jan 2024 15:22:56 GMT</pubDate>
            <atom:updated>2024-01-26T15:48:19.240Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xv-vEaqu__wdtR08i3fLUg.jpeg" /><figcaption>Figure 1.1: Kubernetes readiness and liveness probes</figcaption></figure><p><strong><em>Readiness and Liveness probes are used in Kubernetes to check the health and availability of containers within a pod.</em></strong></p><h3><strong>ReadinessProbe:</strong></h3><h4>Purpose:</h4><p>The “readinessProbe” is used to determine whether the container is ready to accept incoming network traffic. It’s typically used to ensure that a container has started and is in a good state to serve requests.</p><h4>Action on Failure:</h4><p>In case the readiness probe fails, the container is temporarily removed from the service’s pool, and it won’t receive incoming requests. However, the container is not restarted.</p><h4>Example Use Case:</h4><p>At the initiation of a web application, the readiness probe verifies critical components such as established database connections, loaded configurations, and readiness of all dependencies. It ensures that the application only begins to receive traffic once all these essential elements are confirmed to be in a functional state.</p><h3><strong>LivenessProbe:</strong></h3><h4>Purpose:</h4><p>The “livenessProbe” serves the crucial role of determining the ongoing availability and health status of a container. If this probe indicates a failure, Kubernetes initiates corrective measures, often by restarting the container, to maintain the continuous availability of the application.</p><h4>Action on Failure:</h4><p>In case the liveness probe fails, Kubernetes takes responsive actions, typically involving a container restart. This ensures that the application within the container is restored to a healthy state.</p><h4>Example Use Case:</h4><p>Consider a scenario where a web server application in a container encounters issues, leading to unresponsiveness or a crash. The liveness probe acts as a watchdog, detecting such anomalies and triggering an automatic container restart. This functionality is vital for sustaining the reliability and availability of the application within a Kubernetes environment.</p><h4><strong>How to configure readiness and liveness probes?</strong></h4><p>There are mainly 3 approaches to configure readiness and liveness probes in K8s, you have to choose the most suitable one and integrate it with your YAML file that contains the pod definition, deployment, statefulset, … etc.</p><p>Under each container, you have to define the configuration that will be used to check the readiness and liveness of this container by choosing one of the following approaches: <strong><em>HTTP, TCP, and Exec Commands.</em></strong></p><blockquote><strong>Note:</strong> Before building the application image, it is essential to implement the logic for readiness and liveness. Expose this logic as endpoints on a specific port. Doing so facilitates Kubernetes in testing the liveness and readiness by hitting these endpoints. The nature of your application determines the specific logic required for liveness and readiness.</blockquote><blockquote>The logic you implement serves as an answer to two crucial questions:</blockquote><blockquote><strong>When should my application be ready to receive requests? </strong>(readiness)<strong><br>When should my application be considered alive and running? </strong>(liveness)</blockquote><blockquote>If you are using a pre-built image (from docker-hub for example), refer to the image/application documentation to find out which endpoints/port are exposed for the readiness and liveness checks.</blockquote><h4>Configure ReadinessProbe:</h4><p><strong>HTTP Readiness Probe:</strong></p><p>The HTTP Readiness Probe stands out as the most widely used type. It dispatches an HTTP GET request to the specified path on the container’s IP address and port number. The container earns a readiness status if the response status code falls between 200 and 399; otherwise, it is deemed not ready.</p><p><strong>Example:</strong></p><pre>readinessProbe:<br>  httpGet:<br>    path: /ready<br>    port: 80</pre><p><strong>TCP Readiness Probe:</strong></p><p>The TCP Readiness Probe conducts a TCP check against the container’s IP address and port number. A successful TCP connection establishes the container’s readiness.</p><p><strong>Example:</strong></p><pre>readinessProbe:<br>  tcpSocket:<br>    port: 3306</pre><p><strong>Exec Readiness Probe:</strong></p><p>The Exec Readiness Probe executes a specified command within the container. If the command concludes with a status code of 0, the container is marked as ready.</p><p><strong>Example</strong>:</p><pre>readinessProbe:<br>  exec:<br>    command:<br>      - cat<br>      - /tmp/ready</pre><h4>Configure LivenessProbe:</h4><p><strong>HTTP Liveness Probe:</strong></p><p>Same as HTTP Readiness Probe with a different endpoint specified for Liveness.</p><p><strong>Example:</strong></p><pre>livenessProbe:<br>  httpGet:<br>    path: /live<br>    port: 80</pre><p><strong>TCP Liveness Probe:</strong></p><p><strong>Example:</strong></p><pre>livenessProbe:<br>  tcpSocket:<br>    port: 3306</pre><p><strong>Exec Liveness Probe:</strong></p><p>The Exec Liveness Probe executes a specified command within the container. If the command concludes with a status code of 0, the container is marked as live.</p><p><strong>Example</strong>:</p><pre>livenessProbe:<br>  exec:<br>    command:<br>      - cat<br>      - /tmp/live</pre><blockquote>As I mentioned before, these probes can be defined in the YAML configuration of your pod under the containers field. And again, The choice of probe type depends on the nature of your application and what criteria are meaningful for assessing its readiness and liveness.</blockquote><p>Here&#39;s an illustrative example featuring a combination of probes:</p><pre>containers:<br>- name: my-container<br>  image: my-image<br>  ports:<br>  - containerPort: 80<br>  readinessProbe:<br>    httpGet:<br>      path: /ready<br>      port: 80<br>  livenessProbe:<br>    httpGet:<br>      path: /live<br>      port: 80</pre><p>In this scenario, an HTTP readiness probe is checking the /ready path and an HTTP liveness probe checks the /live path.</p><h3>Conclusion</h3><p>In conclusion, Kubernetes probes, whether for readiness or liveness, provide crucial signals for application health. By configuring these probes in the pod definition, developers ensure their applications are ready to receive requests and remain responsive. Whether using HTTP, TCP, or execution-based probes, Kubernetes acts on the results, enhancing the resilience of containerized applications. Embracing Kubernetes probes is key to building robust and self-healing systems in the world of container orchestration.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c8eead045f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Set Up SSH Keys for Passwordless Remote SSH Connections]]></title>
            <link>https://medium.com/@abdelrhmanahmed131/how-to-set-up-ssh-keys-for-passwordless-remote-ssh-connections-689eab4ad32?source=rss-3227207ccd6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/689eab4ad32</guid>
            <dc:creator><![CDATA[Abdelrahman Ahmed]]></dc:creator>
            <pubDate>Fri, 28 Jul 2023 13:08:28 GMT</pubDate>
            <atom:updated>2023-07-28T13:08:28.941Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5fplOijpfHBhlgMKocjrkA.png" /><figcaption>Figure 1.1: SSH Remote Connection</figcaption></figure><p>If you frequently connect to remote hosts through SSH in your daily work, you may have experienced the hassle of entering the SSH password every time you want to establish a connection. In this article, we will explore an effective solution to this inconvenience — setting up permanent connections using SSH Keys. By utilizing SSH Keys instead of passwords, you can eliminate the need to repeatedly enter passwords and enjoy a seamless and secure remote access experience.</p><p>In the article, we will cover the step-by-step process of configuring SSH Keys for passwordless access to your remote hosts. Once implemented, this approach will enhance your productivity and improve the security of your remote connections.</p><p>The process is very simple and consists of some steps:</p><blockquote><strong>First:</strong> Create SSH Keys on your local system (Linux, macOS, or Windows).</blockquote><blockquote><strong>Second:</strong> Copy the public key from your local system to your remote host.</blockquote><blockquote><strong>Third:</strong> Connect to your remote host through SSH.</blockquote><h3>Create SSH Keys on your local system (Linux, macOS, or Windows)</h3><p>Regardless of the operating system installed on your local system (the system you’re using to connect to remote hosts … your laptop for example), you will need to generate SSH Keys first on that system. To do this, first, make sure you have an SSH Client enabled on your local system.</p><blockquote><strong>SSH Keys</strong> are a pair of keys used for secure authentication and communication in the SSH protocol, and they consist of two keys (public key/private key):</blockquote><blockquote><strong>Public Key:</strong> This one can be shared with others or stored on remote hosts/servers. It plays the role of encrypting the data that can only be decrypted by the corresponding private key.</blockquote><blockquote><strong>Private Key:</strong> This one will be stored only on your local system. It can’t be shared, and it is very sensitive to be exposed to others. It plays the role of decrypting the data that is encrypted by the corresponding public key.</blockquote><blockquote>So if you are using a single local system to connect to 10 remote hosts, you will store your public key on the 10 hosts but will keep your private key only on your system.</blockquote><blockquote>These keys play a significant role in the authenticity of establishing a secure connection between your local system and remote hosts/servers.</blockquote><h4>Creating SSH Key pair on Linux (RHEL)</h4><p>SSH Keys could be generated in multiple types such as ‘rsa’, ‘dsa’, ‘ed25519’, or ‘ecdsa’. The types can vary based on the SSH version you are using and can be specified by the ‘<strong>-t</strong>’ flag. I will go for the ‘<strong>rsa</strong>’ type…</p><p>Open up your terminal and write the following command:</p><pre>ssh-keygen -t rsa</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*jtjB8-hfrnUWLJ4sT1oyGA.png" /><figcaption>Figure 1.2: Create SSH Keys</figcaption></figure><p>As per the image above, you will be prompted to specify the file or path you want the Keys to be saved in. Hitting enter button will let the Keys be saved in the default path: <strong>~/.ssh</strong>.</p><p>You will also be prompted to specify a passphrase, which is an extra layer of security added to your private key. In my case, I will not be using a passphrase, so I am gonna hit enter too.</p><p>Let’s navigate to ~/.ssh to validate that Keys are generated successfully…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*9JaQGmohuxlChNTp_tUyMQ.png" /><figcaption>Figure 1.3: Validate SSH Keys creation</figcaption></figure><p>We have succeeded. As you see, there are two keys generated here: <strong>id_rsa</strong> is the private key, and <strong>id_rsa.pub</strong> is the public one.</p><h3>Copy the public key from your local system to your remote Host</h3><p>let’s suppose we want to make an SSH remote connection to the host 192.168.13.23, so we need to copy/append the content of our <strong>id_rsa.pub</strong> to the file: <strong>~/.ssh/authorized_keys</strong> that exists on this remote host.</p><p>To do this, you need to run this command on your local system …</p><pre><br># if the ~/.ssh path is not created yet on the remote host, create it<br>cat .ssh/id_rsa.pub | ssh root@192.168.13.23 &quot;mkdir ~/.ssh; cat &gt;&gt; ~/.ssh/authorized_keys&quot;<br><br># path ~/.ssh exist<br>cat .ssh/id_rsa.pub | ssh root@192.168.13.23 &quot;cat &gt;&gt; ~/.ssh/authorized_keys&quot;<br><br># let&#39;s break down this command<br><br>## cat .ssh/id_rsa.pub |<br>### The | here will inform the terminal to not print the content of the file on the screen<br>### Instead, make it an input to the operation that comes after the |<br><br>## ssh root@192.168.13.23<br>### will establish a new ssh connection to &#39;root&#39; user on the host &#39;192.168.13.23&#39;<br>### Do you think this will require entering a password?<br><br>## &quot;mkdir ~/.ssh; cat &gt;&gt; ~/.ssh/authorized_keys&quot;<br>### Commands withis &quot;&quot; will be executed on the remote host<br>### A directory with the name .ssh will be created<br>### The content of id_rsa.pub will be added to ~/.ssh/authorized_keys <br>### please note that authorized_keys will be created if not exist</pre><p>If you’re using Windows as your local system, instead of <strong>cat</strong>, you have to use <strong>type</strong>, so the command will be as follows …</p><pre># if the ~/.ssh path is not created yet on the remote host, create it<br>&gt; type .ssh/id_rsa.pub | ssh root@192.168.13.23 &quot;mkdir ~/.ssh; cat &gt;&gt; ~/.ssh/authorized_keys&quot;<br><br># path ~/.ssh exist<br>&gt; type .ssh/id_rsa.pub | ssh root@192.168.13.23 &quot;cat &gt;&gt; ~/.ssh/authorized_keys&quot;</pre><p>By this point, your public key should be copied to the file ‘authorized_keys’ on the remote host. Let’s check on this…</p><p>First, I will run the copying command on my local system to copy the public key to the remote host. Keep in mind that it will prompt for the remote host password.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*Ta2iyjY24KvpQGLb1Jy7Xw.png" /><figcaption>Figure 1.4: Copy SSH Public Key to remote host</figcaption></figure><p>As you see in the above image, it went successfully.</p><h3>Connect to your remote host through SSH</h3><p>As creating the Keys and copying the public key to the remote host was done successfully, let’s try to connect to our remote host from the local system using SSH and check if it will prompt for a password or not.</p><p>To open an SSH connection with the remote host, use the following command …</p><pre>ssh user@Host_IP<br># in our case: ssh root@192.168.13.23<br></pre><p>Let’s try this out …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*ECvWAtQEAT_iDcFJCuc9Ww.png" /><figcaption>Figure 1.5: Establish SSH connection with the remote host</figcaption></figure><p>It worked :)</p><h4>Conclusion</h4><p>In this article, we have provided a comprehensive guide on how to set up SSH Key pairs for seamless and secure connections to remote hosts. By following our step-by-step example, you can establish a passwordless connection, enhancing both convenience and security.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=689eab4ad32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Compute Services | Serverless Explained]]></title>
            <link>https://medium.com/@abdelrhmanahmed131/aws-compute-services-serverless-explained-b14d3c82a279?source=rss-3227207ccd6b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b14d3c82a279</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Abdelrahman Ahmed]]></dc:creator>
            <pubDate>Tue, 20 Jun 2023 22:39:32 GMT</pubDate>
            <atom:updated>2023-06-20T22:39:32.245Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gdB2_abXWbuOzL1ZE6HPmQ.png" /><figcaption>Figure 1.1: AWS Serverless Computing</figcaption></figure><h4><strong>What does “compute” word mean?</strong></h4><p>In the client-server model, when running an application, a server is required to host the application. The server plays a crucial role in handling HTTP requests from clients and returning the responses after processing the requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*WssKA03_nVOy8y8Z.png" /><figcaption>Figure 1.2: Client-Server model</figcaption></figure><p>Typically, a server refers to a computer or a network of computers that are connected to the internet and equipped with the necessary infrastructure to support your application. Its primary function is to execute your application and effectively manage the incoming HTTP requests.</p><p>Server infrastructure consists of essential components such as <strong><em>CPU</em></strong>, <strong><em>memory</em></strong>, <strong><em>storage</em></strong>, and <strong><em>networking capacity</em></strong>. It’s important to note that the server also has an operating system that facilitates communication between the infrastructure and your running application(s).</p><p><strong>Returning to the definition of “compute”...</strong></p><blockquote>In the context of cloud computing, “<strong>compute</strong>” refers to the infrastructure resources offered by cloud service providers to host and execute applications. These resources include CPU, memory, storage, and networking capacity required to run applications within the cloud environment.</blockquote><h4>AWS <strong><em>compute </em></strong>services</h4><p>AWS provides various compute services to support different application needs, these services include:</p><ul><li><strong>Elastic Compute Service (EC2):</strong> this service offers virtual machine instances for flexible computing power.</li><li><strong>Container Services (ECS, EKS):</strong> these services are used for running and managing containers.</li><li><strong>Serverless Services:</strong> these services allow running applications without provisioning or managing any servers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P9dtU28ii853LrzBAZ92IA.png" /></figure><p>This article&#39;s scope will be Serverless Services.</p><h4>What is Serverless?</h4><p>I believe in learning through examples, imagine using EC2 as a compute service. With EC2, you can create virtual machine instances and run your application on those instances while AWS takes care of the physical hardware. However, you still have to manage the guest operating system, handle security and patching, manage networking, and scale the instances as needed when your application grows.</p><p>If you decided to use container compute services, you typically run your containers on EC2 instances. In this case, you still need to manage the underlying EC2 instances.</p><p>But what if you want to eliminate the overhead of managing EC2 instances and simply deploy your applications without worrying about the infrastructure, scalability, and availability of your applications? This is where Serverless compute services come into play.</p><p><strong>Serverless services provide a compute capacity that offers the following for you:</strong></p><ul><li>No need to provision or manage any infrastructure since there are no servers to provision.</li><li>The availability of compute resources is already built-in and handled by AWS.</li><li>Scalability is automatically managed by AWS, so you don’t have to worry about it as your application needs to grow.</li><li>You only pay for what you use, avoiding costs for idle resources. You utilize the compute capacity you truly need and nothing more.</li></ul><p>In essence, Serverless compute services allow you to focus on your code, business use cases, and application development without being burdened by any infrastructure concerns.</p><h4>AWS Serverless compute services</h4><p>AWS offers two primary serverless compute services:</p><ol><li><strong>AWS Lambda:</strong> AWS Lambda is an event-driven compute service that enables you to execute your application code in response to various triggers. It follows a pay-as-you-go model, where you only pay for the actual usage of your functions.</li><li><strong>AWS Fargate:</strong> AWS Fargate is a compute service designed for running containers. It is well-suited for applications that leverage AWS ECS (Elastic Container Service) or AWS EKS (Elastic Kubernetes Service). With AWS Fargate, you can focus on deploying and managing your containers without the need to provision or manage the underlying infrastructure.</li></ol><h4>AWS Lambda</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/75/1*pFtl5Qr0f9jhnOqhFIszkQ.png" /><figcaption>Figure 1.3: AWS Lambda icon</figcaption></figure><p>Lambda is the most widely used serverless compute service offered by AWS. It operates on an event-driven model, which means it is triggered only when necessary. This approach optimizes resource utilization, reduces costs, and saves time. With Lambda, you don’t need to maintain continuous compute resources; they are automatically activated as needed based on the demand of your application.</p><p><strong><em>How does Lambda work?</em></strong></p><p>When deploying code with Lambda, the first step is to package and upload your code to the AWS Lambda service, creating what is known as a Lambda function. Lambda functions are not continuously running; instead, they are triggered in response to specific events.</p><p>Triggers play a crucial role in initiating your Lambda function and enabling your application as needed. The Lambda service continuously monitors for these triggers and executes the Lambda function (executing your code) when the specified trigger event occurs.</p><p><strong>There is a wide range of triggers available to invoke Lambda functions, including:</strong></p><ul><li>HTTP requests</li><li>S3 bucket events (such as file uploads to Amazon S3)</li><li>Integration with load balancers</li><li>Messaging services (e.g., Kafka)</li><li>Mobile application events</li><li>Integration with other AWS services like DynamoDB, CloudFront, CloudWatch, and more.</li></ul><blockquote><strong>The most commonly used approaches for triggering Lambda functions are sending HTTP requests and uploading files to Amazon S3.</strong></blockquote><p>When a trigger is received, Lambda executes your application within a managed and isolated environment. This environment is designed to be scalable, highly available, and maintained by AWS. As the number of triggers increases, Lambda automatically creates additional environments to handle the workload, ensuring isolation and optimal performance for your application.</p><p>When creating a Lambda function, you have the opportunity to configure various aspects, including the runtime (the programming language used in your code), CPU architecture, permissions, and any necessary dependencies. These configurations allow you to tailor the function to meet your specific requirements.</p><h4>Let’s go through a simple example that demonstrates how to create a Lambda function using the AWS console …</h4><p><strong><em>To Create the Lambda function, </em></strong>Go to the AWS console and search for Lambda in the search bar and choose Lambda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bx6ochaZ2e1fWYwiXqCzgg.png" /><figcaption>Figure 1.4: Searching for AWS Lambda service on AWS Console</figcaption></figure><p>Click on <strong><em>Functions</em></strong> then click on <strong><em>Create function. </em></strong>At this point, you will have multiple options to choose from for creating your Lambda function, these options include:</p><ol><li><strong>Author from scratch:</strong> Create the function on your own.</li><li><strong>Use blueprint:</strong> Using a sample from a pre-created function by others.</li><li><strong>Container image:</strong> Using containers instead of written code.</li><li><strong>Browse serverless app repository:</strong> Pick a sample Lambda application from AWS serverless repository.</li></ol><p>I will go for creating the function from scratch …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ISXPeWm8jWEB0bvXSe0lA.png" /><figcaption>Figure 1.5: Searching for AW</figcaption></figure><p><strong><em>Then configure the basic information</em></strong> such as Function name, Runtime, and CPU Architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ohKQjLTebL79yPnM_fpm7Q.png" /><figcaption>Figure 1.6: Configure Lambda Function</figcaption></figure><p>The next step is <strong><em>defining the rules and permissions</em></strong> you may need for your Lambda function, in this example I will stick to the default Lambda permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tu80e1JewQVViuIlGWep0A.png" /><figcaption>Figure 1.7: Defining rules and permissions</figcaption></figure><p>Click<strong><em> Create function …</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*ABdY-cQmpwDRhSVNadNyIw.png" /><figcaption>Figure 1.8: Create function</figcaption></figure><p>And our Lambda function is here :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iViBoTpYBrzP3vHXdUehdg.png" /><figcaption>Figure 1.9: Created Lambda Function</figcaption></figure><p>Scroll down to the code section, and edit the code to return <strong><em>“Hello From Lambda!”</em></strong> message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*MsJ_CceauibiKZYGbTlHkw.png" /><figcaption>Figure 2.0: Lambda code (runtime) editor</figcaption></figure><blockquote>At this point, A Lambda function has been created with a simple functionality (just response with hello message), but it won’t work unless some trigger set and invoked. After invoking the trigger, AWS Lambda service will execute the function and a response with <strong><em>“Hello from Lambda!”</em></strong> message should be returned.</blockquote><h4>Let’s set an HTTP request as a trigger for our Lambda function …</h4><p><strong><em>Creating an API Gateway</em></strong> to trigger the Lambda function.</p><p>Go to the console and search for <strong><em>API Gateway</em></strong> in the same way you searched for Lambda service.</p><p><strong><em>Choose REST API</em></strong> then make the initial configuration and click Create API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZAWgGTC3Hc8qfm1X6FCtw.png" /><figcaption>Figure 2.1: Configure Amazon API Gateway</figcaption></figure><p>Click<strong><em> Actions</em></strong> &gt; <strong><em>Create method</em></strong> &gt; <strong><em>Choose GET</em></strong> &gt; <strong><em>Integration type:</em></strong> <strong><em>Lambda Function</em></strong> and enter your Lambda function name then click <strong><em>Save</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jqhDTRgizOP2XztYbqQWpw.png" /><figcaption>Figure 2.2: Configure Amazon API Gateway ||</figcaption></figure><p>At this point, your API has been created and integrated with your Lambda function as a trigger.</p><p>To make sure of that let’s take a look at the Lambda diagram and see what is exist in the triggers section.</p><p>We did it :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*bfeQKm505bS39LrQSReVqQ.png" /><figcaption>Figure 2.3: Lambda Diagram — API Trigger is here</figcaption></figure><p>And by following the clarified navigation, you can find the path of your created API as follows …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m-7ah7JVB9AkWmkodbNjGw.png" /><figcaption>Figure 2.4: API Path</figcaption></figure><p>At this point, the API has been created and integrated with Lambda as a trigger. However, to use the API, it should be deployed first.</p><p>So, let’s deploy this API …</p><p>Click <strong><em>Actions</em></strong> and choose <strong><em>Deploy API</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*laXdZNRN2GChBE67ObQ_8A.png" /><figcaption>Figure 2.5: API Actions — Deploy</figcaption></figure><p><strong><em>Create a new stage</em></strong> to add your API deployment details then click <strong><em>Deploy.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*awRa_eA_-atYz7yDgshQ3w.png" /><figcaption>Figure 2.6: Creating a new deployment stage</figcaption></figure><p>The API deployment will be created and you can find the details on the next page …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KGvUyg3q3XPPunJmGouMiw.png" /><figcaption>Figure 2.7: Deployment details</figcaption></figure><p>On the same deployment page, you will find the EndPoint for your Lambda function, this is the URL that you gonna use to send an HTTP request to trigger your Lambda Function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*KeIiHtYqqUkE4hRtl5s12w.png" /><figcaption>Figure 2.8: Lambda function endpoint</figcaption></figure><p>By accessing the URL, you will be able to see the response generated by the Lambda function, confirming that it has been triggered successfully and has provided the expected output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*QB13GAgZBXBrU7yLtrW5YQ.png" /></figure><h4>Benefits of using Serverless</h4><p>From a <strong><em>business perspective</em></strong>, there are several benefits to adopting a serverless approach:</p><ul><li><strong><em>Cost Reduction:</em></strong> With serverless, you only pay for the actual usage of resources, eliminating the need for running a server 24/7 and reducing costs.</li><li><strong><em>Faster Time to Market:</em></strong> Since you don’t have to provision servers or manage infrastructure, you can deploy your applications quickly and bring them to market faster.</li></ul><p>From a <strong><em>development perspective,</em></strong> the benefits of serverless computing are:</p><ul><li><strong><em>Simplified Infrastructure Management:</em></strong> Serverless removes the burden of server provisioning and infrastructure management, allowing developers to focus on writing code rather than worrying about the underlying infrastructure.</li><li><strong><em>Scalability and High Availability:</em></strong> Serverless services come with built-in capabilities for scalability and high availability, ensuring that your applications can handle increased traffic and remain accessible to users.</li><li><strong><em>Easy Feature Addition:</em></strong> Serverless architecture makes it easier to add new features to your application without worrying about infrastructure changes, so it will be fast and easy to add new features to your application.</li></ul><h4>When you should go Serverless and when you should not?</h4><p>Adopting Serverless approach will be evaluated based on your application behavior and requirements. Here are some factors to consider:</p><ol><li><strong><em>Application Availability:</em></strong> If your application needs to be running continuously or for extended periods of time, a serverless architecture may not be the most suitable solution. Serverless computing is designed for work-as-needed scenarios rather than 24/7 applications.</li><li><strong><em>Scalability Requirements:</em></strong> If your application experiences rapid growth or has unpredictable scalability needs, serverless can be a good choice. Serverless platforms automatically scale resources based on demand, allowing your application to handle varying workloads effectively.</li><li><strong><em>Real-Time Computing: </em></strong>If your application requires real-time processing or low-latency response times, a serverless architecture may not be the best option. Serverless functions typically have a warm-up time and might not offer real-time capabilities.</li></ol><p>So, using Serverless Architecture based services depends on your application characteristics and requirements, you have to evaluate multiple factors such as Availability, Scalability, and many other factors to determine whether Serverless Architecture will suit your application or not.</p><h4>Conclusion</h4><p>AWS Serverless services offer a comprehensive suite of infrastructure computing resources and features that give developers the ability to concentrate on application development while enabling business owners to achieve a rapid time-to-market and alleviate the complexities of infrastructure and resource management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b14d3c82a279" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>