<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mouaaz Farrukh on Medium]]></title>
        <description><![CDATA[Stories by Mouaaz Farrukh on Medium]]></description>
        <link>https://medium.com/@mouaazfarrukh99?source=rss-f7772e0e7277------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*xr33FF_pWFut0a_e</url>
            <title>Stories by Mouaaz Farrukh on Medium</title>
            <link>https://medium.com/@mouaazfarrukh99?source=rss-f7772e0e7277------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:18:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mouaazfarrukh99/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data & AI Solutions for Healthcare]]></title>
            <link>https://medium.com/@mouaazfarrukh99/data-ai-solutions-for-healthcare-45e6d400c2ee?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/45e6d400c2ee</guid>
            <category><![CDATA[boomi]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Mon, 30 Sep 2024 20:09:49 GMT</pubDate>
            <atom:updated>2024-09-30T20:09:49.379Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EGxCiUZ5tcjtxFh4hrGQbg.png" /></figure><p>The aim of this resource is to provide insights in how to design and develop data and AI based solutions for the healthcare industry.</p><h4>Legacy Systems: The Stumbling Block for Modern Healthcare</h4><p>Many healthcare organizations today still rely on legacy systems, which were never designed to handle the dynamic data demands of modern healthcare operations. The biggest challenges include:</p><ul><li><strong>Data Fragmentation</strong>: Systems like traditional hospital information systems (HIS) and outdated electronic health record (EHR) platforms (e.g., older versions of Cerner EMR) trap data in silos, making it difficult to create a 360-degree view of patient health.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*vgRrRk3CugcBw2x_Ogse7w.png" /></figure><ul><li><strong>Inflexible Integration</strong>: Legacy systems often lack modern integration standards such as FHIR (Fast Healthcare Interoperability Resources) or HL7, which means connecting them to new-age applications like AI-based analytics tools or telemedicine platforms is a challenge.</li><li><strong>Compliance Overhead</strong>: Legacy systems typically require more effort to comply with modern privacy regulations such as GDPR or HIPAA, as they were built with older security models in mind.</li></ul><h4>The Cloud as the New Backbone of Healthcare Operations</h4><p>Healthcare organizations are increasingly migrating to the cloud to overcome these legacy challenges, gain operational flexibility, and scale AI-driven initiatives. Cloud migration, however, is not a one-size-fits-all approach. Here’s how organizations make the shift:</p><ul><li><strong>Hybrid Cloud for Gradual Transition</strong>: Many healthcare organizations, especially those with sensitive patient data, start with a hybrid cloud model. This allows them to keep critical EMRs (like Cerner or Epic) on-premise for data security while moving analytics and non-sensitive operations to cloud platforms like AWS, Azure, or GCP. For example, Azure has <strong>Azure Health Data Services</strong> for FHIR and DICOM integration.</li><li><strong>Cloud-Native Solutions</strong>: Some organizations go fully cloud-native to take advantage of seamless scalability and advanced AI capabilities. For instance, organizations using <strong>Epic’s Cloud Offering</strong> on AWS can benefit from rapid deployment and global availability of their health systems with built-in regulatory compliance features.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e-36HJ43xAI1rSCq" /><figcaption>High-level Overview of Azure providing Data Solutions for Healthcare</figcaption></figure><h4>iPaaS Solutions for Healthcare Integration: Picking the Right Tool</h4><p>Integration Platform as a Service (iPaaS) solutions are used to orchestrate and automate data flows between healthcare systems. Selecting the right iPaaS for a healthcare data platform can depend on several factors:</p><ul><li><strong>Boomi</strong>: A popular choice for healthcare organizations looking for a quick, cloud-based integration solution. Boomi offers good connectors for healthcare standards (HL7, FHIR, etc.). However, a limitation is that Boomi’s <strong>control plane</strong> is based in specific regions, which can raise concerns for highly regulated healthcare data environments, particularly in regions with strict data residency laws.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*S-od5X_Os7O65EilAIgmPA.png" /><figcaption>Why Boomi</figcaption></figure><ul><li><strong>MuleSoft</strong>: MuleSoft’s <strong>Anypoint Platform</strong> is often seen as the gold standard for large-scale healthcare integration projects. It provides robust support for <strong>on-premise, hybrid, and multi-cloud</strong> integrations. However, MuleSoft isn’t as cloud-native as some of its competitors, and its complex setup can be overkill for smaller organizations looking for more agile solutions.</li><li><strong>Workato</strong>: A more user-friendly option with low-code integration capabilities. Workato is particularly useful for healthcare organizations wanting to rapidly build workflows across disparate systems without relying on large development teams. It’s great for cloud-first environments but lacks the deep healthcare-specific integrations that MuleSoft provides.</li></ul><p>Each iPaaS platform has different strengths depending on the scale and complexity of the healthcare system. A large hospital system with multiple on-prem sources might lean towards <strong>MuleSoft</strong>, while a smaller, agile startup in telehealth might find <strong>Boomi</strong> or <strong>Workato</strong> better suited.</p><h4>Building the Data Platform: When to Choose Databricks, Cloudera, Azure, or AWS</h4><p>Choosing the right data platform is critical to ensure scalability, performance, and compliance. The decision often comes down to organizational size, cloud strategy, and the need for advanced analytics.</p><ul><li><strong>Databricks</strong>: If an organization is aiming for a cloud-native, scalable data lake and has a focus on advanced AI and ML, <strong>Databricks</strong> (available on AWS, Azure, and GCP) is a top choice. It excels at unifying data engineering, data science, and machine learning in a single platform, making it suitable for organizations that need to process large-scale datasets (e.g., genomic data).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*Kvb4EyRrQe6EEGQCOXlNxQ.png" /><figcaption>Why databricks?</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*J3znXyVc4y-_v_AwQvvLBQ.png" /><figcaption>Why Databricks</figcaption></figure><ul><li><strong>Cloudera</strong>: For organizations needing a hybrid solution or more on-prem control (especially for sensitive data), <strong>Cloudera</strong> is often chosen. It provides strong integration with Hadoop-based storage systems and can handle structured and unstructured healthcare data across on-prem and cloud environments. However, Cloudera requires significant operational overhead and may not suit organizations looking for fully managed services.</li><li><strong>Azure and AWS</strong>: Both <strong>Azure</strong> and <strong>AWS</strong> provide comprehensive cloud solutions for healthcare data platforms. Azure’s integration with <strong>Microsoft 365</strong> makes it an excellent choice for organizations already embedded in the Microsoft ecosystem.</li></ul><h4>ETLs, Storage, and Data Processing</h4><ul><li><strong>ETL Tools</strong>: Extract, Transform, and Load (ETL) pipelines form the backbone of healthcare data platforms, moving data from EMRs, lab systems, and IoT devices into the data warehouse. Popular ETL tools include <strong>Azure Data Factory</strong>, <strong>AWS Glue</strong>, and <strong>Talend</strong>. Healthcare-specific transformations often involve converting <strong>HL7 messages to FHIR</strong>, de-identifying patient data for compliance, and normalizing imaging formats (DICOM).</li><li><strong>Storage</strong>: Cloud-based storage solutions such as <strong>Azure Data Lake</strong> or <strong>AWS S3</strong> are designed to store vast amounts of structured and unstructured healthcare data, from patient records to genomic sequences. <strong>Delta Lake on Databricks</strong> offers healthcare organizations a way to manage real-time data processing while ensuring ACID transactions for compliance.</li></ul><h4>Potential Domains and Use Cases for Data &amp; AI in Healthcare</h4><p>Healthcare organizations can apply data and AI solutions across multiple domains:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*gkf_58fjc04M6ccc.png" /></figure><ul><li><strong>Patient Diagnostics</strong>: AI models assist in early diagnosis, predicting diseases based on historical data, or detecting patterns in imaging.</li><li><strong>Operational Optimization</strong>: AI can improve resource allocation, bed management, and streamline workflows in hospitals.</li><li><strong>Predictive Analytics</strong>: By analyzing historical patient data, predictive models help forecast disease outbreaks, readmission rates, or patient deterioration.</li><li><strong>Personalized Medicine</strong>: AI can tailor treatment plans based on individual genetic data, lifestyle factors, and historical medical records.</li><li><strong>Clinical Research</strong>: AI accelerates drug discovery and clinical trials by analyzing vast datasets quickly.</li></ul><h3>Conclusion: Navigating the Data-Driven Future of Healthcare</h3><p>Healthcare organizations are at a critical juncture where adopting data and AI solutions is no longer optional but essential. With the right strategies for cloud migration, system integration, and AI-driven analytics, healthcare providers can not only overcome the limitations of legacy systems but also lead the way in delivering personalized, efficient, and scalable patient care.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kICQy4tl4RY_mGBq" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45e6d400c2ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Containerizing ETL Jobs on AWS]]></title>
            <link>https://medium.com/@mouaazfarrukh99/containerizing-etl-jobs-on-aws-c85ad6cc7b08?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/c85ad6cc7b08</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Mon, 29 Jan 2024 10:54:09 GMT</pubDate>
            <atom:updated>2024-01-29T10:54:09.805Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*RZHaF8oSp1C5vbd4L61Tng.png" /><figcaption>Containerizing ETL Jobs on AWS</figcaption></figure><p>Leverage Batch Processing for ETL Jobs using ECS on AWS.</p><p><strong>Problem Statement:</strong></p><p>An e-commerce company receives a large volume of daily transaction data that needs to be processed and loaded into a data warehouse for analytics. The current process relies on traditional ETL (Extract, Transform, Load) jobs running on a fixed schedule, which results in occasional performance bottlenecks and delays during peak times.</p><p><strong>Existing State:</strong></p><p>The current ETL process is handled by a monolithic application running on a single server. The system struggles to cope with the increasing volume of daily transactions, leading to slower processing times and occasional job failures. Scaling the existing infrastructure to handle peak loads is both time-consuming and resource-intensive.</p><p><strong>Proposed Solutions:</strong></p><p>Containerize ETL jobs and deploy them on Amazon ECS. This allows for efficient packaging, rapid scaling, and dynamic resource allocation based on demand.</p><p><strong>Advantages:</strong></p><ul><li><strong>Scalability:</strong> Easily scale the number of containers to handle varying workloads.</li><li><strong>Isolation:</strong> Each ETL job runs in its own container, ensuring resource isolation and fault tolerance.</li><li><strong>Resource Efficiency: </strong>Containers share the underlying infrastructure, maximizing resource utilization.</li></ul><p><strong>Estimated Costs/Quotas:</strong></p><ul><li>Costs are based on the number of containers and their resource usage.</li><li>AWS provides a free tier for ECS, covering container resources.</li><li>Costs may include data transfer &amp; storage for processed data S3 or RDS.</li></ul><p><strong>Step by Step Implementation:</strong></p><p>For complete end-to-end implementation, follow this github repo: <a href="https://github.com/syedmouaazfarrukh/hands-on-clouds/tree/main/use-cases/containers/containerize-etljobs-ecs">Hands-on-Cloud</a>. A simple overview of the steps are as follows:</p><ul><li>Download the dataset from Kaggle.</li><li>Setup your problem</li><li>Separate two ETL jobs</li><li>Create Dockerfile for each job in their respective folders</li><li>Build and push your docker images on ECR (Elastic Container Registry)</li><li>Create ECS <em>*Tasks Definitions*</em> using a JSON format.</li><li>Create ECS cluster and connect it with your task definitions.</li><li>The cluster will only execute your tasks/ETL jobs once.</li><li>In order to run at regular intervals, Go to CloudWatch/Events/Rules, create a EventBrideRule and connect it with the cluster and tasks.</li></ul><p><strong>Conclusion:</strong></p><p>The proposed solution uses containerized batch processing with Amazon ECS to address performance bottlenecks in an e-commerce company’s ETL process. By containerizing ETL jobs, the solution offers scalability, isolation, and resource efficiency.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c85ad6cc7b08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Bucket and Upload/Retrieve Objects from GCS using Python]]></title>
            <link>https://medium.com/@mouaazfarrukh99/create-bucket-and-upload-retrieve-objects-from-gcs-using-python-5ebd0c5e9246?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ebd0c5e9246</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[storage]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Mon, 11 Sep 2023 11:51:21 GMT</pubDate>
            <atom:updated>2023-09-11T11:51:21.674Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OslBw0Pydk7LSaSpTiodng.png" /><figcaption>Buckets &amp; Objects in Cloud Storage using Python</figcaption></figure><p>Learn how to create buckets in google cloud storage services and then upload and retrieve objects from it using python.</p><h3><strong>Overview of Buckets:</strong></h3><p>Google Cloud Storage (GCS) Buckets are the top-level containers used to organize and control access to your data. Here are some key characteristics of GCS buckets:</p><ol><li><strong>Objects:</strong> A bucket is essentially a container for storing objects. Objects can be files, images, videos, or any data you want to store in the cloud.</li><li><strong>Unique Names</strong>: Each bucket must have a globally unique.</li><li><strong>Access Control:</strong> You can control access to buckets by configuring permissions and authentication.</li><li><strong>Storage Classes:</strong> GCS offers different storage classes for buckets, such as <strong><em>Standard, Nearline, Coldline</em></strong>, and <strong><em>Archive</em></strong>.</li></ol><p>For more check out the original documentation at <a href="https://cloud.google.com/storage/docs">Google Cloud Storage</a>.</p><h4>Step 1: Create a GCS Bucket</h4><p>First, let’s create a new GCS bucket using Python:</p><pre>def create_gcs_bucket(project_id, bucket_name, service_account_file):<br><br>    from google.cloud import storage<br><br>    &quot;&quot;&quot;<br>    project_id (str): Your Google Cloud Platform (GCP) project ID.<br>    bucket_name (str): Name of the new bucket to be created.<br>    service_account_file (str): Your service account JSON key file<br>    &quot;&quot;&quot;<br>    <br>    try:<br>        # Initialize a GCS client<br>        client = storage.Client.from_service_account_json(service_account_file)<br><br>        # Create a new bucket<br>        bucket = client.create_bucket(bucket_name)<br><br>        print(f&#39;Bucket {bucket_name} created successfully.&#39;)<br>    <br>    except Exception as e:<br><br>        print(f&#39;Error creating bucket: {str(e)}&#39;)<br><br></pre><h3>Overview of Objects:</h3><p>Google Cloud Storage (GCS) Objects are the individual pieces of data stored within GCS buckets. Here are some key characteristics of GCS objects:</p><ol><li><strong>File-Level Data:</strong> Objects can represent files, data blobs, or any binary data you want to store. Each object is identified by a unique key within a bucket.</li><li><strong>Metadata:</strong> Objects can have associated metadata, which includes key-value pairs providing information about the object, such as content type, creation date, and custom attributes.</li><li><strong>Access Control:</strong> Like buckets, objects can have their own access control settings. You can specify who can access, read, and modify individual objects.</li><li><strong>Immutability:</strong> Objects in GCS are immutable, meaning you cannot directly modify an object once it’s created. If you need to update an object, you typically create a new version of it.</li><li><strong>Versioning:</strong> GCS allows you to enable versioning for objects, so you can maintain multiple versions of the same object over time.</li><li><strong>Storage Classes:</strong> Like buckets, you can specify the storage class for each object, which affects its performance and cost.</li></ol><p>A simple way of explaining <em>Buckets</em> &amp; <em>Objects</em> is that GCS buckets are the organizational containers that hold GCS objects, while GCS objects are the individual data units that you store within buckets.</p><h4>Step 2: Upload an Object in Bucket</h4><p>You can upload an object using the following function by passing your ID and project credentials.</p><pre><br>def upload_object_to_bucket(bucket_name, source_file, destination_blob_name, service_account_file):<br>    <br>    from google.cloud import storage<br>    <br>    &quot;&quot;&quot;<br>    bucket_name (str): Name of the GCS bucket where the object will be uploaded.<br>    source_file (str): Path to the local file to be uploaded.<br>    destination_blob_name (str): Name of the object in the GCS bucket.<br>    service_account_file (str): Your service account JSON key file<br>    &quot;&quot;&quot;<br><br>    try:<br>        # Initialize a GCS client<br>        client = storage.Client.from_service_account_json(service_account_file)<br><br>        # Get a reference to the target GCS bucket<br>        bucket = client.get_bucket(bucket_name)<br><br>        # Upload the local file to GCS<br>        blob = bucket.blob(destination_blob_name)<br>        blob.upload_from_filename(source_file)<br><br>        print(f&#39;File {source_file} uploaded to {bucket_name}/{destination_blob_name}&#39;)<br>    <br>    except Exception as e:<br>        print(f&#39;Error uploading file: {str(e)}&#39;)<br><br></pre><h4>Step 3: Retrieve an Object from Bucket</h4><pre>from google.cloud import storage<br><br>def retrieve_object_from_bucket(project_id, bucket_name, object_name, destination_file_path, service_account_file):<br>    <br>    &quot;&quot;&quot;<br>        project_id (str): Your Google Cloud project ID.<br>        bucket_name (str): The name of the GCS bucket.<br>        object_name (str): The name of the object you want to retrieve.<br>        destination_file_path (str): The path to save the retrieved object locally.<br>    &quot;&quot;&quot;<br>    <br>    try:<br>        # Initialize a GCS client<br>        client = storage.Client.from_service_account_json(service_account_file)<br><br>        # Get the bucket<br>        bucket = client.get_bucket(bucket_name)<br><br>        # Get the blob (object) from the bucket<br>        blob = bucket.blob(object_name)<br><br>        # Download the blob to the specified file path<br>        blob.download_to_filename(destination_file_path)<br><br>        print(f&quot;Object &#39;{object_name}&#39; retrieved and saved to &#39;{destination_file_path}&#39;.&quot;)<br><br>    except Exception as e:<br>        print(f&quot;Error: {e}&quot;)<br><br></pre><h3>Conclusion</h3><p>Through this, you’ll have a good understanding of how Google Cloud Storage, Buckets and Objects should be used and how to perform functions on these using Python.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ebd0c5e9246" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with Pub/Sub using Python]]></title>
            <link>https://medium.com/@mouaazfarrukh99/getting-started-with-pub-sub-using-python-305a19901f1a?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/305a19901f1a</guid>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[messaging-system]]></category>
            <category><![CDATA[subscription]]></category>
            <category><![CDATA[pub-sub]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Sun, 10 Sep 2023 16:44:31 GMT</pubDate>
            <atom:updated>2023-09-10T16:44:31.625Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MrBij6Wbwcu9tV1l6EWvXg.png" /><figcaption>Getting Started with Pub/Sub using Python</figcaption></figure><p>Google Cloud Pub/Sub is a fully managed messaging service that empowers developers to build event-driven systems.</p><h3>Step 1: Overview of Pub/Sub</h3><p>Pub/Sub offers a scalable, reliable, and fully managed solution for messaging. It supports multiple message types and enables different parts of your application to communicate independently.</p><h4>Components of Pub/Sub</h4><p>The Pub/Sub (Publisher-Subscriber) service consists of several key components which are as follows:</p><ol><li><strong>Topics: </strong>Topics are named channels or categories to which messages are sent by publishers.</li><li><strong>Subscriptions: </strong>Subscriptions are associated with specific topics and define where the messages should be delivered.</li><li><strong>Publishers: </strong>Publishers are entities that send messages to topics. <em>Publishers can be applications, devices, or services that generate data.</em></li><li><strong>Subscribers: </strong>Subscribers are entities that consume and process messages from Pub/Sub topics.</li><li><strong>Messages: </strong>Messages are units of data or events sent by publishers to topics.</li></ol><h3>Step 2: Difference between Pub/Sub &amp; Pub/Sub Lite</h3><p>While Pub/Sub is a powerful messaging service, Google offers a simplified version called <strong>Pub/Sub Lite</strong>. Understanding the differences between these two services is crucial for choosing the right fit for your project.</p><p>Pub/Sub is suitable for global applications with high throughput, while Pub/Sub Lite shines in regional applications or when cost optimization is paramount.</p><h3>Step 3: Types of Subscriptions</h3><p>Subscriptions are a fundamental concept in Pub/Sub, determining how messages are delivered to subscribers. Let’s explore the 4 types:</p><ol><li><strong>Push Subscriptions:</strong> Push subscriptions allow Pub/Sub to deliver messages to a user-defined HTTP/HTTPS endpoint in real-time.</li><li><strong>Pull Subscriptions:</strong> Subscribers request messages from the subscription by sending pull requests.</li><li><strong>Cloud Storage Subscriptions:</strong> Cloud Storage subscriptions are used when you want to send/receive data from or to buckets in GCS.</li><li><strong>BigQuery Subscriptions:</strong> BigQuery subscriptions enable you to stream data changes from BigQuery tables into Pub/Sub topics.</li></ol><h3>Step 4: Create/Write/Read from PubSub</h3><h4>Create a Topic:</h4><pre><br>def create_topic(project_id: str, topic_id: str, service_account_file: str) -&gt; None:<br><br>    &quot;&quot;&quot;Creating a new Pub/Sub topic.&quot;&quot;&quot;<br><br>    # Importing libraries within function<br>    from google.cloud import pubsub_v1<br><br>    # Creating Pub/Sub Publisher Client, Specifying path of Topic<br>    publisher = pubsub_v1.PublisherClient.from_service_account_file(service_account_file)<br>    topic_path = publisher.topic_path(project_id, topic_id)<br><br>    # Creating a Topic Object at the topic_path<br>    topic = publisher.create_topic(request={&quot;name&quot;: topic_path})<br><br>    # Printing the created Topic<br>    print(f&quot;Created topic: {topic.name}&quot;)</pre><h4>Create Subscriptions (Push/Pull/GCS)</h4><pre><br>def create_pull_subscription(project_id: str, topic_id: str, subscription_id: str, service_account_file: str) -&gt; None:<br>    <br>    &quot;&quot;&quot;Create a new pull subscription on the given topic.&quot;&quot;&quot;<br>    # [START pubsub_create_pull_subscription]<br>    from google.cloud import pubsub_v1<br><br>    # TODO(developer)<br>    # project_id = &quot;your-project-id&quot;<br>    # topic_id = &quot;your-topic-id&quot;<br>    # subscription_id = &quot;your-subscription-id&quot;<br><br>    publisher = pubsub_v1.PublisherClient.from_service_account_file(service_account_file)<br>    subscriber = pubsub_v1.SubscriberClient.from_service_account_file(service_account_file)<br>    topic_path = publisher.topic_path(project_id, topic_id)<br>    subscription_path = subscriber.subscription_path(project_id, subscription_id)<br><br>    # Wrap the subscriber in a &#39;with&#39; block to automatically call close() to<br>    # close the underlying gRPC channel when done.<br>    with subscriber:<br>        subscription = subscriber.create_subscription(<br>            request={&quot;name&quot;: subscription_path, &quot;topic&quot;: topic_path}<br>        )<br><br>    print(f&quot;Subscription created: {subscription}&quot;)<br>    # [END pubsub_create_pull_subscription]<br><br><br><br><br>def create_push_subscription(project_id: str, topic_id: str, subscription_id: str, endpoint: str, service_account_file: str) -&gt; None:<br>    <br>    &quot;&quot;&quot;Create a new push subscription on the given topic.&quot;&quot;&quot;<br>    # [START pubsub_create_push_subscription]<br>    from google.cloud import pubsub_v1<br><br>    # TODO(developer)<br>    # project_id = &quot;your-project-id&quot;<br>    # topic_id = &quot;your-topic-id&quot;<br>    # subscription_id = &quot;your-subscription-id&quot;<br>    # endpoint = &quot;https://my-test-project.appspot.com/push&quot;<br><br>    publisher = pubsub_v1.PublisherClient.from_service_account_file(service_account_file)<br>    subscriber = pubsub_v1.SubscriberClient.from_service_account_file(service_account_file)<br>    topic_path = publisher.topic_path(project_id, topic_id)<br>    subscription_path = subscriber.subscription_path(project_id, subscription_id)<br><br>    push_config = pubsub_v1.types.PushConfig(push_endpoint=endpoint)<br><br>    # Wrap the subscriber in a &#39;with&#39; block to automatically call close() to<br>    # close the underlying gRPC channel when done.<br>    with subscriber:<br>        subscription = subscriber.create_subscription(<br>            request={<br>                &quot;name&quot;: subscription_path,<br>                &quot;topic&quot;: topic_path,<br>                &quot;push_config&quot;: push_config,<br>            }<br>        )<br><br>    print(f&quot;Push subscription created: {subscription}.&quot;)<br>    print(f&quot;Endpoint for subscription is: {endpoint}&quot;)<br>    # [END pubsub_create_push_subscription]<br><br><br><br><br>def create_cloudstorage_subscription(project_id: str, topic_id: str, subscription_id: str, bucket: str, service_account_file: str) -&gt; None:<br>    <br>    &quot;&quot;&quot;Create a new CloudStorage subscription on the given topic.&quot;&quot;&quot;<br>    # [START pubsub_cloudstorage_subscription]<br>    from google.cloud import pubsub_v1<br>    from google.protobuf import duration_pb2<br><br>    # TODO(developer)<br>    # project_id = &quot;your-project-id&quot;<br>    # topic_id = &quot;your-topic-id&quot;<br>    # subscription_id = &quot;your-subscription-id&quot;<br>    # bucket = &quot;my-bucket&quot;<br><br>    filename_prefix = &quot;log_events_&quot;<br>    filename_suffix = &quot;.avro&quot;<br>    # Either CloudStorageConfig.AvroConfig or CloudStorageConfig.TextConfig<br>    # defaults to TextConfig<br>    avro_config = pubsub_v1.types.CloudStorageConfig.AvroConfig(write_metadata=True)<br><br>    publisher = pubsub_v1.PublisherClient.from_service_account_file(service_account_file)<br>    subscriber = pubsub_v1.SubscriberClient.from_service_account_file(service_account_file)<br>    <br>    topic_path = publisher.topic_path(project_id, topic_id)<br>    subscription_path = subscriber.subscription_path(project_id, subscription_id)<br>    max_duration = duration_pb2.Duration()<br>    max_duration.FromSeconds(300)<br><br>    cloudstorage_config = pubsub_v1.types.CloudStorageConfig(<br>        bucket=bucket,<br>        filename_prefix=filename_prefix,<br>        filename_suffix=filename_suffix,<br>        avro_config=avro_config,<br>        # Min 1 minutes, max 10 minutes<br>        max_duration=max_duration,<br>        # Min 1 KB, max 10 GiB<br>        max_bytes=2000,<br>    )<br><br>    # Wrap the subscriber in a &#39;with&#39; block to automatically call close() to<br>    # close the underlying gRPC channel when done.<br>    with subscriber:<br>        subscription = subscriber.create_subscription(<br>            request={<br>                &quot;name&quot;: subscription_path,<br>                &quot;topic&quot;: topic_path,<br>                &quot;cloud_storage_config&quot;: cloudstorage_config,<br>            }<br>        )<br><br>    print(f&quot;CloudStorage subscription created: {subscription}.&quot;)<br>    print(f&quot;Bucket for subscription is: {bucket}&quot;)<br>    print(f&quot;Prefix is: {filename_prefix}&quot;)<br>    print(f&quot;Suffix is: {filename_suffix}&quot;)<br>    # [END pubsub_cloudstorage_subscription]<br></pre><h4>Send Messages</h4><pre><br>def publish_messages(project_id: str, topic_id: str, service_account_file: str) -&gt; None:<br>    <br>    &quot;&quot;&quot;Publishes multiple messages to a Pub/Sub topic.&quot;&quot;&quot;<br>    # [START pubsub_quickstart_publisher]<br>    # [START pubsub_publish]<br>    from google.cloud import pubsub_v1<br><br>    # TODO(developer)<br>    # project_id = &quot;your-project-id&quot;<br>    # topic_id = &quot;your-topic-id&quot;<br><br>    publisher = pubsub_v1.PublisherClient.from_service_account_file(service_account_file)<br>    # The `topic_path` method creates a fully qualified identifier<br>    # in the form `projects/{project_id}/topics/{topic_id}`<br>    topic_path = publisher.topic_path(project_id, topic_id)<br><br>    for n in range(1, 10):<br>        data_str = f&quot;Message number {n}&quot;<br>        # Data must be a bytestring<br>        data = data_str.encode(&quot;utf-8&quot;)<br>        # When you publish a message, the client returns a future.<br>        future = publisher.publish(topic_path, data)<br>        print(future.result())<br><br>    print(f&quot;Published messages to {topic_path}.&quot;)<br>    # [END pubsub_quickstart_publisher]<br>    # [END pubsub_publish]<br><br></pre><h4>Read Messages</h4><pre><br>def read_pubsub_messages(project_id: str, subscription_id: str, service_account_file: str):<br>    <br>    from google.cloud import pubsub_v1    <br>    # Initialize the Pub/Sub client<br>    subscriber = pubsub_v1.SubscriberClient.from_service_account_file(service_account_file)<br><br>    # Define the subscription path<br>    subscription_path = subscriber.subscription_path(project_id, subscription_id)<br><br>    def callback(message):<br>        # Process the received message<br>        print(f&quot;Received message: {message.data}&quot;)<br>        # Acknowledge the message to mark it as processed<br>        message.ack()<br><br>    # Subscribe to the specified subscription and start receiving messages<br>    streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)<br><br>    print(f&quot;Listening for messages on {subscription_path}...\n&quot;)<br><br>    # Keep the script running to continue receiving messages<br>    try:<br>        streaming_pull_future.result()<br>    except KeyboardInterrupt:<br>        streaming_pull_future.cancel()<br></pre><h3>Conclusion</h3><p>In this comprehensive guide, we’ve explored the world of Google Cloud Pub/Sub and its components, compared it with Pub/Sub Lite, and delved into the types of subscriptions available. Lastly we’ve also seen how to create/read and write on Pub/Sub.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=305a19901f1a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with BigTable using client libraries — Python]]></title>
            <link>https://medium.com/@mouaazfarrukh99/getting-started-with-bigtable-using-client-libraries-python-6cc97e7b6fad?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cc97e7b6fad</guid>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[big-table]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Sat, 09 Sep 2023 05:28:46 GMT</pubDate>
            <atom:updated>2023-09-09T05:28:46.910Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8AkKaZ2Uu62f-qx9i3gC7w.png" /><figcaption>Get Started with GCP’s BigTable using Python</figcaption></figure><h3>Getting started with BigTable using client libraries — Python</h3><p>Google Cloud Bigtable is a scalable and fully managed NoSQL database service designed for high-performance and low-latency applications.</p><p>In this article, we’ll walk you through the basics of creating, populating, and reading data from a Bigtable instance using Python client libraries. Whether you’re building a high-traffic web application or an IoT platform, Bigtable can help you efficiently store and retrieve data.</p><h3>Section 1: Create a Bigtable Instance</h3><h4>Step 1: Set Up Google Cloud Project</h4><p>Before you can create a Bigtable instance, you need to have a Google Cloud project. If you don’t have one, you can create it on the Google Cloud Console.</p><h4>Step 2: Enable Bigtable API</h4><p>On the Google Cloud Console, enable the Bigtable API for your project. This API allows you to manage Bigtable instances programmatically.</p><h4>Step 3: Create a Bigtable Instance</h4><p>Now, using Python client libraries, you can create a Bigtable instance in your project using a service account json. This instance will serve as your data store.</p><pre>def create_cbt(project_id, instance_id, table_id,service_account_file,column_family):<br><br>    from google.cloud import bigtable<br><br>    # The client must be created with admin=True because it will create a table.<br>    client = bigtable.Client.from_service_account_json(service_account_file,project=project_id, admin=True)<br>    instance = client.instance(instance_id)<br><br>    print(&quot;Creating the {} table.&quot;.format(table_id))<br>    table = instance.table(table_id)<br><br>    print(&quot;Creating column family cf1 with Max Version GC rule...&quot;)<br><br>    # Define the column family names and GC rules for each<br>    column_families = {<br>        column_family[0]: column_family.MaxVersionsGCRule(2)}<br><br>    if not table.exists():<br>        table.create(column_families=column_families)<br>        print(&quot;Table {} created.&quot;.format(table_id))<br>    else:<br>        print(&quot;Table {} already exists.&quot;.format(table_id))</pre><p>Here I’m considering a column family of two, which are being passed in the function definition as column family list.</p><h3>Section 2: Populate Data into Bigtable</h3><h4>Step 4: Define Your Data Model</h4><p>Before you populate data into Bigtable, plan your data model. Define column families, column qualifiers, and row keys according to your application’s needs. Best practices for designing table schemas can be found at: <a href="https://cloud.google.com/bigtable/docs/schema-design">Schema design best practices</a></p><h4>Step 5: Write Data to Bigtable</h4><p>Using Python, write code to insert data into your Bigtable instance. You can use the Bigtable Python client library to achieve this.</p><pre><br>def write_data_to_cbt(project_id, instance_id, table_id,service_account_file, row_key, data_dict):<br>            <br>    from google.cloud import bigtable<br>    # Create an empty dictionary to store the combined data<br><br>    &#39;&#39;&#39;<br>    data_dict = {<br>    &quot;A&quot;: &quot;2023-09-04T12:36:33.618545&quot;},<br>    &quot;B&quot;: &quot;2023-09-04T12:36:33.618545&quot;},<br>    &quot;C&quot;: &quot;2023-09-04T12:36:33.618545&quot;}}<br>    &#39;&#39;&#39;<br><br>    <br>    client = bigtable.Client.from_service_account_json(service_account_file,project=project_id, admin=True)<br>    instance = client.instance(instance_id)<br>    table = instance.table(table_id)<br>    <br>    # Row key defines the entry of the row at which we&#39;re loading data<br>    row = table.row(row_key)<br><br>    for tag_name, timestamp in data_dict.items():<br>        row.set_cell(&quot;time&quot;, Alphabets, timestamp)<br><br>    &#39;&#39;&#39;<br>    &gt;&gt; &quot;time&quot; is the column family<br>    &gt;&gt; Alphabets here are the columns <br>    &gt;&gt; timestamp is the value in dictionary<br>    &#39;&#39;&#39;<br>                <br>    table.mutate_rows([row])<br>    client.close()<br><br></pre><h3>Section 3: Read Data from Bigtable</h3><h4>Step 6: Retrieve Data by Row Key</h4><p>You can retrieve data from Bigtable using the row key. Implement code that reads data from your Bigtable instance based on specific row keys.</p><pre><br>def read_cbt_by_rw_key(project_id, instance_id, table_id, service_account_file, row_key):<br>    <br>    import google.cloud.bigtable<br><br>    client = google.cloud.bigtable.Client.from_service_account_json(service_account_file, project=project_id)<br>    instance = client.instance(instance_id)<br>    table = instance.table(table_id)<br>    <br>    # Read the entire row using the row key<br>    row = table.read_row(row_key)<br>    <br>    if row:<br>        # Retrieve all the data from the row<br>        cells = row.cells<br><br>        # Initialize dictionaries to store read data<br>        read_data = {&quot;time&quot;: {}, &quot;Alphabets&quot;: {}}<br><br>        # Iterate through cells and populate dictionaries<br>        for family, columns in cells.items():<br>            for column, cells_list in columns.items():<br>                for cell in cells_list:<br>                    value = cell.value.decode(&#39;utf-8&#39;)<br>                    if family == &quot;time&quot;:<br>                        read_data[&quot;time&quot;][column] = value<br>                    elif family == &quot;Alphabets&quot;:<br>                        read_data[&quot;Alphabets&quot;][column] = value<br><br>        # Close the Bigtable client<br>        client.close()<br><br>        return read_data<br><br>    else:<br>        print(f&quot;Row with key {row_key} does not exist.&quot;)<br>        client.close()<br>        return None</pre><h4>Step 7: Query Data by Column Family and Qualifier</h4><p>Bigtable allows you to query data efficiently by column family and qualifier. Learn how to use Python to retrieve data from specific columns.</p><h4>Conclusion:</h4><p>In this article, we’ve covered the fundamentals of working with Google Cloud Bigtable using Python client libraries. You’ve learned how to create a Bigtable instance, populate it with data, and read data based on row keys, column families, and qualifiers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cc97e7b6fad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Hybrid Solution for container based workloads on AWS]]></title>
            <link>https://awstip.com/designing-a-hybrid-solution-for-container-based-workloads-on-aws-387c6941dfeb?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/387c6941dfeb</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-system-manager]]></category>
            <category><![CDATA[aws-storage-gateway]]></category>
            <category><![CDATA[hybrid-solution]]></category>
            <category><![CDATA[aws-rds]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Wed, 31 May 2023 17:49:16 GMT</pubDate>
            <atom:updated>2023-06-02T11:07:52.014Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t0ivrDNGzeSWWZMvIqTBAg.png" /></figure><p>A customer use case where an enterprise insurance company needs to migrate some of their container workloads to AWS by using a hybrid model.</p><h3>Learning Outcomes:</h3><ul><li>How to design hybrid solutions on AWS</li><li>Evaluate when to use Amazon Relational Database Services</li><li>Differentiate between Amazon ECS &amp;Amazon ECS Anywhere</li><li>Ways to migrate container workloads to AWS using hybrid model</li><li>Learn different ways the solution could be optimized using container-based workloads</li></ul><h3>Use Case:</h3><p>Design a hybrid solution using AWS container services (ECS/EKS) for running containerized workloads. Host a PostgreSQL database on AWS using Amazon RDS. Utilize common orchestration tools, Implement resilience measures like multi-AZ deployments, backups, and proper monitoring for high availability, flexibility, scalability, and resilience for container-based workloads while integrating with on-premises infrastructure.</p><h3>User Requirements:</h3><p>The requirements that our user has requested are listed below:</p><ul><li>Run Containers on AWS</li><li>Host a PostgreSQL database on AWS</li><li>Use common tooling for orchestration across environments</li><li>Store on-prem data on AWS with minimal refactoring</li><li>Optimize for resilience</li></ul><p>Considering the customer don’t want to rewrite applications as much as possible, we’ll use <strong>Pure lift and shift</strong> for this migration as much as we can. Since we’re dealing with containers and due to their portable nature ther are able to run the same on AWS as they do on premises.</p><h3>Architecting Solution:</h3><p>Considering that we have two sources for our workloads i.e the on-prem and AWS cloud the architecture needs to set in place can be depicted as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*v2tsNSP3qSqXnIafXEJN1w.png" /><figcaption>Architecting Hybrid Solutions</figcaption></figure><h4>1. Hybrid Networking and Connectivity Services:</h4><ul><li>Public Internet</li><li>AWS VPN (Virtual Private Network)</li><li>AWS Direct Connect</li></ul><p>Considering that we know our customer is going to send a high volume of traffic to AWS, and they want reliability with the throughput for this connection, I think AWS Direct Connect will be the right choice for this architecture. Its is a service that provides a hosted private connection to AWS through a Direct Connect Delivery Partner or through AWS. Read more at <a href="https://aws.amazon.com/directconnect/">AWS Direct Connect</a>.</p><h4>2. Running containers on AWS &amp; NAT Gateways:</h4><p>AWS provides a number of ways to run container workloads that are:</p><ul><li>Amazon EKS</li><li>Amazon ECS</li><li>Amazon EC2 Powered Clusters</li><li>Amazon Fargate Powered Clusters</li></ul><p>Considering that our customer wants want something easy to use that supports lots of containers running at the same time and want to provide their own image for the cluster servers, and also want to be able to SSH into servers whenever they want.</p><p>The best is to use ECS as the orchestration tool, and then second, use EC2 instances for the cluster. ECS will go here in the AWS Region but not in the VPC, and then, EC2 goes in the VPC. Now, these EC2 instances will be in a private subnet because we are told by customer that these are private workloads. This means that these instances will have no internet connectivity, either ingress or egress.</p><p>Now since there is not a path to the internet from our private server. Because of that, add a NAT instance, or NAT gateway. A NAT resource, which stands for Network Address Translation, would sit in a public subnet and would have access to the internet. They are selected based on cost, scalability and reliability</p><h4>3. Amazon Relational Database Service:</h4><p>The best service in our case is Amazon RDS. Whenever you create an RDS DB instance, you select which database engine you want to have installed and configured on that instance. We’ll be having RDS on either side of the private subnet and due to data migration between the two platforms we would be using AWS DMS (Database Migration Service). Read more about <a href="https://aws.amazon.com/rds/">Amazon RDS</a> and <a href="https://aws.amazon.com/dms/">AWS DMS</a>.</p><h4>4. Where Should Our Customer Store Their Data?</h4><p>Some storage services that come to mind when designing the solution are:</p><ul><li>Amazon S3 Storage</li><li>Amazon EFS, FSx, EBS</li><li>AWS Storage Gateway</li></ul><p>You can read their respective documentation from AWS however the most suitable for our case will be <strong>AWS Storage Gateway</strong>. It is a service that provides on-prem applications access to cloud storage. There are three different services outlined under the Storage Gateway umbrella. First, there is File Gateway, and this will provide either NFS or SMB protocol. Then, Storage Gateway also provides Volume Gateway, which is for the iSCSI protocol. And finally, there is Tape Gateway, which is for the iSCSI VTL protocol. This will be added on both components of the hybrid workloads.</p><h4>5. Hybrid Solutions on AWS:</h4><p>Since our customer wants to use the same operational tools across AWS and on premises, where possible. Luckily, AWS has a lot of different services that work for a hybrid deployment, like the ones that we are using here:</p><ul><li>Amazon ECS Anywhere for Compute</li><li>Amazon Storage Gateway for Storage</li><li>AWS Backup for Backup across platforms</li><li>Direct Connect for Networking</li><li>AWS Systems Manager for Management</li></ul><p>For containers running in a hybrid deployment like our customer has, there are two major choices for container orchestration tools, Amazon ECS and Amazon EKS. We have already chosen Amazon ECS for our customer’s containers in the cloud, but AWS also has services that provide consistent tooling for managing containers on premises, as well as in the cloud. These services are <strong>Amazon ECS Anywhere</strong> and Amazon EKS Anywhere. With these two services, you can reduce support costs for running containers in a hybrid environment, and you can reduce any redundant efforts for managing these containers. So go with ECS Anywhere to our on-prem part.</p><p>Then there is <strong>AWS Backup</strong>, which centralizes data-protection management and compliance for your applications running on AWS or in hybrid environments. Using AWS Backup, you can protect VMware workloads running on premises and on AWS, as well as data stored on AWS Storage Gateway volumes. Our customer will benefit from AWS Backup to manage their backups across environments.</p><p>Now the <strong>AWS Systems Manager</strong> is a secure, end-to-end management solution for hybrid cloud environments. What Systems Manager is all about, is helping you manage and operate your resources at scale. For example, let’s say I have a massive fleet of servers, both on AWS and on premises, and I need to run the same script on all of these servers. I could, of course, do this all manually. Log in to each instance, run the script, record that I ran the script, and then, do it again and again until it’s all done. The issue with this, is that it is one, time consuming; and two, prone to human error. This is revolved by AWS Systems Manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*jBWAxVtkcz9ofzVkyrO1aQ.png" /><figcaption>Services Selected in Architecture</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*JS6NGZG_M0QpNt2kWzDDpw.png" /><figcaption>Digital Model for the Solution</figcaption></figure><h3>Conclusion:</h3><p>In conclusion, our customer can integrate AWS seamlessly with their on-premises infrastructure. By using AWS container services like ECS, database on Amazon RDS. Additionally, by incorporating AWS Storage Gateway for storing on-prem data and using ECS Anywhere and AWS Backup for consistent tooling and backup management, the customer can efficiently migrate their container workloads to AWS while minimizing refactoring efforts. Lastly for complete management of the hybrid environment, AWS Systems Manager can be used. For more visit the following resources:</p><ul><li><a href="https://www.coursera.org/learn/architecting-solutions-on-aws/home/week/4">Architecting Solutions on AWS</a></li><li><a href="https://www.techtarget.com/searchdatacenter/tip/Hybrid-cloud-infrastructure-design-for-AWS-Azure-Google-Anthos">Hybrid cloud infrastructure design for AWS, Azure, Google Anthos</a></li><li><a href="https://www.youtube.com/watch?v=BLFBJWA0RoI">Learn to deploy Containerized applications in hybrid cloud env</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=387c6941dfeb" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/designing-a-hybrid-solution-for-container-based-workloads-on-aws-387c6941dfeb">Designing a Hybrid Solution for container based workloads on AWS</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing Data Analytics Solution on AWS]]></title>
            <link>https://medium.com/@mouaazfarrukh99/designing-data-analytics-solution-on-aws-c1e9aac4d525?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1e9aac4d525</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[athena]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[kinesis-firehose]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Mon, 15 May 2023 16:46:32 GMT</pubDate>
            <atom:updated>2023-05-15T16:52:47.612Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BgkHF8OpSYxiZqhQKqPaQ.png" /><figcaption>Designing Data Analytics Solution on AWS</figcaption></figure><p>A customer use case where a software company needs a data analytics solution in AWS so that they can derive insights from clickstream data.</p><h3>Use Case:</h3><p>The company uses an S3 static website to host an HTML webpage containing restaurant menus. They have also developed a solution, where restaurant admins can log in to a system and update that object in the bucket. In addition, they have a solution to generate the QR codes pointing to the bucket, and they provide a self-service system for the restaurant admins to copy the QR code image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*7m8XPLT1d9eIhX1h7HvqWA.png" /></figure><p>They already have written a JavaScript library to collect the clickstream information. So, they just need a RESTful HTTPS endpoint to push that data.</p><h3>User Requirements:</h3><ul><li>Provide a HTTPS endpoint for Data Ingestion</li><li>Prefer using AWS Managed services</li><li>Services with Per-Usage Billing</li><li>Have Cross-Region data replication &amp; encryption</li><li>Use different storage classes to save on cost &amp; encryption</li></ul><p>We’ll be designing our solution that closely considers these requirements and come up with the most optimal way for fulfilling them.</p><h3>Architecting Solution:</h3><h4>1. Selecting a Storage Layer:</h4><p>AWS offer data services to transform data while it’s being ingested, or after it has arrived. In our scenario, the customer will be using a JavaScript library to produce data, and they are controlling how that data is sent. That makes things simpler because we would just need a service to ingest and store data as-is. The best way to choose from a variety of services by AWS, is by understanding what each service does and you can find that on <a href="http://aws.amazon.com/big-data/datalakes-and-analytics">Analytics on AWS</a>.</p><p>We have three most suitable services in our scenarios which are:</p><ul><li><a href="https://aws.amazon.com/ebs/">Amazon EBS</a></li><li><a href="https://aws.amazon.com/efs/">Amazon EFS</a></li><li><a href="https://aws.amazon.com/s3/">Amazon S3</a></li></ul><p>We won’t be using Amazon EBS or EFS, because they need an EC2 instance which does not align with our requirements for per-usage billing moreover EBS also does not provide Cross-region replication. However S3 is serverless and provides CRR therefore we’ll go with this. Read more at <a href="https://aws.amazon.com/s3/">Amazon S3</a>.</p><h4>2. Choosing a Service for Data Ingestion</h4><p>With S3 as the storage layer, we need to have a service for ingesting data into the bucket and that can be achieved by using one of the following services:</p><ul><li><a href="https://aws.amazon.com/data-exchange/">AWS Data Exchange</a></li><li><a href="https://aws.amazon.com/dms/">AWS DMS</a></li><li><a href="https://aws.amazon.com/emr/">Amazon EMR</a></li><li><a href="https://aws.amazon.com/kinesis/">Amazon Kinesis (Data Firehose, Data Streams, Data Analytics)</a></li></ul><p>Since there are no databases to be migrated, just clickstream data that needs to be sent therefore no need for <strong>DMS</strong> and as for <strong>AWS Data Exchange</strong>, it provides data catalogs, and is more suitable for integrating third-party data into a data lake for further analysis. <strong>Amazon EMR</strong> is somewhat good but It charges per time and requires some big data knowledge to operate therefore, we won’t be using it.</p><p><strong>Amazon Kinesis</strong> is a perfect match for this job. It is designed to ingest large amounts of small bits of data. Out of the three from Kinesis we’ll be using <strong><em>Kinesis Data Firehose</em></strong> as it does not requires writing some additional code which will be convenient for our customer.</p><p>Now, the thing is, in order to interact with Amazon Kinesis, we need to perform Kinesis API calls, such as PutRecord, regardless whether we are using Kinesis Data Firehose or Kinesis Data Streams.</p><p>The customer already has a JavaScript library that they wants to use, and that JS library is looking to make an HTTPS POST. You cannot do that directly to Kinesis. You have to use one of the Kinesis API calls. We’ll be using <strong>API Gateway</strong> to proxy these HTTPS requests to Kinesis in order to avoid exposing our Kinesis Data Firehose delivery streams over the internet. We could also use <strong>Amazon Cognito</strong> to do so but It would provide an identity pool where you can use AWS credentials in order to perform Kinesis API operations.</p><h4>4. Accessing the Ingested Data</h4><p>Separating data storage from processing is important because we can then use the right tool for the job. In our case, we want to process clickstream data that sits in S3 with large amounts of small files. We have the following services offered by AWS</p><ul><li><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html">Amazon S3 Select</a></li><li><a href="https://aws.amazon.com/glue/">AWS Glue</a></li><li><a href="https://aws.amazon.com/emr/">Amazon EMR</a></li><li><a href="https://aws.amazon.com/athena/">Amazon Athena</a></li></ul><p><strong>S3 Select</strong> processes one file/query at a time which is not feasible in our solution. <strong>AWS Glue</strong> or <strong>Amazon EMR </strong>are strong candidates for data processing, but they make the architecture somewhat complex which we don’t want. You can read more about them on the above links.</p><p><strong>Amazon Athena</strong> is our best choice as it starts querying data instantly. Get results in seconds. Pay only for the queries you run. Athena makes it easy for anyone with SQL skills to quickly analyze large-scale datasets.</p><h4>4. Visualizing the Data</h4><p>We need to add a data visualization layer to complete our solution. The visualization layer needs to connect with Amazon Athena, which is our query service. There are multiple data visualization options in AWS. Among them, candidates for our needs are</p><ul><li><a href="https://aws.amazon.com/cloudwatch/">Amazon CloudWatch</a></li><li><a href="https://aws.amazon.com/opensearch-service/">Amazon OpenSearch Service</a></li><li><a href="https://aws.amazon.com/grafana/">Amazon Managed Grafana</a></li><li><a href="https://aws.amazon.com/quicksight/">Amazon QuickSight</a></li></ul><p><strong>Amazon CloudWatch</strong> offers dashboards but it is primarily a monitoring service for operational metrics. <strong>OpenSearch Service</strong> would also a dashboard but the cost of its cluster would not make it feasible here. <strong>Grafana</strong> really shines when you need to visualize time series data and since we have clickstream data which has timestamped data, so Grafana can be a fit in our case.</p><p>However <strong>QuickSight</strong>, which is a data visualization service integrates well with Athena making it an ideal candidate for our architecture. It has no servers to manage, offers a pay-per-use pricing model, provides native AWS service integrations with built-in security, also enabling data insights. Therefore I would be going with QuickSight.</p><h3>Solution Overview:</h3><p>The solution leverages various AWS services to meet the customer’s requirements for a data analytics solution.</p><ol><li><strong>Amazon S3</strong> is used as the scalable and highly available storage layer to store both structured and unstructured data.</li><li>Data ingestion is handled by <strong>Amazon Kinesis Data Firehose</strong>, which receives the clickstream data and stores it in the S3 bucket.</li><li>For processing <strong>Amazon Athena</strong> is chosen, allowing for efficient querying of large-scale datasets. It provides fast results and cost-effective pricing based on the queries executed.</li><li><strong>Amazon QuickSight</strong> is selected for visualization, offering seamless integration with Athena and enabling the creation of interactive dashboards and visualizations for deriving insights from the data.</li></ol><p>The architecture also includes <strong>AWS API Gateway</strong> to securely proxy HTTPS requests from the customer’s JavaScript library to Kinesis. The following block diagram showcases the solution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*zudXqHfcNwGVYEEIUBq1pg.png" /><figcaption>Designing Data Analytics Solution in AWS</figcaption></figure><p>Overall, the solution provides a scalable, highly available, and cost-efficient data analytics platform for the customer’s clickstream data analysis needs.</p><h3>Conclusion:</h3><p>The designed AWS architecture provides a scalable and highly available solution for ingesting, storing, processing, and visualizing data. It utilizes Amazon S3 for storage, Amazon Kinesis Data Firehose for data ingestion, Amazon Athena for data processing, and Amazon QuickSight for data visualization. This end-to-end workflow enables the customer to derive valuable insights from their data.</p><ul><li><a href="https://aws.amazon.com/s3/">Amazon S3</a>, <a href="https://aws.amazon.com/kinesis/">Data Firehose</a>, <a href="https://aws.amazon.com/athena/">Athena</a>, <a href="https://aws.amazon.com/quicksight/">QuickSight</a></li><li><a href="https://docs.aws.amazon.com/solutions/latest/automated-data-analytics-on-aws/design-considerations.html">AWS Design considerations</a></li><li><a href="https://www.youtube.com/watch?v=u_8pKVPviE8">Build a Big Data Analytics Pipeline Using Modern Data Architecture</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1e9aac4d525" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Serverless Web Backend on AWS]]></title>
            <link>https://medium.com/@mouaazfarrukh99/designing-a-serverless-web-backend-on-aws-b4f49902b05d?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4f49902b05d</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[lambda]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Fri, 12 May 2023 16:24:31 GMT</pubDate>
            <atom:updated>2023-05-12T16:24:31.018Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJnOpa0eRDjlSv0rka84CA.png" /><figcaption>Designing a Serverless Web Backend on AWS</figcaption></figure><p>A customer use case providing solution to an ecommerce company that needs to migrate a backend web service to AWS along with other needs.</p><h3>Use Case:</h3><p>A company has been working on moving their systems to AWS over the last few years, and already have a decent amount of things on AWS, but have just moved workloads to AWS without much refactoring. Now, we have an opportunity to completely rewrite our next workload as part of a migration.</p><h3>User Requirements:</h3><p>The company has asked you to provide a solution on AWS that provides:</p><ul><li>Managed scaling for compute and database components as per demand</li><li>Decoupling components to maximize resilience to eliminate single point failure for the app.</li><li>Centralized monitoring &amp; logging to see the state of app at one place.</li><li>Optimizing cost, Performance efficiency, &amp; Operational overhead.</li></ul><p>For the above requirements, going <strong>serverless</strong> is one of the best options as it helps in scaling up and down as per demand and along with its well integrated with <strong>Amazon CloudWatch</strong> and its logs for monitoring and while doing all of it, serverless tends optimize everything in terms of cost and performance. Decoupling can be resolved by breaking the order service down into multiple components.</p><h3>Architecting Solution:</h3><h4>1. Selecting a Serverless Compute Service:</h4><p>To review a high-level overview of the different AWS compute services AWS, see <a href="https://aws.amazon.com/products/compute/">Compute on AWS — Compute for any workload</a>. Since we’ve decided to go serverless we have two options, <strong>AWS Fargate</strong> or <strong>AWS Lambda Function</strong>.</p><ul><li><strong>AWS Fargate</strong> is a technology that you can use with Amazon ECS to run containers without managing servers or clusters of EC2 instances. AWS Fargate reduces your need to provision, configure, or scale clusters of virtual machines to run containers. Thus, it also minimizes your need to choose server types, decide when to scale your clusters, or optimize cluster packing.</li><li><strong>AWS Lambda</strong> is a compute service that provides serverless compute functions that run in response to events or triggers. When an event or trigger is detected, a Lambda function is spun up in its own secure and isolated runtime environment, which is called an execution environment. When you use Lambda, you are responsible only for your code, which can make it easier to optimize for operational efficiency and low operational overhead.</li></ul><p>Considering that our client isn’t familiar with container technology, we’ll be using AWS Lambda function as serverless compute service.</p><h4>2. Choosing an AWS Database:</h4><p>For Database services on AWS, see <a href="https://aws.amazon.com/products/databases/">AWS Cloud Databases.</a>. Considering that we’re building a serverless solution therefore our best options is to go with either Amazon Aurora Serverless or Amazon DynamoDB.</p><ul><li><strong>Amazon Aurora Serverless</strong> is a relational database that is available through Amazon RDS. So, it’s kind-of under the Amazon RDS umbrella. Aurora is built for massive scale and for performance. This is really a database that is meant for large workloads that you would usually run on a commercial database that can offer the level of data storage and performance that you would need for enterprise applications.</li><li><strong>DynamoDB</strong> is a non-relational database option that is also designed for massive scale and high performance. DynamoDB is best used for applications that are looking to store key-value data, where the application has potentially a high number of concurrent users and connections to the database. DynamoDB is fully serverless, and it scales the underlying storage of your data under the hood.</li></ul><p>Now considering that we have a pretty simple table which can create, read, update, and delete orders based on an order ID, we can use DynamoDB because of the less complexity it offers as compared to Aurora Serverless.</p><h4>3. Building Event-Driven Architectures:</h4><p>Since the order-service code is all wrapped up into one application, if any part of that process breaks, the entire order fails, or some of the downstream calls to inventory, accounting, or fulfillment can break. Then, the next subsequent downstream call is not made. Part of our plan for the order service is to break it down into smaller, more decoupled components to solve these problems, while also meeting the requirements of improved resiliency, lower cost and operational overhead, as well as managed scaling. To solve we’ll be using Event-Driven Architecture as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*94aC-PzzNDiLCjJBnWkOzQ.png" /><figcaption>See Details at <a href="https://aws.amazon.com/event-driven-architecture/">Event Driven Architecture</a></figcaption></figure><p>We’ll be chosing Amazon SNS over AmazonEventBridge as EventBridge is a more complex and powerful tool than SNS. Both SNS and EventBridge are serverless, so we don’t need to worry about scaling here, and both services would help us decouple our order service and improve its resilience. So they both technically meet the requirements. However, SNS wins over EventBridge for cost, and it wins for simplicity. Considering our customer’s simple use case, SNS can handle the job very well.</p><h4>4. Decoupling AWS Solutions:</h4><p>To resolve customer’s issues with latency, we’ll be suggesting that they migrate to an asynchronous model, where the order is first stored, and then processed shortly after. With this model, end user can receive a response more quickly from the backend. When the backend system receives an order, it can immediately respond and then process the request asynchronously. For this we’ll be using <strong>Amazon SQS</strong> which is a fully managed, message queuing service that you can use to decouple and scale microservices, distributed systems, and serverless applications.</p><h3>Solution Overview:</h3><p>Knowing that are client is open to totally rewriting the code, they have the frontend clients making requests. All these requests are going to be directed to <strong>Amazon API Gateway</strong>, which acts as the front door for your API. Once it passes through the authentication and validation, API Gateway will then send the message to an <strong>SQS queue</strong>. The message will remain in the queue until an AWS Lambda function is spun up to process the message. This Lambda function will contain the application code for the order service related to order processing and storing the orders. The orders will be stored in an <strong>Amazon DynamoDB</strong> table. Once an order is stored in the table, an entry gets added to the <strong>DynamoDB stream</strong>, which then will need to be processed to send the order to the downstream functions. Another <strong>Lambda function</strong> comes to read the information on the DynamoDB stream and publish the order information to <strong>Amazon SNS, </strong>which followes a fan-out pattern that will send that message to all of the subscribed endpoints services such as fulfillment, accounting, and inventory. Following is the arhitecture of our solution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*XVCEdDSfBd0f4zSbfacL4Q.png" /><figcaption>Designing a Serverless Web Backend on AWS</figcaption></figure><h3>Conclusion:</h3><p>This article helps you learn how can we a managed scaling for compute and database components as per demand, Decouple components to maximize resilience to eliminate single point failure for the app and do that while having a centralized monitoring &amp; logging with optimizing cost, Performance efficiency, &amp; Operational overhead. For more resources, check out the following links:</p><ul><li><a href="https://www.coursera.org/learn/architecting-solutions-on-aws/">Architecting Solutions on AWS</a></li><li><a href="https://www.youtube.com/watch?v=GhZpSYQ6F9M">Build a Serverless Web App for a Theme Park</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4f49902b05d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a CI/CD pipeline in AWS]]></title>
            <link>https://medium.com/@mouaazfarrukh99/build-a-ci-cd-pipeline-in-aws-f143f7052575?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/f143f7052575</guid>
            <category><![CDATA[aws-code-deploy]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-codepipeline]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Fri, 14 Apr 2023 15:03:33 GMT</pubDate>
            <atom:updated>2023-04-14T15:03:33.072Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NGLZ6b3GuY8v3Sn0eCNrng.png" /><figcaption>Build a CI/CD pipeline in AWS</figcaption></figure><p>A basic overview of the steps involved in building a CI/CD pipeline in AWS.</p><p>A CI/CD pipeline in AWS is a continuous integration and continuous delivery pipeline that automates the software release process. It consists of a series of steps that include building, testing, and deploying code changes to production. Some basic steps to build a pipeline are given below:</p><h4>1. Create a Code Repository</h4><p>Store your application code in a version control system such as Git. To create a code repository and store your application code in a version control system such as Git, you can follow these steps:</p><ol><li>Sign in to your AWS account &amp; go to the AWS Management Console.</li><li>Select the AWS service that you want to use to host your Git repository. For example, you can use AWS CodeCommit, which is a fully-managed source control service that makes it easy to host Git repositories.</li><li>Create a new Git repository by following the steps provided in the AWS CodeCommit console. You’ll need to provide a name for your repository and configure any relevant settings, such as permissions and access controls.</li></ol><pre>aws codecommit create-repository — repository-name MyNewRepository — repository-description </pre><p>“This is a new Git repository” — region us-east-1</p><p>Once your repository is created, you can clone it to your local machine using Git. You’ll need to install Git on your machine if you haven’t already done so. For example, if your repository URL is <a href="https://git-codecommit.us-east-1.amazonaws.com/v1/repos/MyNewRepository">https://git-codecommit.us-east-1.amazonaws.com/v1/repos/MyNewRepository</a>, you would run the following command:</p><pre>git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/MyNewRepository</pre><p>Commit your application code to your local Git repository and push your changes to the remote repository hosted on AWS.</p><h4>2. Setup a Build Server</h4><p>Use an AWS service such as AWS CodeBuild or Jenkins to compile, test, and package your code. Here are the steps you can follow to build a build server using AWS CodeBuild:</p><ol><li>Start by creating a new CodeBuild project in the AWS Management Console. You will need to specify the build settings, such as the build environment, source code location &amp; build commands. Resources on <a href="https://docs.aws.amazon.com/codebuild/latest/userguide/create-project.html">docs.aws.amazon.com/codebuild/latest/userguide/create-project.html</a></li><li>Set up triggers that automatically start a build when new code is pushed to the repository. Use AWS CodePipeline or other CI/CD tools for this. Resources on docs.aws.amazon.com/codebuild/latest/userguide/trigger-create.html</li><li>Monitor Build Progress by using the AWS CodeBuild console. Resources on docs.aws.amazon.com/codebuild/latest/userguide/monitoring-builds.html</li></ol><h4>3. Create a Deployment Package:</h4><p>Use AWS CodeDeploy to create a deployment package that contains your application code, configuration files, and deployment scripts.</p><ol><li>Create a deployment group in AWS CodeDeploy.</li><li>Create a deployment package that contains your application code, configuration files, deployment scripts.</li><li>Upload the deployment package to Amazon S3.</li><li>Create a deployment in AWS CodeDeploy. Specify the deployment group that you created in step 1, and the Amazon S3 bucket and key that contain the deployment package.</li><li>Monitor the deployment using AWS CodeDeploy console or AWS CLI.<br>Verify that your application is running correctly by using AWS CloudWatch Logs to view application logs and troubleshoot any issues.</li></ol><p>For more resources, visit docs.aws.amazon.com/codedeploy/latest/userguide/deployments-create.html</p><h4>4. Defining and Configuring Pipelines:</h4><p>Use AWS CodePipeline to define a pipeline that automates the build, test, and deployment process.</p><ol><li>Start by creating a source stage in AWS CodePipeline. Specify the source code location, such as a code repository in AWS CodeCommit, GitHub.</li><li>Now create a build stage in CodePipeline. Specify the build provider, such as AWS CodeBuild, and configure the build settings.</li><li>Create a test stage in CodePipeline. Specify the testing provider, such as AWS CodeBuild, and configure the test settings.</li><li>Create a deployment stage in CodePipeline. Specify the deployment provider, ie AWS CodeDeploy, and configure the deployment settings.</li><li>Configure the transitions between stages in the pipeline. You can configure the pipeline to move to the next stage automatically when the previous stage is complete, or you can configure manual approval steps to ensure that the pipeline moves forward only when the changes are approved.</li><li>Test the pipeline by making a change to the source code repository. Verify that the pipeline runs automatically, builds the code, runs the tests, and deploys the changes to the environment.</li><li>AWS CodePipeline provides detailed information about the status of the pipeline. You can monitor the pipeline, using AWS CodePipeline console or the AWS CLI.</li></ol><p>For more guidance, visit the following resource <a href="https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html">docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html</a></p><h4>Conclusion</h4><p>This article emphasizes the role Git and of AWS services like CodeBuild, CodeDeploy, and CodePipeline to automate the build, test, and deployment processes. Overall, the article is a useful starting point for developers looking to build a CI/CD pipeline in AWS, although the specific steps involved may vary depending on the project requirements</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f143f7052575" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get started with EC2 in AWS]]></title>
            <link>https://medium.com/@mouaazfarrukh99/get-started-with-ec2-in-aws-5e9b4292b73a?source=rss-f7772e0e7277------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e9b4292b73a</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Mouaaz Farrukh]]></dc:creator>
            <pubDate>Sat, 08 Apr 2023 17:58:29 GMT</pubDate>
            <atom:updated>2023-04-08T18:03:06.070Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pFo2HD1OcaTJD8Xae950kw.png" /><figcaption>Get started with EC2 in AWS</figcaption></figure><p>Everything that you need to know regarding EC2 from setting it up to its various use cases in Cloud, DevOps, Data and Machine Learning domains.</p><h3><strong>What is EC2 instance?</strong></h3><p>Elastic Compute Cloud (EC2) instance is a virtual machine in AWS that allows you to rent computing resources on the cloud. It provides scalable computing capacity in the cloud and is designed to make web-scale cloud computing easier for developers. You can choose the type of instance you want based on your computing requirements, including the amount of memory, CPU power, storage, and networking capacity.</p><h3>Get Started with EC2:</h3><h4>1. Setup an AWS account:</h4><ol><li>Visit (<a href="https://aws.amazon.com/">https://aws.amazon.com/</a>) and click “Create an AWS Account”.</li><li>Enter your credentials such as Name, Password, Card Info etc.</li><li>For beginners, select AWS free tier provides a set of services for free.</li><li>Verify your account using the phone number or credit card info.</li><li>After verification, you will be prompted to create your first IAM user.</li></ol><p><em>- An IAM user is an identity that you can use to access AWS resources. You can create additional users later if needed.</em></p><p>6. Afterwards, you will be redirected to the AWS Management Console, where you can start using AWS services, a sample is given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8rnqvdxDCGoXS6pEZR3ZVg.png" /><figcaption>AWS Management Console</figcaption></figure><h4>2. Starting an EC2 instance:</h4><p>Once you’re in the management console, follow these steps:</p><ol><li>Navigate to the EC2 service.</li><li>In the EC2 Dashboard, click on “Instances” in the left-hand navigation menu.</li><li>Select the instance that you want to start from the list of instances.</li><li>Click on the “Actions” dropdown menu.</li><li>From the dropdown menu, select “Instance State” and then click on “Start.”</li></ol><p>You will be prompted to confirm that you want to start the instance. Click on “Yes, Start” to start the instance. The instance will take a few minutes to start up, and you can monitor its progress in the EC2 console. Once it’s up and running, you can connect to it and start using it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*16vI37S6-rbTgneLxB4QMg.png" /><figcaption>EC2 Dasboard on AWS</figcaption></figure><h3>Use Cases of EC2 Instances:</h3><h4>- Web/Application Hosting:</h4><p>EC2 instances can be used to host scalable web applications, websites, and web services. You can host apps by the following steps:</p><p>After launch an EC2 instance note its public IP in order to connect with instance using SSH. Update the package manager and install and start Apache using the following command:</p><pre>sudo yum update -y</pre><pre>sudo yum install httpd -y</pre><pre>sudo service httpd start</pre><p>Now you need to configure the web server in the following way:</p><pre>sudo nano /etc/httpd/conf/httpd.conf</pre><pre>&lt;VirtualHost *:80&gt;<br>    ServerName subdomain.example.com<br>    DocumentRoot /var/www/html/subdomain<br><br>    &lt;Directory &quot;/var/www/html/subdomain&quot;&gt;<br>        AllowOverride All<br>        Require all granted<br>    &lt;/Directory&gt;<br>&lt;/VirtualHost&gt;</pre><p>Configure the VirtualHost settings to serve your web application.</p><p>Now upload your web application using FTP, SCP, or other file transfer tools to upload your web application to the /var/www/html directory on the EC2 instance and configure your application</p><p>For example, if you are using WordPress, you would update the wp-config.php file to include the correct database name, username, and password:</p><pre>define( &#39;DB_NAME&#39;, &#39;database_name_here&#39; );<br>define( &#39;DB_USER&#39;, &#39;username_here&#39; );<br>define( &#39;DB_PASSWORD&#39;, &#39;password_here&#39; );</pre><p>Restart Apache, open a web browser and navigate to the public IP address of your EC2 instance to test your web application.</p><h4>- Big Data Processing:</h4><p>Install and configure the required software and tools on the EC2 instance. This may include Hadoop, Spark, Pig, and other big data processing frameworks by the following:</p><pre>sudo apt-get update <br>sudo apt-get install -y openjdk-8-jdk<br><br>// Install Hadoop<br><br>wget https://downloads.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz<br>tar -xzf hadoop-3.3.1.tar.gz<br>sudo mv hadoop-3.3.1 /usr/local/hadoop<br><br>// Install Spark<br><br>wget https://downloads.apache.org/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz<br>tar -xzf spark-3.1.1-bin-hadoop3.2.tgz<br>sudo mv spark-3.1.1-bin-hadoop3.2 /usr/local/spark<br><br>// Install Spark<br><br>wget https://downloads.apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz<br>tar -xzf pig-0.17.0.tar.gz<br>sudo mv pig-0.17.0 /usr/local/pig</pre><p>In case of Hadoop, edit its configuration files to configure HDFS. The main configuration files are located in the /usr/local/hadoop/etc/hadoop. Here is a sample hdfs-site.xml file that configures HDFS to use the /hadoop/data directory for storage after which it can be started:</p><pre>&lt;configuration&gt;<br>  &lt;property&gt;<br>    &lt;name&gt;dfs.replication&lt;/name&gt;<br>    &lt;value&gt;3&lt;/value&gt;<br>  &lt;/property&gt;<br>  &lt;property&gt;<br>    &lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;<br>    &lt;value&gt;/hadoop/data/namenode&lt;/value&gt;<br>  &lt;/property&gt;<br>  &lt;property&gt;<br>    &lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;<br>    &lt;value&gt;/hadoop/data/datanode&lt;/value&gt;<br>  &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p>Launch one or more EC2 instances and set up a cluster. This involves configuring the instances to work together as a single unit, using tools like Apache Ambari, Amazon EMR.</p><p>Submit your big data processing jobs to the cluster using a batch processing framework like Apache Hadoop or Apache Spark.</p><p>Monitor and manage the processing jobs using tools like YARN, Apache Mesos, or Amazon CloudWatch.</p><h4>- Container Orchestration</h4><p>Managing containers on EC2 involves several steps, including provisioning EC2 instances, installing and configuring container runtimes like Docker, creating and managing container images, and orchestrating containers using tools like Docker Compose or Kubernetes.</p><p>Here is a high-level overview of how to manage containers using EC2:</p><p><strong>Provision EC2 instances: </strong>The first step is to provision one or more EC2 instances on which the containers will run. You can use the Amazon EC2 console, AWS CLI, or AWS SDKs to create and manage instances.</p><p><strong>Install container runtime: </strong>Once you have provisioned the EC2 instances, you need to install and configure a container runtime like Docker on instances. Connect to each instance using an SSH client and run the following commands individually:</p><pre>sudo yum update -y                       // update the instance<br>sudo amazon-linux-extras install docker  // install docker<br>sudo service docker start                // start docker<br>sudo usermod -a -G docker ec2-user       // add user to use Docker<br>docker version</pre><p><strong>Create Container Images:</strong> With the container runtime installed, you can create container images that include the application code and dependencies. Create a new directory for your application by using the following commands:</p><pre>mkdir my-app <br>cd my-app</pre><p>Now create a file called Dockerfile in the directory “vi Dockerfile”:</p><pre>FROM node:14-alpine<br>WORKDIR /app<br>COPY package*.json ./<br>RUN npm install<br>COPY . .<br>EXPOSE 3000<br>CMD [ “npm”, “start” ]</pre><p>Build and Verify the Docker image using the Docker CLI by:</p><pre>docker build -t my-app-image         // Build images<br>docker images                        // Verify images</pre><p><strong>Push container images to a registry:</strong> Once you have created the container images, you can push them to a container registry like ECR</p><p><strong>Deploy and manage containers:</strong> Once the container images are in a registry, you can deploy them to the EC2 instances using container orchestration tools like Docker Compose or Kubernetes.</p><ul><li><strong>Install Kubernetes:</strong> Install Kubernetes on your EC2 instance by following the installation instructions for your operating system.</li><li><strong>Create a Kubernetes deployment:</strong> Create a Kubernetes deployment YAML file that defines the container image, configuration, and desired number of replicas. For example, the following deployment YAML file deploys a web application and a database:</li></ul><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-app-deployment<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: my-app<br>  template:<br>    metadata:<br>      labels:<br>        app: my-app<br>    spec:<br>      containers:<br>      - name: my-app-container<br>        image: my-registry/my-app-image:latest<br>        ports:<br>        - containerPort: 3000<br>      - name: db-container<br>        image: my-registry/postgres:latest</pre><p>Deploy and verify the deployments using the following command:</p><pre>kubectl apply -f DEPLOYMENT_FILE.yaml      // Deploy<br>kubectl get deployments                    // Verify</pre><p><strong>Manage container lifecycle:</strong> As the application changes and evolves, you may need to update the container images and deploy new versions of the application. Container orchestration tools like Kubernetes allow you to manage the container lifecycle, including rolling updates, scaling, and fault tolerance.</p><p>With these steps, you can create a scalable and reliable platform for deploying and managing containerized applications on EC2.</p><h3>Conclusion</h3><p>This article provides everything to have a basic understanding of EC2 instance and be aware of how its used in various scenarios.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e9b4292b73a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>