<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sara M. on Medium]]></title>
        <description><![CDATA[Stories by Sara M. on Medium]]></description>
        <link>https://medium.com/@mengineer?source=rss-b76a0fcaeed4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9S1ORG8fgD8VRjZ6N2Vchw.jpeg</url>
            <title>Stories by Sara M. on Medium</title>
            <link>https://medium.com/@mengineer?source=rss-b76a0fcaeed4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:26:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mengineer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Gift Is Never Really About the Gift]]></title>
            <link>https://medium.com/@mengineer/the-gift-is-never-really-about-the-gift-994d8da63890?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/994d8da63890</guid>
            <category><![CDATA[motherhood]]></category>
            <category><![CDATA[kids]]></category>
            <category><![CDATA[life]]></category>
            <category><![CDATA[mothers-day]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Mon, 13 Apr 2026 17:49:40 GMT</pubDate>
            <atom:updated>2026-04-13T17:49:40.784Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ACidbWu2RVejyi2Ta-GbKg.jpeg" /></figure><p>I still remember the first gift I ever gave my mother. I had no money — not a single coin. So I took a plain piece of paper, folded it in half, and made her a card with my own hands. I drew flowers on the front. I wrote words I meant with my whole chest. No glitter, no stickers, no envelope. Just a folded piece of paper and everything I felt.</p><p>She cried when she opened it.</p><p>She still has it.</p><p>That moment — that small, humble, handmade moment — taught me something I’ve never forgotten: a gift doesn’t need to cost anything to mean everything. What it needs is thought. Intention. The simple act of saying, <em>I made this for you, because I love you.</em></p><p>“A gift is a signal. It says: I was thinking about you, even when you weren’t in the room.”</p><h3>Why gifts matter more than we admit</h3><p>We live in a culture that’s become a little uncomfortable with sentimentality. We tell ourselves that gifts are materialistic, that presence matters more than presents. And while that’s sometimes true, there’s something we miss when we dismiss the act of giving altogether.</p><p>Giving a gift is an act of attention. It asks you to slow down and really think about someone — who they are, what they love, what would make them stop and smile. That effort is visible. People feel it, deeply, in the difference between something chosen with care and something grabbed without a second thought.</p><h3>Siblings: the relationship we take for granted</h3><p>Of all the bonds in our lives, the sibling relationship is perhaps the most quietly neglected. We grew up alongside our brothers and sisters — shared rooms, arguments, inside jokes, dinner tables. And then life pulls us apart. Jobs, cities, partners, kids. The calls get shorter. The visits become rare.</p><p>A gift to a sibling says what the phone calls often can’t: <em>I still see you. I still know you.</em> Not just the child I grew up with, but the person you’ve become. A small card, a thoughtful note — it can bridge months of distance in seconds.</p><h3>Your mother: the person who gave first</h3><p>There’s something strange that happens when we become adults: we forget to give back to the person who gave to us first, and most, and longest.</p><p>Our mothers spent years anticipating our needs before we could even name them. Warm food when we were sick. A hand on our back when we were scared. A presence so steady and constant that we stopped noticing it — the way you stop noticing the air until it’s gone.</p><p>Gifting your mother isn’t about repayment. That math never works. It’s about acknowledgment. It’s about placing something in her hands and saying, without words: <em>I see you. I see everything you’ve done. And I love you for all of it.</em></p><p>Even a card made from a folded piece of paper can say that. My mother proved it.</p><p>“The best gifts aren’t the biggest ones. They’re the ones that prove someone was paying attention.”</p><h3>Kids: teaching them what love looks like</h3><p>When you give a child a gift — really give it, with thought behind it — you’re doing something quietly powerful. You’re showing them what care looks like. You’re teaching them that the people who love you notice you. That celebration is a language, and it’s worth learning to speak it.</p><p>Children don’t remember the price tag. They remember the feeling. The warmth of someone going out of their way for them. Those small moments become the emotional blueprint they carry into every friendship, every relationship, for the rest of their lives.</p><h3>Thoughtful gifts don’t have to be expensive</h3><p>My paper card cost nothing. Literally nothing. And it was the most treasured gift my mother ever received from me — she told me so herself, years later, when I was old enough to understand what she meant.</p><p>Today, you don’t have to choose between thoughtful and affordable. Some of the most beautiful cards and gifts cost less than a cup of coffee. What matters isn’t the price — it’s that someone made them with care, and you chose them with intention.</p><p>If you’re looking for something to give your mother this season — something that carries real warmth without the heavy price tag — here are two I made myself, both around $3:</p><p>For mom · Card bundle</p><p>Floral Mother’s Day Card Bundle (English)</p><p><a href="https://www.etsy.com/listing/4487631587/floral-mothers-day-card-bundle-english">View on Etsy →</a></p><p>For mom · Single card</p><p>Purple Watercolor Mother’s Day Card</p><p><a href="https://www.etsy.com/listing/4487895083/purple-watercolor-mothers-day-card">View on Etsy →</a></p><p>Both are printable — meaning you can download, print, and have something beautiful in your hands today. No shipping wait. No excuses. Just a card that says everything.</p><p>Because sometimes the most meaningful thing you can give someone costs less than $3 — and means more than they’ll ever say out loud.</p><p>The people you love deserve to be celebrated. Not just on the big days, but on the ordinary Tuesdays when a small act of thoughtfulness can shift someone’s entire day.</p><p>Go give something. Not because you have to. Because you were thinking of them, even when they weren’t in the room.</p><p><em>— And that, more than anything, is what a gift really is.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=994d8da63890" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning AWS CodePipeline Hands-on learning: Part2]]></title>
            <link>https://towardsaws.com/learning-aws-codepipeline-hands-on-learning-part2-59c2b1b5a6e1?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/59c2b1b5a6e1</guid>
            <category><![CDATA[elastic-beanstalk]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[rds]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[codebuild]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 17:13:06 GMT</pubDate>
            <atom:updated>2024-06-10T19:18:27.510Z</atom:updated>
            <content:encoded><![CDATA[<h4>AWS CodeBuild, ElasticBeanstalk, WebApp, CICD</h4><h4>Automate the deployment of a web application using <strong>AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, ElasticBeanstalk and RDS</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gO2lwpxKHC1ONmFP8nAlvw.png" /><figcaption>Image by Author</figcaption></figure><p>In our previous tutorial, we explored how to automate a static website deployment process on AWS using AWS CodePipeline and AWS CodeCommit.</p><p>Today, we continue this journey by exploring another component of the AWS <strong>CI/CD</strong> ecosystem: <strong>AWS CodeBuild</strong>.</p><p>AWS <strong>CodeBuild </strong>is a fully managed build service that compiles source code, runs tests, and produces software packages ready for deployment.</p><p>In this article, we will demonstrate a very common scenario: How to automate the deployment of a web application made with flask framework using <strong>Git, AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, ElasticBeanstalk and RDS </strong>Database ?</p><p>This tutorial is designed to be clear and easy to follow. While it may appear lengthy, this is due to the inclusion of detailed screenshots for each step. This tutorial is intended for those who, like me, prefer written instructions over video tutorials, saving you valuable time. 😉</p><h3>Prerequisites</h3><ul><li>Git locally installed (versions 1.7.9 and later),</li><li>AWS account.</li></ul><h3>Table of contents</h3><p>· <a href="#3f73">Step1: CodeCommit</a><br>· <a href="#e5ce">Step2: CodeBuild</a><br><em> ∘ </em><a href="#8813"><em>Buildspec.yml file</em></a><em><br> ∘ </em><a href="#639e"><em>Create the build project</em></a><br>· <a href="#9993">Step3: Elastic Beanstalk</a><br>· <a href="#611e">Step4: AWS RDS</a><br>· <a href="#fbd4">Step5: Setting Necessary Security groups and permissions</a> <br><em> ∘ </em><a href="#495e"><em>Necessary permissions to access the AWS Secrets Manager:</em></a><em> <br> ∘ </em><a href="#67ff"><em>Security Group Configuration</em></a><em> </em><br> <em>∘ </em><a href="#0427"><em>VPC Configuration</em></a><br>· <a href="#be9f">Step6: CodePipeline</a> <br><em> ∘ </em><a href="#0061"><em>Choose pipeline settings</em></a><em><br> ∘ </em><a href="#4d9a"><em>Source Stage</em></a><em><br> ∘ </em><a href="#2cee"><em>Build Stage</em></a><em><br> ∘ </em><a href="#b267"><em>Deploy Stage</em></a><br>· <a href="#9f5a">Troubleshooting</a><br><em> ∘ </em><a href="#89a3"><em>IAM Role or Policy Issues</em></a><em><br> ∘ </em><a href="#8e52"><em>Database Connection Issues</em></a><em><br> ∘ </em><a href="#d5da"><em>VPC and Subnet Configuration Issues</em></a><br>· <a href="#4b8f">Conclusion</a></p><p>In this demo, we will use a login/register web application built with python and flask Framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*75EmqJWTprsD465kx9OTQw.png" /><figcaption>The project’s structure</figcaption></figure><p>Please find code in <a href="https://github.com/saraengineer/FlaskWebApp-cicd-aws-console">the github repository</a>.</p><p><a href="https://github.com/saraengineer/FlaskWebApp-cicd-aws-console">GitHub - saraengineer/FlaskWebApp-cicd-aws-console: How to automate the deployment of a web application made with flask framework using Git, AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, ElasticBeanstalk and RDS Database ?</a></p><h3>Step1: CodeCommit</h3><p>First, we will need to create a repository in CodeCommit and push our project. Please refer to the <a href="https://medium.com/towards-aws/learning-aws-codepipeline-hands-on-learning-part1-5768cde0f5c7">previous article</a> if you need step by step demo.</p><p><a href="https://towardsaws.com/learning-aws-codepipeline-hands-on-learning-part1-5768cde0f5c7">Learning AWS CodePipeline Hands-on learning: Part1</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9muIATamb3eL4ZMbLVd9aw.png" /></figure><h3>Step2: CodeBuild</h3><p>But, why do we need to use <strong>CodeBuild</strong>?</p><p><strong>AWS CodeBuild</strong> is the service that compiles source code, runs tests, and produces software packages that are ready to deploy.</p><p>In our case, we have a flask application that does not need to be compiled, what we need in this scenario is preparing the application for deployment, especially to install all the dependencies required by the application.</p><h4><strong>Buildspec.yml file</strong></h4><p>AWS CodeBuild use a yaml file to define the build process for the project. This file specifies the commands that CodeBuild should run during different phases of the <strong>build</strong>, the environment settings, and how to handle artifacts and logs.</p><p>Here is our yaml file:</p><pre>version: 0.2<br><br>phases:<br>  install:<br>    runtime-versions:<br>      python: 3.11<br>    commands:<br>      - pip install -r requirements.txt<br>  build:<br>    commands:<br>      - echo Build started on `date`<br>  post_build:<br>    commands:<br>      - echo Build completed on `date`<br>      - chmod +x start.sh<br><br>artifacts:<br>  files:<br>    - &#39;**/*&#39;</pre><p>As you can see, we did define 3 phases: install, build, and post_build phases.</p><ul><li>In the <strong>install phase</strong>, we specify the runtime environment “Python 3.1”and the commands to run during this phase where we’re gonna install the Python dependencies listed in requirements.txt.</li><li>In the <strong>build phase</strong>, since there is no code to be compiled, we are just printing a message indicating that the build has started, along with the current date.</li><li>In the <strong>post-build phase, </strong>we are going to change the permissions of the start.sh script to make it executable.</li></ul><h4>Create the build project</h4><p>Now, that the yaml file is ready, we just need to create a build project, but we are not doing it in this step, it will be a part of the CodePipeline creation process.</p><h3>Step3: Elastic Beanstalk</h3><p>But, why do we need to use <strong>AWS Elastic Beanstalk</strong>?</p><p>AWS Elastic Beanstalk is a platform as a service (PaaS) offered by Amazon Web Services that simplifies the deployment and management of applications. In our case, we will need an AWS Elastic Beanstalk instance to deploy our web application (used as a deploy provider, we will see that in the next steps).</p><p>So let’s start!</p><ul><li>Go to AWS <a href="https://eu-west-3.console.aws.amazon.com/elasticbeanstalk/">Elastic Beanstalk</a>, and create an environment, we will choose a web server, python as a platform and use single instance since it’s free.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UCmYTCU2ppy1eTqzpUnPXw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*VjSMkI80icv4pnbmOedrzQ.png" /></figure><ul><li>In service access, choose create and use a new service role and choose your key pair ( if you don’t have one, go to aws ec2 and create one).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*qHsZdVr_vnaD9JxLCIkdNQ.png" /></figure><p>In monitoring, choose a basic system,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*XaGkHp999SpCOhU6uXbARg.png" /></figure><p>And that’s it.</p><h3>Step4: AWS RDS</h3><p>But, why do we need to use <strong>AWS RDS</strong>?</p><p>Since we are using a register/login application, we need to store user info in a database, and we did choose AWS RDS to create a postgreSQL database for our demo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hK2HtmE5B6qntKbejSGAkQ.png" /><figcaption>mydb</figcaption></figure><p>I am not detailing this step since it’s easy.</p><p>But I need to precise that we will use AWS Secrets Manager secret for the database credentials.</p><p>That’s why we will use boto3 to retrieve user and password info.</p><pre>import os<br>import json<br>import boto3<br>from botocore.exceptions import ClientError<br><br>class Config:<br>    SECRET_KEY = os.urandom(24)<br>    SQLALCHEMY_TRACK_MODIFICATIONS = False<br><br>    @staticmethod<br>    def get_secret():<br>        secret_name = &quot;XXXXXXXXXXFGFDRTYJHGFGHJKJHGFGHJ&quot;<br>        region_name = &quot;eu-west-3&quot;<br><br>        # Create a Secrets Manager client<br>        session = boto3.session.Session()<br>        client = session.client(<br>            service_name=&#39;secretsmanager&#39;,<br>            region_name=region_name<br>        )<br><br>        try:<br>            get_secret_value_response = client.get_secret_value(<br>                SecretId=secret_name<br>            )<br>        except ClientError as e:<br>            raise e<br><br>        secret = get_secret_value_response[&#39;SecretString&#39;]<br>        secret_dict = json.loads(secret)<br>        db_username = secret_dict[&#39;username&#39;]<br>        db_password = secret_dict[&#39;password&#39;]<br>        db_host = &quot;mydb-xxxxxxxxxxxxxxxxxxxxxxxxxxx.amazonaws.com&quot;<br>        db_name = &quot;mydb&quot;<br>        return db_username, db_password, db_host, db_name<br>        print(db_name)<br><br>    db_username, db_password, db_host, db_name = get_secret()<br>    SQLALCHEMY_DATABASE_URI = f&quot;postgresql://{db_username}:{db_password}@{db_host}:5432/{db_name}&quot;</pre><h3>Step5: Setting Necessary Security groups and permissions</h3><h4>Necessary permissions to access the AWS Secrets Manager:</h4><p>Elastic Beanstalk EC2 instances need to have the necessary permissions to access the AWS Secrets Manager to retrieve the database credentials.</p><p>For that we need to add a policy to the aws-elasticbeanstalk-service-role role that allows it to call the “secretsmanager:GetSecretValue” action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YQls3IXifKuAdJUA8NZUEg.png" /></figure><h4><strong>Security Group Configuration</strong></h4><ul><li>We need to ensure that the<strong> security group associated with the RDS instance</strong> allows <strong>inbound traffic</strong> on port 5432 from the Elastic Beanstalk instance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yj497cvf9WJXuEw6MciQvg.png" /></figure><ul><li>Also ensure that the <strong>security group associated with the Elastic Beanstalk instance</strong> allows <strong>outbound </strong>traffic to the RDS instance on port 5432.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1q15ZqjjVVxKzeWdOFYnLw.png" /></figure><h4><strong>VPC Configuration</strong></h4><ul><li>Ensure that your RDS instance and Elastic Beanstalk environment are in the same VPC.</li><li>Verify that the subnet and routing configurations allow communication between the Elastic Beanstalk instance and the RDS instance.</li></ul><h3>Step6: CodePipeline</h3><h4><strong>Choose pipeline settings</strong></h4><p>We are going to name our pipeline and let other settings in default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6g9MvstqUJw6z1OlqjFqpw.png" /></figure><h4>Source Stage</h4><p>We will specify that we are using CodeCommit as a source provider, and select the repository and the branch we just created for our project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NTTSG979eLN0Tw9ftgr4lg.png" /></figure><h4>Build Stage</h4><p>In the build stage, we need to use AWS CodeBuild as a Build provider, choose a region and click on “create project” to be redirected to CodeBuild page to create our build project</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Q8BruDzrfOq1zJuuoZYrQ.png" /></figure><ul><li>Choose a name,</li><li>Choose the environment image where “The build commands will be executed”,</li><li>And choose “use buildspec file” and click next, to be redirected again to codepipeline.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fnt93lNKhRA0N7LiHffB0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tm9ix0bAYI2r0NffUxRS9g.png" /></figure><h4>Deploy Stage</h4><p>In deploy stage, we need to choose AWS Elastic Beanstalk as a deploy provider, choose the region as well as the applcation / environment we did create in Elastic Beanstalk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*giBus-72ijgQ1YzvEMViRQ.png" /></figure><p>And that’s it, your pipeline will be running!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*gUxs9OSKn2nPi4n45nrFXA.png" /></figure><p>Once, it’s completed, you can go to AWS Elastic Beanstalk to see in details the deployment of your application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x34ClXv4768ODSeeTJhYnQ.png" /></figure><p>And click on the endpoint:</p><p>If all good, you will see the home page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lTAFNXYJ8BYeCgEeZNVmgA.png" /><figcaption>Home Page</figcaption></figure><p>Let’s try to register:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*X42pdw-JPN74jrsnU0G2OA.png" /><figcaption>Register Page</figcaption></figure><p>Try to login with the regsitration information we just entered:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*vfN2xWNHmXy8u5Eti3JTDQ.png" /><figcaption>Login Page</figcaption></figure><p>And Tadda, we are login successfully!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*aPib5IzujXv3hmdx9cDxYA.png" /><figcaption>Connected</figcaption></figure><h3>Troubleshooting</h3><p>Here is a list of common issues you may encounter during the deployment and how you could resolve them.</p><p>First off all, you need to see logs to identify what cause the problem. You can go check logs in the logs tab of your Elastic Beanstalk environment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gGPU_MAj0SO2ZureZc5zog.png" /></figure><h4><strong>IAM Role or Policy Issues</strong></h4><p>If there is an issue with accessing AWS Secrets Manager in your web application, it will likely result in a 500 Internal Server Error.</p><ul><li><strong>Error:</strong> botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::123456789012:assumed-role/role-name/i-0abcd1234efgh5678 is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:region:123456789012:secret:secret-name</li><li><strong>Solution:</strong> Ensure that the IAM role attached to your Elastic Beanstalk instance has the necessary permissions to access AWS Secrets Manager.</li></ul><h4><strong>Database Connection Issues</strong></h4><p>If there is an issue with connecting to the RDS database, it will likely result in a 500 Internal Server Error.</p><ul><li><strong>Solution:</strong> Ensure that the RDS instance is accessible from the Elastic Beanstalk instance. Check security group settings to allow traffic on port 5432. Verify that the database credentials you are using in config.py.</li></ul><h4><strong>VPC and Subnet Configuration Issues</strong></h4><p>If there networking issues between Elastic Beanstalk and RDS it will likely result in a 500 Internal Server Error.</p><ul><li><strong>Solution: </strong>Ensure that both the Elastic Beanstalk environment and the RDS instance are in the same VPC. Verify that both the Elastic Beanstalk environment and the RDS instance are in the same subnet or have correct routing tables to allow communication between them. Ensure you have the necessary Security Groups (seen above).</li></ul><h3>Conclusion</h3><p>In conclusion, AWS CodeBuild, along with AWS CodePipeline, Elastic Beanstalk, and RDS, provides a powerful and seamless way to automate the deployment of web applications. By following the steps outlined in this tutorial, you can efficiently set up a CI/CD pipeline for a Flask-based web application.<br>With detailed screenshots and clear instructions, this guide is tailored for those who prefer written tutorials over videos, offering a step-by-step walkthrough that saves time and enhances understanding. Whether you are a beginner or an experienced developer, leveraging these AWS services will help streamline your development workflow and improve the overall efficiency of your deployment processes.</p><p>Happy deploying! 🚀</p><p>If you enjoyed this article or have any questions, please don’t hesitate to leave a comment. <strong>Your feedback is greatly appreciated</strong>!</p><p><strong>Clap </strong>my article 50 times 👏, that will really help me out and boost this article to others ✍🏻❤️. <strong>Follow </strong>me on <a href="https://medium.com/@mengineer">Medium</a> to get my latest article.</p><p>Thank you 🫶!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59c2b1b5a6e1" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/learning-aws-codepipeline-hands-on-learning-part2-59c2b1b5a6e1">Learning AWS CodePipeline Hands-on learning: Part2</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning AWS CodePipeline Hands-on learning: Part1]]></title>
            <link>https://towardsaws.com/learning-aws-codepipeline-hands-on-learning-part1-5768cde0f5c7?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/5768cde0f5c7</guid>
            <category><![CDATA[aws-codecommit]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws-codepipeline]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Sat, 01 Jun 2024 19:29:28 GMT</pubDate>
            <atom:updated>2024-06-02T13:31:03.590Z</atom:updated>
            <content:encoded><![CDATA[<h4>AWS CodeCommit</h4><h4>Create a Pipeline from CodeCommit to deploy your Static Website</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*88xqSj_RH70dJuRTjdg80g.png" /><figcaption>Image By <a href="https://medium.com/@mengineer">Author</a></figcaption></figure><p>Today, I decided to start a series of tutorials on a highly relevant topic, especially with the increasing use of AWS services: <strong>How to automate your software deployment process on AWS using AWS CodePipeline?<br>Automating the deployment is crucial for several reasons</strong>. Manual deployments can be <strong>time-consuming and very hard</strong>, often leading to inconsistencies and errors. Automation increases efficiency, reduces the risk of human error, and ensures consistency across deployments. By automating the deployment process, you can focus more <strong>on developing features and less on managing deployments</strong>, ultimately <strong>accelerating your development cycle and improving the reliability of your applications</strong>.</p><p>In this article, we will demonstrate how to automate the deployment of a static website using <strong>Git, AWS CodeCommit, AWS CodePipeline, and an S3 bucket</strong>. The tutorial is designed to be clear and very easy to follow, guiding you step-by-step through the process.</p><h3>Prerequisites</h3><ul><li>Git locally installed (versions 1.7.9 and later),</li><li>AWS account.</li></ul><h3>Set up CodeCommit</h3><p>The first step is creating a CodeCommit repository and pushing the code from the local Git repository to AWS CodeCommit.</p><ul><li>Go to CodeCommit page to create a repository. Give your repository a name and a description (optional).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SvBx3eeGCoMy7Z6ydmS9mw.png" /></figure><p>And that’s it, you will be redirected to this page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oREIUCPscT6uJuDw-gkhEw.png" /></figure><p>You can see that our repository is ready.</p><ul><li>To connect to it, we can use HTTPS, SSH or the HTTPS (GRC) protocol.</li></ul><p>In this demo, we’re going to use HTTPS protocol.</p><ul><li>To do that, we need first to set up the <a href="https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-gc.html?icmpid=docs_acc_console_connect_np">HTTPS Git credentials for AWS CodeCommit</a> in the IAM page.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*riQBWFb3qdpw3Rw5LsUICA.png" /></figure><ul><li>Config git in local</li></ul><p>First, create an index.html page for the demo.</p><p>I am using this sample html page:</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html lang=&quot;en&quot;&gt;<br>&lt;head&gt;<br>    &lt;meta charset=&quot;UTF-8&quot;&gt;<br>    &lt;title&gt;Index Demo website&lt;/title&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>    &lt;header&gt;<br>        &lt;h1 style=&quot;color:rgb(194, 13, 140)&quot;&gt;Welcome to Our Website!&lt;/h1&gt;<br>    &lt;/header&gt;<br>    &lt;main&gt;<br>        &lt;article&gt;<br>            &lt;p&gt;This is a demo website.&lt;/p&gt;<br>        &lt;/article&gt;<br>    &lt;/main&gt;<br>    &lt;footer&gt;<br>        &lt;p&gt;&amp;copy; V1 version My Website. All rights reserved.&lt;/p&gt;<br>    &lt;/footer&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>Then open your terminal in the same directory.</p><p>To initialize the local git repository, we run the command:</p><pre>git init</pre><p>To add our code, run :</p><pre>git add .</pre><p>To commit:</p><pre>git commit -m &quot;add website&quot;</pre><ul><li>Now, we need to add the remote repository on CodeCommit:</li></ul><pre>git remote add origin https://&lt;username&gt;:&lt;password&gt;@httpsURLOfYourRemoteRepository</pre><ul><li>Push the local change to the remote repository in branch master:</li></ul><pre>git push origin master</pre><pre>Enumerating objects: 3, done.<br>Counting objects: 100% (3/3), done.<br>Delta compression using up to 8 threads<br>Compressing objects: 100% (2/2), done.<br>Writing objects: 100% (3/3), 399 bytes | 133.00 KiB/s, done.<br>Total 3 (delta 0), reused 0 (delta 0)<br>remote: Validating objects: 100%<br>To xxxx<br>* [new branch]      master -&gt; master</pre><p>And, that is, the code is successfully pushed to our remote repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eZXrSPdXtlcfjsyUKns-NQ.png" /></figure><h3>Create the pipeline &amp; deploy website</h3><p>The AWS CodePipeline will enable us to set up a (CI/CD) service.</p><p>So, we will set up a pipeline in AWS CodePipeline that automatically detects changes in our CodeCommit repository we just created and triggers the deployment process.</p><ul><li>Go in creating a new pipeline page, choose a name, and let the other default settings.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FFfsItFUb4ooNF3URs9yug.png" /></figure><ul><li>Next, we will set the “source stage”:</li></ul><p>The Source stage is responsible for detecting changes in the code repository, and in our case it is the AWS CodeCommit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q6i-mmQOa8v-bJZHCbI-YQ.png" /></figure><ul><li>We’re going to skip “build stage”, and go to “deploy stage”.</li><li>We are using S3 bucket to deploy our website.</li><li>Choose a region and the bucket you are using to deploy your code.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*LYI2HYsP5lCuf6npb6ThQQ.png" /></figure><ul><li>don’t forget to enable unzipping, by selecting “Extract file before deploy” because the source artifact will be saved to the artifact store as a zip package after it was pulled from the repository.</li><li>In additional configuration, select “public-read” in “Canned ACL” to give read access to the website content deployed.</li></ul><p>And click next:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*oBUL6r2vLLRHH0GlXhrykg.png" /></figure><p>And Voila, Our pipeline is created and ran successfully !</p><p>Let’s go check the bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Fj6zNRk4lD_CqSOSOnxUg.png" /></figure><p>We see that our file “index.html” is deployed.</p><ul><li>Make sure that the bucket is configured to deploy a static website and that it has the necessary permissions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fbNtGIKycRe4G1HTrLbfWA.png" /></figure><ul><li>Let’s go see our website, to do so, just click on the endpoint shown in “static website hosting” section:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8_ZEYmpQlzFcHiDLat0MAA.png" /></figure><h4>What if we want to add changes to our website, how can we re-deploy the new website ?</h4><ul><li>First, let’s made a change to our index.html file and save it.</li></ul><pre>&lt;!DOCTYPE html&gt;<br>&lt;html lang=&quot;en&quot;&gt;<br>&lt;head&gt;<br>    &lt;meta charset=&quot;UTF-8&quot;&gt;<br>    &lt;title&gt;Index Demo website&lt;/title&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>    &lt;header&gt;<br>        &lt;h1 style=&quot;color:rgb(194, 13, 140)&quot;&gt;Welcome to Our Website!&lt;/h1&gt;<br>    &lt;/header&gt;<br>    &lt;main&gt;<br>        &lt;article&gt;<br>            &lt;p&gt;This is a demo website.&lt;/p&gt;<br>        &lt;/article&gt;<br>    &lt;/main&gt;<br>    &lt;footer&gt;<br>        &lt;p&gt;&amp;copy; V2 version My Website. All rights reserved.&lt;/p&gt;<br>    &lt;/footer&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><ul><li>then, run:</li></ul><pre>git add .</pre><pre>git commit -m &quot;add changes&quot;</pre><pre>git push origin master </pre><ul><li>if we go check our pipeline, you can see that the pipeline was triggered automatically when we pushed our changes to our CodeCommit (you can go to “Execution history” page and check the different execution IDs) .</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M0aE-7pOAp6JvV_IXyAYag.png" /></figure><ul><li>Let’s go see the new version of the website:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n3AIDRns8pOAm0G4c1lJdQ.png" /></figure><p>We see that the version number in the footer has clearly changed.</p><h3>Conclusion:</h3><p>In this tutorial, we demonstrated how to automate the deployment of a static website using <strong>Git, AWS CodeCommit, AWS CodePipeline, and an S3 bucket</strong>. In the next article of this series, we will explore how to use AWS CodeDeploy for more complex deployment scenarios.</p><p>If you enjoyed this article or have any questions, please don’t hesitate to leave a comment. <strong>Your feedback is greatly appreciated</strong>!</p><p><strong>Clap </strong>my article 50 times 👏, that will really help me out and boost this article to others ✍🏻❤️. <strong>Follow </strong>me on <a href="https://medium.com/@mengineer">Medium</a> to get my latest article.</p><p>Thank you 🫶!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5768cde0f5c7" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/learning-aws-codepipeline-hands-on-learning-part1-5768cde0f5c7">Learning AWS CodePipeline Hands-on learning: Part1</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detect Ai Generated Content with Bert Model]]></title>
            <link>https://medium.com/@mengineer/detect-ai-generated-content-with-bert-model-555096b4443a?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/555096b4443a</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Mon, 13 May 2024 00:20:50 GMT</pubDate>
            <atom:updated>2024-05-18T13:59:11.512Z</atom:updated>
            <content:encoded><![CDATA[<h4>This an introduction to Bert model training for AI Content Detection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XxdHs95rknkgCHh0" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Did you ever use AI to generate content ?</blockquote><p>Whether it’s answering an email, generating reports, or even creating article, AI’s capabilities are becoming an integral part of our digital lives.</p><p>As we increasingly rely on AI to assist with our writing tasks, distinguishing between human-generated and AI-generated content has become crucial. This is not just about looking for who wrote the text, but also about ensuring authenticity and accuracy.</p><p>In this article, I am offering you a test guide to implement your own solution. From setting up your coding environment to training and evaluating your model until using the model.</p><h3>🔑Some Key Concepts</h3><p>Before we start coding, it’s important to get familiar with some machine learning key concepts. If you are already familiar with it, feel free to skip this section.</p><h4>Natural Language Processing (NLP)</h4><p>NLP is a branch of artificial intelligence that focuses on the interaction between computers and humans through natural language. The goal of NLP is to enable computers to understand, interpret, and generate human language in a valuable way.</p><h4>Deep Learning</h4><p>Deep learning is a subset of machine learning where artificial neural networks, algorithms inspired by the human brain, learn from large amounts of data. Deep learning is especially effective in identifying patterns and making predictions.</p><h4>Transformers</h4><p>Transformers are a type of neural network architecture that has proven highly effective for various NLP tasks. Introduced in the paper “Attention is All You Need” in 2017, transformers use mechanisms called attention to dramatically improve the efficiency of predicting sequences, making them more efficient than prior models on tasks like translation and text summarization.</p><h4>BERT (Bidirectional Encoder Representations from Transformers)</h4><p>In AI technology, models like <strong>BERT </strong>(Bidirectional Encoder Representations from Transformers) are very powerful. These models excel in understanding the nuances of language, making them perfect tools for tasks that involve natural language processing (NLP). One application of such models is <strong>detecting AI-generated content </strong>— a field that has garnered significant interest both academically and commercially.</p><p>BERT is based on the <strong>Transformer architecture</strong>, which relies on self-attention mechanisms. Unlike previous models that processed words in sequence (one after the other), the Transformer processes all words in a sentence simultaneously. This allows the model to learn the context of a word based on all of its surroundings (both left and right of the word).</p><h4>Fine-Tuning a Model</h4><p>Fine-tuning is a process that takes a pre-trained model (a model trained on a large dataset) and adapts it to a specific task by continuing the training on a smaller, task-specific dataset. This is crucial for adapting BERT to specific NLP tasks, including the detection of AI-generated text.</p><h4>Detection of AI-Generated Content</h4><p>The detection of AI-generated content involves identifying whether text or content was created by an AI model. This is important for various applications, including maintaining the integrity of information and preventing the spread of misinformation.</p><h4>Hugging face</h4><p>Hugging Face is a company known for its work in natural language processing (NLP) and artificial intelligence (AI). It provides a popular platform for sharing and using pre-trained models, particularly through its “Transformers” library. Additionally, Hugging Face provides a comprehensive hub called the “Datasets” library, which offers easy access to hundreds of NLP datasets for various tasks, along with tools for effectively managing and manipulating these datasets.</p><p><a href="https://huggingface.co/">Hugging Face - The AI community building the future.</a></p><p>Here are the Prerequisites you need</p><h3>Prerequisites</h3><ul><li>Python 3.7</li><li>Space minimum in your disk</li><li>Enough resources ( CPU, space disk, RAM). Cache will be used a lot.</li></ul><p>For this demo, I am using python on windows, with:</p><ul><li><strong>Intel Core i7–8565U</strong> <strong>Cores </strong>( 4 physical cores and allows each physical core to handle two threads simultaneously which gives 8 logical processors).</li><li><strong>RAM </strong>8 GO.</li><li>50 Go free space in the <strong>disk</strong>.</li></ul><h3>Step 1: Environment Setup 🔧</h3><p>1- <strong>Create a virtual environment: </strong>It’s recommended to use a virtual environment to avoid a lot of compatibility errors .</p><p>You need first to have “virtualenv” installed in your machine, if it’s not the case, run this command:</p><pre>pip install virtualenv</pre><p>Then, we create our virtual environment named “myenv”.</p><pre>python&lt;version&gt; -m venv myenv</pre><p>Now, you need to activate it:</p><pre>source env/bin/activate # in linux<br><br>or<br><br>\myenv\Scripts\activate # in windows</pre><p><strong>2. Install Necessary Libraries</strong>: We will need to install the necessary libraries with this command:</p><pre>pip install transformers torch datasets</pre><p>transformers provides access to <strong>BERT </strong>and other <strong>pre-trained models</strong> along with their <strong>tokenizers</strong>.</p><p>torch is the framework that <strong>supports </strong>neural network layers used by transformers.</p><p>datasets is a lightweight and extensible library provided by <strong>Hugging Face</strong> to easily share and load dataset and evaluation metrics.</p><p>Now, that our environment is ready, let’s start preparing the Data.</p><h3>Step 2: Data Preparation</h3><p>We’re using the “<a href="https://huggingface.co/datasets/dmitva/human_ai_generated_text">https://huggingface.co/datasets/dmitva/human_ai_generated_text</a>” Dataset from Hugging Face, where there is human and ai text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nhz5IgD_EZgXFe_G9SsPQg.png" /><figcaption>Dataset from <a href="https://huggingface.co/datasets/dmitva/human_ai_generated_text">Hugging Face</a></figcaption></figure><p><strong>Load Data from Hugging Face</strong>: Let’s load the data from Hugging Face using datasets library.</p><pre>from datasets import load_dataset<br><br>dataset = load_dataset(&#39;dmitva/human_ai_generated_text&#39;)</pre><h3>Step 3: Preprocess the Data</h3><p><strong>Explore the Dataset</strong>: First step is to understanding the structure of your data, to know how to process it.</p><pre>print(dataset[&#39;train&#39;][0])  # View the first element to understand the structure</pre><p><strong>Prepare Features and Labels</strong>: Next, We’re going to concatenate the human and AI-generated texts into one dataset. AI texts will have a label 1 human texts 0.</p><pre><br>def prepare_features(example):<br>    # This function will duplicate each row, one for human_text and one for ai_text<br>    features = {&#39;text&#39;: [example[&#39;human_text&#39;], example[&#39;ai_text&#39;]], &#39;label&#39;: [0, 1]}<br>    return features<br><br>dataset = dataset.map(prepare_features)<br></pre><p><strong>Shuffle the Data:</strong> And now, we need to shuffle the data so that the values are not in chronological order.</p><pre># Shuffle the dataset<br>shuffled_dataset = dataset[&#39;train&#39;].shuffle(seed=42)</pre><blockquote>In the context of shuffling a dataset, the seed parameter is used to initialize the random number generator that determines the order in which the elements of the dataset are arranged.</blockquote><p><strong>Split the Data</strong>: Then, divide the dataset into training and testing sets. A typical split might be 80% for training and 20% for testing.</p><blockquote>The training dataset is used to <strong>teach </strong>the model to recognize patterns and make predictions, while the testing dataset helps <strong>assess how well the model performs on new, unseen</strong> examples.</blockquote><pre>from datasets import DatasetDict<br><br># Perform an 80/20 split<br>train_test_split = shuffled_dataset.train_test_split(test_size=0.2)<br><br># Now, train_test_split is a DatasetDict containing both the train and test splits<br>train_dataset = train_test_split[&#39;train&#39;]<br>test_dataset = train_test_split[&#39;test&#39;]</pre><h3>Step 4: Tokenization for BERT</h3><p>This step is about setting up a tokenizer for a BERT model and using it to preprocess a dataset.</p><p>preparing the dataset by converting <strong>raw text</strong> into a <strong>format </strong>that can be fed into a <strong>BERT model</strong>, making sure all inputs are tokenized and standardized in terms of length. This preprocessing step is crucial for training and evaluating models on natural language tasks.</p><p><strong>Tokenizer Setup</strong>: Initialize a tokenizer suited for BERT.</p><pre>from transformers import BertTokenizer<br><br>tokenizer = BertTokenizer.from_pretrained(&#39;bert-base-uncased&#39;)</pre><p><strong>Tokenize Data</strong>: Convert raw text data into a format suitable for BERT.</p><pre>def tokenize_function(examples):<br>    return tokenizer(examples[&#39;text&#39;], padding=&quot;max_length&quot;, truncation=True, max_length=512)</pre><h3>Step 5: Create Dataset for Training</h3><p><strong>Load Tokenized Data into a Dataset</strong>: Convert the data into a format that can be efficiently handled by the PyTorch DataLoader.</p><pre>from torch.utils.data import DataLoader<br><br>tokenized_train = train_dataset.map(tokenize_function, batched=True, batch_size=4000, num_proc=6)<br>tokenized_test = test_dataset.map(tokenize_function, batched=True, batch_size=4000, num_proc=6)<br><br>train_loader = DataLoader(tokenized_train, batch_size=6, shuffle=True)<br>test_loader = DataLoader(tokenized_test, batch_size=6)</pre><p>You need to find the right number of processes and the right size of batch depending on your resources.</p><h3>Step 6: Initialize BERT Model</h3><p><strong>Model Setup</strong>: Configure BERT for sequence classification.</p><pre>from transformers import BertForSequenceClassification<br><br>model = BertForSequenceClassification.from_pretrained(&#39;bert-base-uncased&#39;, num_labels=2)  # Adjust num_labels based on your dataset specifics</pre><h3>Step 7: Training</h3><p><strong>Define Training Arguments</strong>: Set up the training configuration</p><pre>from transformers import Trainer, TrainingArguments<br><br>training_args = TrainingArguments(<br>    output_dir=&#39;./results&#39;,                 # Directory for storing outputs<br>    num_train_epochs=3,                     # Total number of training epochs<br>    per_device_train_batch_size=32,         # Batch size per device during training<br>    evaluation_strategy=&quot;epoch&quot;,            # Evaluation is performed at the end of each epoch<br>    save_strategy=&quot;epoch&quot;,                  # The model is saved at the end of each epoch<br>    logging_dir=&#39;./logs&#39;,                   # Directory for storing logs<br>    logging_steps=500,                      # Log training information every 500 steps<br>)<br><br>trainer = Trainer(<br>    model=model,<br>    args=training_args,<br>    train_dataset=train_dataset,<br>    eval_dataset=test_dataset<br>)</pre><p><strong>Start Training</strong>: Begin the training process.</p><pre>trainer.train()</pre><h3>Step 8: Evaluation</h3><ol><li><strong>Evaluate the Model</strong>: Assess the performance of your model on the test set.</li></ol><pre>results = trainer.evaluate()<br>print(results)</pre><h3>Step 9: Deployment</h3><ol><li><strong>Make Predictions</strong>: Utilize the trained model to make predictions on new data.</li><li><strong>Save the Model</strong>: Optionally, save your trained model for future use.</li></ol><pre>model.save_pretrained(&#39;./my_model&#39;)<br>tokenizer.save_pretrained(&#39;./my_model&#39;)</pre><h3>Step 10: Use the Model for Inference</h3><p><strong>Model Inference</strong>: To check if a new text is AI-generated, you can use the trained model to predict on new data.</p><pre><br>    def predict(text):<br>        # Ensure the tokenizer is defined in the scope used by the function<br>        inputs = tokenizer(text, return_tensors=&quot;pt&quot;, padding=True, truncation=True, max_length=512)<br>        with torch.no_grad():  # Ensure no gradients are calculated during inference<br>            outputs = model(**inputs)<br>            logits = outputs.logits<br>            predictions = torch.argmax(logits, dim=-1)<br>        return &quot;AI-generated&quot; if predictions.item() == 1 else &quot;Human-written&quot;<br><br>    example_text = &quot;Enter your text here&quot;<br>    print(predict(example_text))</pre><h3>Let’s run the script</h3><p>Finally, we are going to see some real executions !</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FCS8Fgb5BAynaDGY1Me%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Ffallontonight-wow-finally-CS8Fgb5BAynaDGY1Me&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExNXpwOGVzdzFkd2M4dzM5bHhxZjdteDdmMncwZWduZmtteXFrbm5wcSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FCS8Fgb5BAynaDGY1Me%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="310" frameborder="0" scrolling="no"><a href="https://medium.com/media/261c07db893c2736dd34fd27e02b9e0c/href">https://medium.com/media/261c07db893c2736dd34fd27e02b9e0c/href</a></iframe><p>Since, there is not enough resources in the test environment used ( described above ), I have made some changes to the code so I don’t wait hours for the script to be executed.</p><p>Here are the main changes:</p><ul><li>I have used a subset of 10 rows.</li><li>A batch_size of 400 in the tokenized</li><li>A batch_size of 8 in the loader</li><li>A per_device_train_batch_size of 2</li><li>A logging_steps of 10</li></ul><h4>Execute the script</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba56c03c5690c2a6ec0e84942b83dc17/href">https://medium.com/media/ba56c03c5690c2a6ec0e84942b83dc17/href</a></iframe><p>Run the script in your virtual environment:</p><pre>python .\ai_detector_bert.py</pre><p>Here are the results:</p><pre>{&#39;eval_loss&#39;: 0.5319023728370667, &#39;eval_runtime&#39;: 12.5334, &#39;eval_samples_per_second&#39;: 0.16, &#39;eval_steps_per_second&#39;: 0.08, &#39;epoch&#39;: 1.0}  <br>{&#39;eval_loss&#39;: 0.3947715163230896, &#39;eval_runtime&#39;: 13.5184, &#39;eval_samples_per_second&#39;: 0.148, &#39;eval_steps_per_second&#39;: 0.074, &#39;epoch&#39;: 2.0}<br>{&#39;eval_loss&#39;: 0.3486124277114868, &#39;eval_runtime&#39;: 13.2514, &#39;eval_samples_per_second&#39;: 0.151, &#39;eval_steps_per_second&#39;: 0.075, &#39;epoch&#39;: 3.0}<br>{&#39;train_runtime&#39;: 203.3794, &#39;train_samples_per_second&#39;: 0.118, &#39;train_steps_per_second&#39;: 0.03, &#39;train_loss&#39;: 0.5257925987243652, &#39;epoch&#39;: 3.0}<br>{&#39;eval_loss&#39;: 0.3486124277114868, &#39;eval_runtime&#39;: 16.8589, &#39;eval_samples_per_second&#39;: 0.119, &#39;eval_steps_per_second&#39;: 0.059, &#39;epoch&#39;: 3.0}<br><br>AI-generated</pre><h4>Interpretation of the results &amp; Improvements</h4><p><strong>Evaluation Loss</strong>: The evaluation loss decreases over the epochs, which suggests that your model is learning and improving its performance on the evaluation set. Here’s how the loss decreased:</p><ul><li>Epoch 1: 0.5319023728370667</li><li>Epoch 2: 0.3947715163230896</li><li>Epoch 3: 0.3486124277114868</li></ul><p>This is a good sign, indicating model convergence.</p><p><strong>Epoch Count</strong>:</p><p>The training and evaluation process runs through 3 epochs as specified, and the model shows consistent improvements, indicating that the training process itself is behaving as expected.</p><p><strong>Runtime and Speed</strong>:</p><ul><li><strong>Evaluation Runtime and Speed</strong>: The evaluation runtime is relatively high given the extremely small size of the dataset, and it even increases slightly from Epoch 1 to Epoch 3.</li><li><strong>Training Runtime and Speed</strong>: The training runtime and speeds (train_samples_per_second: 0.118, train_steps_per_second: 0.03) also suggest significant overhead.</li></ul><p>This was expected since we are not using enough resources for training the model. Choosing the <strong>right resources</strong> is crucial in training models. You need to ensure you are using suitable hardware. For instance, using a GPU could significantly speed up training and evaluation.</p><p>Plus, we can optimize the script to be more performant, so it’s important to utilize profiling tools to understand where most time is spent during your training and evaluation. Python’s cProfile or PyTorch&#39;s built-in profilers can help identify bottlenecks.</p><h3>References</h3><ul><li><a href="https://huggingface.co/docs/transformers/model_doc/bert">BERT</a></li><li><a href="https://www.ibm.com/topics/natural-language-processing">What Is NLP (Natural Language Processing)? | IBM</a></li></ul><p>If you have any questions, feedback, or would like to share your experiences, please feel free to reach out in the comments section.</p><p><strong>Clap </strong>my article 50 times 👏, that will really help me out and boost this article to others ✍🏻❤️. <strong>Follow </strong>me on <a href="https://medium.com/@mengineer">Medium</a> to get my latest article.</p><p>Thank you 🫶!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=555096b4443a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How did I Pass Confluent Certified Administrator for Apache Kafka (CCAAK) at the First Try]]></title>
            <link>https://medium.com/apache-kafka-official/how-did-i-pass-confluent-certified-administrator-for-apache-kafka-ccaak-at-the-first-try-eee2f423d562?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/eee2f423d562</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[stream-processing]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Sat, 30 Mar 2024 05:22:08 GMT</pubDate>
            <atom:updated>2024-04-07T01:41:32.900Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PH2HAzxwBaBWgsuE" /><figcaption>Photo by <a href="https://unsplash.com/@jannerboy62?utm_source=medium&amp;utm_medium=referral">Nick Fewings</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>I will share everything you need to know and some tips</h4><p>2 days ago, I passed the <strong>Confluent Certified Administrator for Apache Kafka (CCAAK) </strong>exam, and I decided to share my experience and share more tips since there is not too much shared about it.</p><h3>What is the certification for ?</h3><blockquote>Getting the certification from Confluent is going to validate the Apache Kafka administration expertise with a well respected and highly recognized Confluent Certification.</blockquote><p><a href="https://www.confluent.io/certification/"><strong>Confluent Certified Administrator for Apache Kafka </strong>or <strong>CCAAK</strong></a><strong> </strong>is a certification targeted to those who administer Kafka cluster environments.<br>It covers the most critical job activities that an <strong>Apache Kafka Administrator</strong> performs, from <strong>configuring and deploying</strong>, to<strong> monitoring, managing</strong>, and <strong>supporting </strong>Kafka clusters.</p><p>The cost of the certification is <strong>150</strong>$ for <strong>every attempt</strong>.</p><h3>My Kafka Journey before the CCAAK</h3><p>I learn Apache Kafka since 2020. I had my first Kafka certification in 2022, it was <strong><em>Confluent Certified Developer for Apache Kafka</em></strong> certification. Being someone with +4 years of experience in developing with Kafka and 2 years of maintaining and managing Kafka Clusters, I didn’t need to prepare for so long to take the exam. I have spent serious 2–3 hours of preparation in one week, that was enough for me to see all concepts needed.</p><h3>Details about the exam</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fg0aLXsVW9kBTbfbJb0%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExMjkxZTdzZ250Z3kxaHo2eGFtbWVpdzNubnI2Z2V6cHo2ZnBxcTlsdSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2Fg0aLXsVW9kBTbfbJb0%2Fgiphy.gif&amp;image=https%3A%2F%2Fmedia3.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExYTlvM25odmJyZmZxb2Qxd2h1YThsOTdrbnZ5YnVkZDYzNnFqczhhNSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2Fg0aLXsVW9kBTbfbJb0%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/c273b757ce85c215b01e85a2724fafe6/href">https://medium.com/media/c273b757ce85c215b01e85a2724fafe6/href</a></iframe><ul><li>The exam has 40 multiple choice questions and passed in 90 minutes.</li><li>It can be taken on demand under remote proctored conditions, so you can take the certification from your computer.</li><li>Available in English.</li><li>The certification is valid for 2 years.</li></ul><h3>Preparation</h3><p>I prepared for the exam using mainly these resources:</p><ul><li>The Apache Kafka official documentation <a href="https://kafka.apache.org/documentation/">https://kafka.apache.org/documentation/</a></li><li>The Kafka Definitive Guide from confluent. <a href="https://www.confluent.io/resources/kafka-the-definitive-guide/">https://www.confluent.io/resources/kafka-the-definitive-guide/</a></li><li>Practical exams like <a href="https://links.datacumulus.com/confluent-certified-ccoak-coupon">Stephane’s Practice Exam course</a>, they are a bit old but they can help you test your knowledge.</li></ul><p>Before the exam, you need to have a very good understanding about:</p><ul><li>Configurations of Kafka brokers, producers, consumers, consumer groups, zookeeper, kafka connectors ( like acks, bach.size, linger.ms, timeout parameters for consumers &amp; producers, idempotence, replication factor, offset reset, lag … )</li><li>Consumers re-balances.</li><li>Role of Zookeeper ( storing ACLs information, elections …)</li><li>Role of Leaders.</li><li>Role of Coordinator.</li><li>Role of Group consumer.</li><li>The partition assignment.</li><li>Security Protocols, authentification and authorisation (secure Kafla brokers / clients / zookeeper …).</li><li>Kafka connect, Kafka Streams, KsqlDB.</li><li>Schema Registry.</li></ul><h3>Feedback about the exam</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FcLLVsGPA6Vky1AZ2Rd%2Fvideo&amp;display_name=GIPHY&amp;url=https%3A%2F%2Fgiphy.com%2Fclips%2Fpbs-science-human-footprint-shane-campbell-staton-cLLVsGPA6Vky1AZ2Rd&amp;image=https%3A%2F%2Fmedia2.giphy.com%2Fmedia%2FcLLVsGPA6Vky1AZ2Rd%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="480" height="270" frameborder="0" scrolling="no"><a href="https://medium.com/media/c286891c919936b67968c402c90a960f/href">https://medium.com/media/c286891c919936b67968c402c90a960f/href</a></iframe><p>I personally find the <strong>CCAAK </strong>certification of <strong>medium </strong>difficulty. And surprisingly I didn’t get any question about schema registry and Kafka Streams. On the other hand, the questions I had were focused on partition assignment, offset reset and consumer lag , security protocols (mTLS, SASL..), Kafka connect (sink vs source, how can we scale the connector), KsqlDB, zookeeper, kafka leader election (exp: the role of Epoch Number<strong>) </strong>,the role of “replication-offset-checkpoint” &amp; “recovery-point-offset-checkpoint” files, garbage collection and its impact on the broker and the shutting down of the broker and its impact.</p><p>Overall, there were more number of scenario based questions. Some answers seems to be alike, so you need to focus and read the question very carefully to choose the right answer(s).</p><p>Happy learning and I wish you good luck with the <strong>CCAAK </strong>certification !</p><p>That’s it, If I have helped you get the certification, or If you have any questions, feedback, or would like to share your experiences, please feel free to reach out in the comments section.</p><p><strong>Clap </strong>my article 50 times 👏, that will really help me out and boost this article to others ✍🏻❤️. <strong>Follow </strong>me on <a href="https://medium.com/@mengineer">Medium</a> to get my latest article.</p><p>Thank you 🫶!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eee2f423d562" width="1" height="1" alt=""><hr><p><a href="https://medium.com/apache-kafka-official/how-did-i-pass-confluent-certified-administrator-for-apache-kafka-ccaak-at-the-first-try-eee2f423d562">How did I Pass Confluent Certified Administrator for Apache Kafka (CCAAK) at the First Try</a> was originally published in <a href="https://medium.com/apache-kafka-official">Apache Kafka Official</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zookeeper Setup with Kafka Cluster : Part 3]]></title>
            <link>https://medium.com/apache-kafka-official/zookeeper-setup-with-kafka-cluster-part-3-1430143b9215?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1430143b9215</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[zookeeper]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Fri, 22 Mar 2024 03:10:31 GMT</pubDate>
            <atom:updated>2024-03-22T16:03:54.056Z</atom:updated>
            <content:encoded><![CDATA[<h3>Zookeeper Setup with Kafka Cluster : Part 3</h3><h4>Your Ultimate Guide to setup Zookeeper on Quorum mode in AWS</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JMGpr3pAtTnfrfI6" /><figcaption>Photo by <a href="https://unsplash.com/@shubzweb3?utm_source=medium&amp;utm_medium=referral">Shubham&#39;s Web3</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Welcome back 🙋‍♀️ to the third part of our guide about setting up Zookeeper with Kafka Cluster on AWS!</p><p>In the previous article, we learned the theory behind this setup and successfully launched an EC2 instance, gaining access to our machine. Now, let’s dive into setting up our Zookeeper cluster.</p><ul><li><a href="https://medium.com/@mengineer/zookeeper-setup-with-kafka-cluster-part-2-753226ffc18d">Zookeeper Setup with Kafka Cluster : Part 2</a></li><li><a href="https://medium.com/apache-kafka-official/zookeeper-setup-with-kafka-cluster-part-1-469dc4671198">Zookeeper Setup with Kafka Cluster : Part 1</a></li></ul><p>Today, we will try to setup our zookeeper cluster in two steps:</p><ul><li>First, setup single zookeeper machine.</li><li>Then, setup zookeeper on quorum mode with 3 servers.</li></ul><h3>Setup Single Zookeeper Machine</h3><p>We will follow the following steps:</p><ul><li>Install packages.</li><li>Disable Ram Swap.</li><li>Add host mappings to mock a DNS server.</li><li>Download and configure Zookeeper.</li><li>Launch Zookeeper.</li></ul><p>First, we will need to have the ability to allow traffic to port 80 (HTTP) and port 443 (HTTPS) so we can download from Internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IC8iBJg7aDAGuihfDmXFbQ.png" /></figure><p>Now, connect to your machine using ssh, and run this command:</p><pre>sudo apt-get update &amp;&amp; \<br>      sudo apt-get -y install wget ca-certificates zip net-tools vim nano tar netcat</pre><p>This will install all the utilities needed.</p><p>Then, we will install Java:</p><pre># Java Open JDK 8<br>sudo apt-get -y install openjdk-8-jdk</pre><p>Next step is disabling Ram Swap.</p><blockquote>RAM swap is a space on a storage device (usually a hard disk drive or SSD) that the operating system uses as additional virtual memory when physical RAM is fully utilized.</blockquote><blockquote>Disabling swap is crucial when using Kafka. It helps preventing potential latency spikes caused by disk swapping !</blockquote><p>To disable Swap on Ubuntu, we will run this command:</p><pre># Disable RAM Swap <br>sudo sysctl vm.swappiness=1</pre><p>and to not losing this setting, we will add it to /etc/sysctl.conf</p><pre>echo &#39;vm.swappiness=1&#39; | sudo tee --append /etc/sysctl.conf</pre><p>Next, we will try to mock DNS for Zookeeper &amp; Kafka</p><pre># Add hosts entries (mocking DNS for zookeeper and kafka<br># that will be installed in this instance<br><br>echo &quot;172.31.11.1 kafka1<br>172.31.11.1 zookeeper1<br>172.31.20.2 kafka2<br>172.31.20.2 zookeeper2<br>172.31.32.10 kafka3<br>172.31.32.10 zookeeper3&quot; | sudo tee --append /etc/hosts</pre><p>Now, we will start installing kafka and zookeeper by running these commands</p><pre># download Zookeeper and Kafka. <br>wget https://archive.apache.org/dist/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz<br>tar -xvzf kafka_2.12-0.10.2.1.tgz</pre><p>Let’s do some cleansing:</p><pre>rm kafka_2.12-0.10.2.1.tgz<br>mv kafka_2.12-0.10.2.1 kafka</pre><p>Now we’re good ! Kafka and zookeeper are installed successfully in our instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*uJvbt-G9LTvsh_R3K6R-Sw.png" /></figure><p>To start Zookeeper, run this command:</p><pre>bin/zookeeper-server-start.sh config/zookeeper.properties</pre><p>If you see these logs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UZF5otylLC2bjRSCMUWO2w.png" /></figure><p>Congratulations 🎉! Your Zookeeper is running.</p><h3>Setup Zookeeper on Quorum Mode</h3><p>Now, we will need to have three instances of zookeeper. To do so, we will need to:</p><ul><li>Create an AMI image from the EC2 machine that we created.</li><li>Create 2 other EC2 machines from the AMI image.</li><li>Configure the Quorum zookeeper mode.</li></ul><h4>AMI Image</h4><p>To create the image, right click on the instance and click on “create image”, and add a name and description for the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCr52tDEvp2v_fDy-hUGyg.png" /></figure><p>We will wait until the image has active status</p><h4>Create 2 other EC2 machines with Zookeeper &amp; Kafka</h4><p><strong>Server2</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*thz7JxKWQZVtsJ6mFHKcSQ.png" /></figure><p>We click on “launch instance from AMI”</p><p>Here, following the same steps as our <a href="https://medium.com/@mengineer/zookeeper-setup-with-kafka-cluster-part-2-753226ffc18d">first creation of the EC2 instance</a> “Server1”, we will add tag “Server2”, and another subnet, for me I am picking <strong>2b </strong>and new network interface with this IP “<strong>172.31.20.2</strong>” and we are going to choose the same Security group that we created for our first instance “Server1”, and we will choose the same Key Pair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EVgDaMbotdR7REqXfVfozA.png" /></figure><p><strong>Server3</strong></p><p>Next, we will launch new instance from the AMI, with tag “<strong>Server3</strong>”, a new subnet “<strong>2c</strong>”, new network interface with IP “<strong>172.31.32.10</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OuSz_lnzR4J2VxY2sr20Zg.png" /></figure><h4>Configure the Quorum zookeeper mode</h4><p>This is the most important step.</p><p>We will ssh to our 3 instances in 3 powershells, and connect to zookeeper in each instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wlVeAeXdpjJRG7hjXk93vA.png" /></figure><p>Next, we will follow the following steps:</p><ul><li><strong><em>Create data dictionary for zookeeper in each instance</em></strong></li></ul><pre>sudo mkdir -p /data/zookeeper<br>#change owner of /data/zookeeper to become Ubuntu <br>sudo chown -R ubuntu:ubuntu /data/</pre><ul><li><strong><em>Declare the server’s identity using the file “myid” in each server</em></strong></li></ul><p>In the first instance</p><pre>echo &quot;1&quot; &gt; /data/zookeeper/myid</pre><p>In the second instance</p><pre>echo &quot;2&quot; &gt; /data/zookeeper/myid</pre><p>In the third instance</p><pre>echo &quot;3&quot; &gt; /data/zookeeper/myid</pre><ul><li><strong>Edit Zookeeper settings</strong></li></ul><p>We will remove the existing properties</p><pre>rm /home/ubuntu/kafka/config/zookeeper.properties</pre><p>And create new properties file</p><pre>nano /home/ubuntu/kafka/config/zookeeper.properties</pre><p>With these configs (if you want to have more details about these configs, see my <a href="https://medium.com/apache-kafka-official/zookeeper-setup-with-kafka-cluster-part-1-469dc4671198">previous article</a>)</p><pre># the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database.<br>dataDir=/data/zookeeper<br># the port at which the clients will connect<br>clientPort=2181<br># disable the per-ip limit on the number of connections since this is a non-production config<br>maxClientCnxns=0<br># the basic time unit in milliseconds used by ZooKeeper. It is used to do heartbeats and the minimum session timeout will be twice the tickTime.<br>tickTime=2000<br># The number of ticks that the initial synchronization phase can take<br>initLimit=10<br># The number of ticks that can pass between<br># sending a request and getting an acknowledgement<br>syncLimit=5<br># zoo servers<br># these hostnamescome from the /etc/hosts file<br>server.1=zookeeper1:2888:3888<br>server.2=zookeeper2:2888:3888<br>server.3=zookeeper3:2888:3888</pre><p>And we are ready !</p><p>Now let’s run the zookeeper cluster.</p><p>First, start zookeeper in one of the servers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pESNKwgsBgkkAU4P6YDfwA.png" /></figure><p>So as expected, zookeeper could not start, I think you know why 😁</p><p>As we explained on a previous article, zookeeper on Quorum=3 will not start if 2 zookeepers are not running. So let’s start zookeeper in another instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dMxyMFGC6a5iAFDZ7-lp_w.png" /></figure><p>And as you can see, as soon as we started another zookeeper, It does work.</p><p>Let’s start the third one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_T3tUfm1P_5UjmIKlSjUPw.png" /><figcaption>Zookeeper on mode Quorum with 1 leader and 2 followers</figcaption></figure><p>And Voila, we have successfully setup a zookeeper cluster with 3 servers, one act as a leader and 2 as followers 🥳🥳🥳 !</p><h3>Conclusion</h3><p>In this guide, we’ve successfully set up a Zookeeper cluster with three servers, with one acting as a leader and the other two as followers. With this setup, we’re now ready to leverage the power of Zookeeper and Kafka for our distributed data processing needs.</p><p>We didn’t finished yet 😜, so don’t hesitate to subscribe to get notified 📧</p><h3>References</h3><ul><li><a href="https://medium.com/@mengineer/zookeeper-setup-with-kafka-cluster-part-2-753226ffc18d">Zookeeper Setup with Kafka Cluster : Part 2</a></li><li><a href="https://docs.aws.amazon.com/"></a></li></ul><p>https://www.udemy.com/course/kafka-cluster-setup/</p><p><a href="https://medium.com/apache-kafka-official/zookeeper-setup-with-kafka-cluster-part-1-469dc4671198">Zookeeper Setup with Kafka Cluster : Part 1</a></p><p>If you have any questions, feedback, or would like to share your experiences, please feel free to reach out in the comments section.</p><p><strong>Clap </strong>my article 50 times 👏, that will really help me out and boost this article to others ✍🏻❤️. <strong>Follow </strong>me on <a href="https://medium.com/@mengineer">Medium</a> to get my latest article.</p><p>Thank you 🫶!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1430143b9215" width="1" height="1" alt=""><hr><p><a href="https://medium.com/apache-kafka-official/zookeeper-setup-with-kafka-cluster-part-3-1430143b9215">Zookeeper Setup with Kafka Cluster : Part 3</a> was originally published in <a href="https://medium.com/apache-kafka-official">Apache Kafka Official</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zookeeper Setup with Kafka Cluster : Part 2]]></title>
            <link>https://towardsaws.com/zookeeper-setup-with-kafka-cluster-part-2-753226ffc18d?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/753226ffc18d</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[zookeeper]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Thu, 21 Mar 2024 05:11:23 GMT</pubDate>
            <atom:updated>2024-03-22T03:58:06.251Z</atom:updated>
            <content:encoded><![CDATA[<h3>Zookeeper Setup with Kafka Cluster : Part 2</h3><h4>Your ultimate guide to setup Zookeeper in AWS</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*fj_r4Nc_rp9rxS8VDDxHMQ.jpeg" /><figcaption>Amazon EC2, Photo by digitalcloud.training</figcaption></figure><p>Now that we are familiar with Zookeeper. It’s time for us to setup a zookeeper cluster in AWS.</p><p>If you want to go through some theory before you start using Zookeeper, you can go check my previous article.</p><p><a href="https://medium.com/apache-kafka-official/zookeeper-setup-with-kafka-cluster-part-1-469dc4671198">Zookeeper Setup with Kafka Cluster : Part 1</a></p><h3>Prerequisites</h3><ul><li>An AWS Account</li></ul><h3>Setup AWS</h3><h4>Launch an EC2 Instance</h4><p>We’re going to use Ubuntu server 22.04. With a T2 medium that has 4 gigabytes of Ram.</p><p>Let’s add a tag to our instance “Server 1”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Ygvpr-ptrOOuiHFeHP17A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R3qSQD4NWmVqJu5oj2EoEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aLaUjT_o9pG0sgc94yHvLw.png" /></figure><p><strong>The next step is setting up the network:</strong></p><ul><li>The network will be set to default.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O9LcZSOdRIRGKkM1M9pIDw.png" /></figure><ul><li>Next, we will add a network interface. We will use one subnet , here I am picking the 2a AZ .</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IraknGUTWcjDEAnIXaADZg.png" /></figure><ul><li>Then, in “advanced network configuration tab”, we set the primary IP 172 .31.11.1 .</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xb-gniA1X_-DQ-YAUtRZqw.png" /></figure><p><strong>Next step is Setting Inbound Security Group Rules:</strong></p><p>That’s the most important step. We’re going to set security group and we’re going to call it “Kafka.Zookeeper”.</p><p>First, we will not allow SSH access to our EC2 instance from any location on the internet. So if you have a security group rule like this “TCP, 22, 0.0.0.0/0”, remove it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TAOZF9caQIjgCw8qNtcqiQ.png" /></figure><ul><li>Then add an ssh from your own IP.</li><li>Then we add 3 custom TCP rule with the zookeeper ports 2181, 2888 and 3888. That’s going to be all our instances.</li><li>And finally, we’re going to add custom TCP rule with 2181 port again so I can access Zookeeper from my computer if I wanted to.</li></ul><p><strong>Setting Storage</strong></p><p>Next step is EBS Volumes. We will use 8Gib. But if you’re serious about your setup, you should maybe increase this to 25 or 50GB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OEThcqK-YlGQu1THXXRTaA.png" /></figure><ul><li>Then the volume type. Here we pick gp2 because it’s an SSD drive and it has a high performance.</li><li>we’re going to disable the delete on termination so we don’t lose the data when the instance is terminated.</li></ul><p><strong>Create a new Key Pair</strong></p><p>Because we haven’t gotten a key pair yet. we need to create one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*DAgUmLLUhYQbZyHJuidy0w.png" /></figure><p>And then we click on Create Key pair.</p><p>It will download a .pem file. This file should not be lost ! You need to back it up and make sure it’s protected.</p><p>Then you click on launch instance.</p><p>And Voila, our instance is ready !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_FoDR3Dc2Fbsbpd3v0xV8A.png" /><figcaption>EC2 Instance, photo by Author</figcaption></figure><p>Let’s access the instance using ssh.</p><p>Since I am on Windows, I am using ssh in Powershell:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3DPGsP6IDk0d1buXKG4ezQ.png" /><figcaption>ssh my instance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1eDKhEaTVI-60kMZWrjuwQ.png" /><figcaption>We’re connected successfully !</figcaption></figure><h3>Conclusion</h3><p>In this article, we have seen step by step how to launch an EC2 instance, configuring network settings, security groups, storage, and creating a new key pair. This step is essential in the process of zookeper setup in AWS. In the <a href="https://medium.com/@mengineer/zookeeper-setup-with-kafka-cluster-part-3-1430143b9215">next Part (Part3)</a>, we will see step by step how to create and use Zookeeper in AWS.</p><h3>References</h3><p><a href="https://docs.aws.amazon.com/ec2/?icmpid=docs_homepage_featuredsvcs">https://docs.aws.amazon.com/</a></p><p><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-linux-inst-ssh.html">Connect to your Linux instance using an SSH client</a></p><p>If you have any questions, feedback, or would like to share your experiences, please feel free to reach out in the comments section.</p><p><strong>Clap </strong>my article 50 times 👏, that will really help me out and boost this article to others ✍🏻❤️. <strong>Follow </strong>me on <a href="https://medium.com/@mengineer">Medium</a> to get my latest article.</p><p>Thank you 🫶!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=753226ffc18d" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/zookeeper-setup-with-kafka-cluster-part-2-753226ffc18d">Zookeeper Setup with Kafka Cluster : Part 2</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zookeeper Setup with Kafka Cluster : Part 1]]></title>
            <link>https://medium.com/@mengineer/zookeeper-setup-with-kafka-cluster-part-1-469dc4671198?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/469dc4671198</guid>
            <category><![CDATA[zookeeper]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Wed, 20 Mar 2024 21:25:25 GMT</pubDate>
            <atom:updated>2024-03-22T21:27:04.535Z</atom:updated>
            <content:encoded><![CDATA[<h3>Zookeeper Setup with Kafka Cluster : Part 1</h3><h4>The theory behind using Zookeeper with Kafka Cluster</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F1EK3946j6tR64PgQ_uk8Q.png" /><figcaption>Architecture using Zookeeper</figcaption></figure><h3>What is Zookeeper ?</h3><p>Zookeeper is a pillar for so many distributed applications.</p><p>It provides a lot of features:</p><ul><li>The first one is <strong>distributed configuration management</strong> so it can manage the configuration of distributed systems.</li><li><strong>Election and consensus</strong>: That means if servers ask who’s the leader? zookeeper will respond.</li><li><strong>Coordination and locks.</strong></li><li>In Kafka’s case, it does <strong>key value store</strong> so it can store many configuration for topics and brokers...</li></ul><p>Zookeeper is not only used with Kafka, but can be used with Hadoop and other big data systems.</p><p>Zookeeper is like a file system, It has an internal structure like a tree.</p><p>We all know what file system looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*Nq4u2kFtUVjLZ3kj1YOkwg.png" /><figcaption>Tree structure, photo by Author</figcaption></figure><p>We have the root at the very top and then you have branches and at the bottom you have leaves or nodes.</p><ul><li>So each node is called a <strong>zNode.</strong></li><li>Each <strong>zNode </strong>has a <strong>path.</strong></li><li>Each <strong>zNode </strong>can be persistent or ephemeral. A persistent <strong>zNode </strong>is something that will stay alive all the time, Zookeeper will remember it all the time. An ephemeral <strong>zNode</strong> is a <strong>zNode</strong> that will just go away if the app disconnects.</li><li>Each node can store multiple <strong>zNode</strong>.</li><li>You cannot rename the node.</li><li>Each <strong>zNode </strong>can be wached for changes. So if changes occurs on some zNode that is wached, he will let you know<strong>.</strong></li></ul><h3>What is the role of Zookeeper?</h3><p>Let’s see in detail what is the role of Zookeeper in Kafka.</p><ul><li>One of the most important thing that he does is <strong>broker registration with heartbeat mechanisms</strong> to keep the list current.</li></ul><blockquote>Heartbeats are very important when you have server to server communications.</blockquote><blockquote>If a broker gets disconnected, Zookeeper will remove it because it hasn’t sent a heartbeat.</blockquote><blockquote>A heartbeat is like a message that usually servers communicate to each other to say that they are alive.</blockquote><ul><li>It also <strong>maintains the list of topics</strong>, so all the topics in Kafka are configured in Zookeeper.</li></ul><blockquote>So it has their configuration, which includes the number of partitions, the replication factor and additional configuration.</blockquote><blockquote>But it also has the list of ESR, which is the list of in sync Replicas for partitions.</blockquote><ul><li>Zookeeper will be used to perform <strong>leader elections.</strong></li></ul><blockquote>Zookeeper is going to do a vote between all his servers and elect a new leader .</blockquote><blockquote>leader election is very important, the faster it’s done, the faster your server will be up and running when one goes down.</blockquote><ul><li>Zookeeper will be used to store the <strong>Kafka cluster ID</strong>(randomly generated at the first startup of the cluster).</li><li>Zookeeper will be used to store the <strong>ACLs </strong>if security is enabled.</li></ul><blockquote>Access control list or ACLs can be related to topics, consumer groups and users.</blockquote><ul><li>Zookeeper will be used to store <strong>quotas </strong>configuration if enabled.</li></ul><h3>Zookeeper Quorums</h3><p>Zookeeper has a strict majority of servers up with 2N+1 quorums.</p><blockquote>If you have one server zookeeper in your quorum, zero servers can go down. So if you have one and it goes down, it’s broken.</blockquote><blockquote>If you have three servers, only one can go down.</blockquote><blockquote>If you have five servers, two can go down.</blockquote><blockquote>…</blockquote><p>Let’s do a comparison between 1, 3, 5 zookeepers to understand which are suitable for certain cases and which are not.</p><h4>1 zookeeper</h4><p>So let’s talk about the easiest, which has one zookeeper in the quorum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*z2iD20gJA9GiNuNAUhCQ7g.png" /><figcaption>1 zookeeper, photo by Author</figcaption></figure><p>When we have 1 zookeeper, it’s very easy to put in place without worrying about distribution.</p><p>But it’s clear that it’s not resilient at all !</p><p>If that zookeeper crashes, your Kafka cluster will crash.</p><blockquote>So Zookeeper one instance is really good for development purposes, but not good at all for production deployments.</blockquote><h4>3 zookeepers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*_AygaFphAYqoIF0g_cqtiQ.png" /><figcaption>3 zookeeper, photo by Author</figcaption></figure><p>In the case of 3 zookeepers one can go down as explained before.</p><p>This kind of server quorum setup is preferred for small Kafka deployments.</p><blockquote>A lot of deployments have done only have three Zookeeper servers, and that’s more than enough.</blockquote><p>Now let’s talk about five.</p><h4>5 zookeepers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*LZWTV68qQAjT_mSAOuqnHw.png" /><figcaption>5 zookeeper, photo by Author</figcaption></figure><p>As you can see, it becomes more complicated. Each zookeeper needs to talk with all each other. It’s very useful for big Kafka deployments. In this case, two servers have the possibility to go both down.</p><blockquote>Big companies like LinkedIn and Netflix, will have at least five Zookeeper servers and this allows for two servers to go down, which gives a little bit more flexibility in production.</blockquote><p>But on the other hand, you need very performant machines.</p><p>There’s so much networking happening and zookeeper is very sensible to latency, you need very performant machines and good structural decisions about the kind of machines and network cables you’re going to use.</p><h3>Zookeeper Configuration</h3><p>Zookeeper configuration and optimization could be really tricky. It depends on many configurations about the network and the cluster used, but there are some defaults we’re gonna use.</p><ul><li>First of all, Zookeeper needs to store data. So you need to indicate a data directory “<strong>dataDir”</strong>.</li><li>It has a <strong>port </strong>and usually is <strong>2181</strong>.</li><li>“<strong>MaxClientCnxns” </strong>connection will set it to zero, which means unlimited. You could disable it by setting a number of maximum number of connections if you want to.</li><li>The “<strong>tickTime ”</strong>is set to define heartbeats and 2000 (2 sec) is a good number.</li><li>“<strong>initLimit </strong>”is how many ticks they can take to initialize and synchronize. So if <strong>initLimit</strong> is 10, that means that 20 second (10*2s) can be used for the initial synchronization. if 20 seconds are over, It will fail.</li><li>“<strong>syncLimit </strong>”is how many sync are possible to be passed between a request and an acknowledgement.</li><li>Zookeepers servers “<strong>server.(n)” to define </strong>all your zookeeper servers</li></ul><blockquote>These configs are really important because they define timeouts for Zookeeper and these show how important network latency and performant machines are for a performant Zookeeper.</blockquote><p>Next step is to setup Zookeeper and Kafka cluster in AWS to see in depth all what we have learnt.</p><p><a href="https://towardsaws.com/zookeeper-setup-with-kafka-cluster-part-2-753226ffc18d">Zookeeper Setup with Kafka Cluster : Part 2</a></p><h3>References</h3><ul><li><a href="https://zookeeper.apache.org/">Apache ZooKeeper</a></li><li><a href="https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#ch_deployment">ZooKeeper: Because Coordinating Distributed Systems is a Zoo</a></li><li><a href="https://developer.confluent.io/courses/architecture/control-plane/">Kafka Control Plane: ZooKeeper, KRaft, and Managing Data</a></li></ul><p><a href="https://www.udemy.com/course/kafka-cluster-setup/?couponCode=KEEPLEARNING">https://www.udemy.com/course/kafka-cluster-setup</a></p><p>Thank you 🙏 for reading the article. Before you go, you can<strong> clap </strong>my article (50 times 👏), that will really help me out and boost this article to others ✍🏻❤️. <strong>Follow </strong>me on <a href="https://medium.com/@mengineer">Medium</a> to get my latest article.</p><p>Thank you 🫶!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=469dc4671198" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Imputation is a Remedy]]></title>
            <link>https://medium.com/@mengineer/data-imputation-is-a-remedy-4ac8ec00a9eb?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ac8ec00a9eb</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Tue, 12 Mar 2024 13:17:15 GMT</pubDate>
            <atom:updated>2024-03-12T16:58:26.064Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Eaa-H48z6yz6V4_hGFbRg.jpeg" /><figcaption>AI Generated, Inspired By <a href="https://fr.freepik.com/photos-gratuite/long-shot-adulte-dans-nature-formes-geometriques-3d_59583667.htm#fromView=search&amp;page=1&amp;position=24&amp;uuid=9509f48c-ea09-4bdf-9a02-1f2e0748200e">Freepik</a></figcaption></figure><h4>Don’t Let Missing Values Stop You !</h4><h3>Data Imputation is your Remedy</h3><h4>How Missing Values could be a hindrance to analysis result</h4><p>During the COVID pandemic, healthcare systems did face a lot of challenges, one of them was the time constraint. Data collection and analysis need to be rapid so they could combat the virus effectively. Health records (EHRs) became a crucial source of data for tracking the spread of the disease and identifying risk factors for severe outcomes.</p><p>However, the data collected were often incomplete due to the overwhelming number of patients, variations in data collection protocols across institutions, and the novelty of the virus itself, which led to missing information on symptoms, comorbidities, treatment outcomes, and demographic details.</p><p>In this article, we will see some of challenges faced during the analysis and how data imputation could be a real remedy.</p><h3>Scenario1: Track CFR over time</h3><p>During the survey, they need to track COVID-19 cases, recoveries, and fatalities over time in multiple locations.</p><p>Let’s simulate a Dataset :</p><pre>import pandas as pd<br>import numpy as np<br><br>data = {<br>    &#39;Date&#39;: [&#39;2020-07-01&#39;, &#39;2020-07-02&#39;, &#39;2020-07-03&#39;, &#39;2020-07-04&#39;, &#39;2020-07-05&#39;],<br>    &#39;Location&#39;: [&#39;Location A&#39;] * 5,<br>    &#39;Total Cases&#39;: [100, 150, 200, 250, 300],<br>    &#39;Total Recoveries&#39;: [50, np.nan, 150, np.nan, 250],  <br>    &#39;Total Deaths&#39;: [10, 20, np.nan, 40, np.nan]  <br>}<br><br>df = pd.DataFrame(data)<br><br>original_df = df.copy()<br><br>original_df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*348ZQUjVGF_1w2RPEQTzXA.png" /><figcaption>Dataset with missing values, Image By Author</figcaption></figure><p>The goal is to calculate the “Inaccurate case fatality rate” (CFR):</p><blockquote><strong>CFR </strong>= (Total Deaths / Total Cases) * 100.</blockquote><p>As you may notice, there is a lot of null values in “Total Recoveries” and “Total Deaths”, thus CFR could not be calculated for some periods.</p><p><strong>Solution</strong></p><p>To fix this problem, we will apply data imputation on the two columns where data are missing using the <strong>mean </strong>method.</p><blockquote>The mean imputation method is commonly used especially when the missingness is assumed to be random or when no trend is assumed in the time series data.</blockquote><p>In python, we will use fillna() and mean() functions</p><pre><br># Imputation with mean <br>df[&#39;Total Recoveries&#39;].fillna(df[&#39;Total Recoveries&#39;].mean(), inplace=True)<br>df[&#39;Total Deaths&#39;].fillna(df[&#39;Total Deaths&#39;].mean(), inplace=True)<br><br># Calculate CFR after imputation<br><br>df[&#39;CFR&#39;] = (df[&#39;Total Deaths&#39;] / df[&#39;Total Cases&#39;]) * 100<br><br>df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bOrpUKqRuV-dxPRyoeRXag.png" /><figcaption>Dataset after data imputation, Image By Author</figcaption></figure><h3>Scenario2: Track COVID-19 Vaccinations</h3><p>During the analysis of daily COVID-19 vaccination over a month, missing values for vaccination rates could exist due to reporting delays or errors.</p><p>Let’s simulate this:</p><pre>from datetime import timedelta<br><br># Generating a date range for one month<br>date_range = pd.date_range(start=&#39;2021-07-01&#39;, end=&#39;2021-07-31&#39;)<br><br># Simulate COVID-19 vaccination rates with some missing values<br>np.random.seed(42)  # For reproducibility<br>daily_vaccinations = np.random.randint(5000, 10000, size=len(date_range))<br><br># Intentionally setting some values to NaN to simulate missing data<br>for i in range(5, 10):<br>    daily_vaccinations[i] = np.nan<br>for i in range(20, 25):<br>    daily_vaccinations[i] = np.nan<br><br># Create the dataset<br>simulated_df = pd.DataFrame({<br>    &#39;Date&#39;: date_range,<br>    &#39;Daily Cases&#39;: daily_cases,<br>    &#39;Daily Vaccinations&#39;: daily_vaccinations<br>})<br><br># Plotting only the original data without imputation<br>fig, ax = plt.subplots(figsize=(12, 6))<br><br># Daily Vaccinations<br>ax.plot(simulated_df[&#39;Date&#39;], simulated_df[&#39;Daily Vaccinations&#39;], label=&#39;Daily Vaccinations&#39;, marker=&#39;o&#39;, linestyle=&#39;-&#39;, color=&#39;green&#39;)<br>ax.set_title(&#39;Daily COVID-19 Vaccinations Over Time&#39;)<br>ax.set_xlabel(&#39;Date&#39;)<br>ax.set_ylabel(&#39;Number of Vaccinations&#39;)<br>ax.legend()<br>ax.grid(True)<br><br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dfwMUhEFP3sx-QsvRojB1w.png" /><figcaption>The Daily COVID-19 Vaccination Over Time before data imputation, Image By Author</figcaption></figure><p>As you can see, the green line represents the daily vaccinations over the month of July 2021, including the days with missing data. There is a lot of gaps that lead to inaccurate assessments of the situation.</p><p><strong>Solution</strong></p><p>So let’s fix this by data imputation. This time we will use the “forward fill” method.</p><blockquote>Here we choose Forward fill method since there is not a lot of changes, so we assume that the best estimate for missing data is the last observed data point.</blockquote><pre><br># Impute missing values using forward fill<br>imputed_df = simulated_df.fillna(method=&#39;ffill&#39;)<br><br># Plotting the original and imputed data again<br>fig, ax = plt.subplots(figsize=(12, 6))<br><br># Daily Vaccinations<br>ax.plot(simulated_df[&#39;Date&#39;], simulated_df[&#39;Daily Vaccinations&#39;], label=&#39;Original Daily Vaccinations&#39;, marker=&#39;o&#39;, linestyle=&#39;-&#39;, color=&#39;green&#39;)<br>ax.plot(imputed_df[&#39;Date&#39;], imputed_df[&#39;Daily Vaccinations&#39;], label=&#39;Imputed Daily Vaccinations&#39;, marker=&#39;x&#39;, linestyle=&#39;--&#39;, color=&#39;orange&#39;)<br>ax.set_title(&#39;Daily COVID-19 Vaccinations Over Time&#39;)<br>ax.set_xlabel(&#39;Date&#39;)<br>ax.set_ylabel(&#39;Number of Vaccinations&#39;)<br>ax.legend()<br>ax.grid(True)<br><br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MxSpNbvgWpIti_pAzF_ouQ.png" /><figcaption>The Daily COVID-19 Vaccination Over Time after data imputation, Image By Author</figcaption></figure><h3>Conclusion</h3><p>Data imputation is a great remedy for the pervasive issue of missing data across various datasets, including those critical to understanding and combating public health crises like the COVID-19 pandemic. The choice of imputation method whether it be filling in missing values with means, carrying forward previous values, or more complex statistical methods — depends on the nature of the data and the specific analysis goals. This process not only enhances the integrity of statistical models and predictions but also enables informed decision-making based on more comprehensive data insights.</p><p>If you have any questions, feedback, or would like to share your experiences, please feel free to reach out in the comments section.</p><p><strong>Clap </strong>my article 50 times 👏, that will really help me out and boost this article to others ✍🏻❤️.</p><p>Thank you 🫶!</p><h3>Want to connect?</h3><ul><li>📖 Follow me on <a href="https://medium.com/@mengineer">Medium</a></li><li>📖 Connect with me on <a href="https://www.linkedin.com/in/smazouzi-engineer/">LinkedIn</a></li></ul><h3>References</h3><p><a href="https://en.wikipedia.org/wiki/Imputation_(statistics)">Imputation (statistics) - Wikipedia</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ac8ec00a9eb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlock the Power of Apache Kafka with The Official Docker Image]]></title>
            <link>https://medium.com/towards-data-engineering/unlock-the-power-of-apache-kafka-with-the-official-docker-image-5a65192e618b?source=rss-b76a0fcaeed4------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a65192e618b</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Sara M.]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 04:01:39 GMT</pubDate>
            <atom:updated>2024-05-18T12:47:41.454Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DAxd3FcNhGkLv5TF" /><figcaption>Photo by <a href="https://unsplash.com/@joshuvadaniel?utm_source=medium&amp;utm_medium=referral">Joshuva Daniel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>A Step by Step Kafka Guide</h4><h4>A detailed Guide To Build and Deploy Kafka With the official docker image in KRaft mode</h4><p>Do you remember our latest Kafka Article ? <br>If you don’t it’s okay, I will tell you ^_^.</p><p>Few days ago,we celebrated the new big event of introducing the official Kafka image in the <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka?source=post_page-----b93d903aeb95--------------------------------">latest release of Kafka 3.7</a></p><p><a href="https://medium.com/@mengineer/new-kafka-anniversary-b93d903aeb95">New Kafka Anniversary</a></p><p>Since this docker image is new, I didn’t find enough articles about the subject.</p><p>So let’s do a detailed guide about deploying a Kafka cluster using docker-compose.</p><h3>Prerequisites</h3><ul><li>Docker</li><li><a href="https://www.google.com/search?client=firefox-b-d&amp;q=portainer">Portainer</a> (to manage containers)</li></ul><h3>Setup a Cluster of 3 nodes</h3><h4>The docker-compose file</h4><p>We’re gonna use the official docker-compose <a href="https://github.com/apache/kafka/blob/trunk/docker/examples/jvm/single-node/plaintext/docker-compose.yml">example published by Apache Kafka in Github</a></p><pre># Licensed to the Apache Software Foundation (ASF) under one or more<br># contributor license agreements.  See the NOTICE file distributed with<br># this work for additional information regarding copyright ownership.<br># The ASF licenses this file to You under the Apache License, Version 2.0<br># (the &quot;License&quot;); you may not use this file except in compliance with<br># the License.  You may obtain a copy of the License at<br>#<br>#    http://www.apache.org/licenses/LICENSE-2.0<br>#<br># Unless required by applicable law or agreed to in writing, software<br># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,<br># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.<br># See the License for the specific language governing permissions and<br># limitations under the License.<br><br>---<br>version: &#39;2&#39;<br>services:<br>  kafka-1:<br>    image: apache/kafka:latest<br>    hostname: kafka-1<br>    container_name: kafka-1<br>    ports:<br>      - 29092:9092<br>    environment:<br>      KAFKA_NODE_ID: 1<br>      KAFKA_PROCESS_ROLES: &#39;broker,controller&#39;<br>      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: &#39;CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT&#39;<br>      KAFKA_CONTROLLER_QUORUM_VOTERS: &#39;1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093&#39;<br>      KAFKA_LISTENERS: &#39;PLAINTEXT://:19092,CONTROLLER://:9093,PLAINTEXT_HOST://:9092&#39;<br>      KAFKA_INTER_BROKER_LISTENER_NAME: &#39;PLAINTEXT&#39;<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:19092,PLAINTEXT_HOST://localhost:29092<br>      KAFKA_CONTROLLER_LISTENER_NAMES: &#39;CONTROLLER&#39;<br>      CLUSTER_ID: &#39;4L6g3nShT-eMCtK--X86sw&#39;<br>      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1<br>      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0<br>      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1<br>      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1<br>      KAFKA_LOG_DIRS: &#39;/tmp/kraft-combined-logs&#39;<br><br>  kafka-2:<br>    image: apache/kafka:latest<br>    hostname: kafka-2<br>    container_name: kafka-2<br>    ports:<br>      - 39092:9092<br>    environment:<br>      KAFKA_NODE_ID: 2<br>      KAFKA_PROCESS_ROLES: &#39;broker,controller&#39;<br>      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: &#39;CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT&#39;<br>      KAFKA_CONTROLLER_QUORUM_VOTERS: &#39;1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093&#39;<br>      KAFKA_LISTENERS: &#39;PLAINTEXT://:19092,CONTROLLER://:9093,PLAINTEXT_HOST://:9092&#39;<br>      KAFKA_INTER_BROKER_LISTENER_NAME: &#39;PLAINTEXT&#39;<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:19092,PLAINTEXT_HOST://localhost:39092<br>      KAFKA_CONTROLLER_LISTENER_NAMES: &#39;CONTROLLER&#39;<br>      CLUSTER_ID: &#39;4L6g3nShT-eMCtK--X86sw&#39;<br>      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1<br>      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0<br>      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1<br>      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1<br>      KAFKA_LOG_DIRS: &#39;/tmp/kraft-combined-logs&#39;<br><br>  kafka-3:<br>    image: apache/kafka:latest<br>    hostname: kafka-3<br>    container_name: kafka-3<br>    ports:<br>      - 49092:9092<br>    environment:<br>      KAFKA_NODE_ID: 3<br>      KAFKA_PROCESS_ROLES: &#39;broker,controller&#39;<br>      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: &#39;CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT&#39;<br>      KAFKA_CONTROLLER_QUORUM_VOTERS: &#39;1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093&#39;<br>      KAFKA_LISTENERS: &#39;PLAINTEXT://:19092,CONTROLLER://:9093,PLAINTEXT_HOST://:9092&#39;<br>      KAFKA_INTER_BROKER_LISTENER_NAME: &#39;PLAINTEXT&#39;<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:19092,PLAINTEXT_HOST://localhost:49092<br>      KAFKA_CONTROLLER_LISTENER_NAMES: &#39;CONTROLLER&#39;<br>      CLUSTER_ID: &#39;4L6g3nShT-eMCtK--X86sw&#39;<br>      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1<br>      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0<br>      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1<br>      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1<br>      KAFKA_LOG_DIRS: &#39;/tmp/kraft-combined-logs&#39;</pre><p>As you can see, this docker compose is deploying 3 Kafka nodes in a plaintext mode.</p><p>There is multiple choices of <a href="https://github.com/apache/kafka/tree/trunk/docker/examples/jvm">docker-compose</a> files, it depends on what you want to deploy.</p><p>Also, notice that there is only Kafka services to deploy, no need for Zookeeper anymore ( It’s important to mention that Zookeeper is planned to be removed in Apache Kafka 4.0).</p><p>So Kafka is gonna work in <strong>KRaft mode</strong>.</p><h4>Kafka in KRaft Mode</h4><p>Apache Kafka Raft (KRaft) is the consensus protocol that was introduced in <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum">KIP-500</a> to remove Apache Kafka’s dependency on ZooKeeper for metadata management.</p><p>This greatly simplifies Kafka’s architecture by consolidating responsibility for metadata into Kafka itself, rather than splitting it between two different systems: ZooKeeper and Kafka.</p><p>KRaft mode makes use of a new quorum controller service in Kafka which replaces the previous controller.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mFGu6h6S0jAbmIPhRAH0iQ.jpeg" /><figcaption>the 2 modes, image by <a href="https://developer.confluent.io/learn/kraft/">Confluent</a></figcaption></figure><p>If we use Kafka in KRaft mode, we do not need to use ZooKeeper for cluster coordination or storing metadata. Kafka coordinates the cluster itself using brokers that act as controllers. Kafka also stores the metadata used to track the status of brokers and partitions.</p><p><strong>1- The first step is creating an ID to identify the cluster</strong></p><p>The ID is used when creating logs for the brokers you add to the cluster.</p><p>If we return to the docker-compose file, we will see the CLUSTER_IDgenerated for the 3 nodes:</p><pre>CLUSTER_ID: &#39;4L6g3nShT-eMCtK--X86sw&#39;</pre><p><strong>2- Setup the necessary configuration for each broker</strong></p><p>If we return to the docker-compose above, we will see</p><ul><li>The three nodes got roles of broker and controller</li></ul><pre>KAFKA_PROCESS_ROLES: &#39;broker,controller&#39;</pre><ul><li>A unique node.id is specified for each node in the cluster</li></ul><pre>KAFKA_NODE_ID: 1</pre><pre>KAFKA_NODE_ID: 2</pre><pre>KAFKA_NODE_ID: 3</pre><ul><li>KAFKA_CONTROLLER_QUORUM_VOTERS to specify a set of brokers that will participate in the quorum for electing the controller, using the format nodeId@host:port. Here, the 3 brokers are included in the list.</li></ul><pre>KAFKA_CONTROLLER_QUORUM_VOTERS: &#39;1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093&#39;</pre><p><strong>3- Set up log directories ( where the Kafka data logs are stored) for each node</strong></p><p>If we return to the docker-compose above, we will see that the KAFKA_LOG_DIRS is set to the same value for the 3 nodes</p><pre>KAFKA_LOG_DIRS: &#39;/tmp/kraft-combined-logs&#39;</pre><h4>Other Parameters &amp; Configs</h4><p>Let’s run through the other parameters in the docker-compose file:</p><ul><li>image</li></ul><pre>image: apache/kafka:latest</pre><p>It does specify the docker image to be used for the container. In this case, apache/kafka:latest indicates that the container will use the latest version of the Official Apache Kafka image available on Docker Hub.</p><ul><li>hostname</li></ul><pre>hostname: kafka-1</pre><pre>hostname: kafka-2</pre><pre>hostname: kafka-3</pre><p>It sets the hostname inside the container. This is particularly useful for network communication between containers, especially in a setup where services need to communicate with each other using hostnames rather than IP addresses.</p><p>In this case, each Kafka broker is given a unique hostname (kafka-1, kafka-2, kafka-3) for identification within the internal network.</p><ul><li>container_name</li></ul><pre>container_name: kafka-1</pre><pre>container_name: kafka-2</pre><pre>container_name: kafka-3</pre><p>This specifies a custom name for the container. It’s a way to assign a readable, user-defined name to the container instead of the auto-generated ones. The names here (kafka-1, kafka-2, kafka-3) match their respective hostnames for consistency and ease of identification.</p><ul><li>ports</li></ul><pre>ports:<br>      - 29092:9092</pre><pre>ports:<br>      - 39092:9092</pre><pre>ports:<br>      - 49092:9092</pre><p>This maps the container’s ports to the host’s ports.The port mappings are defined as &lt;host_port&gt;:&lt;container_port&gt;, allowing external applications (e.g., Kafka clients) to connect to the Kafka brokers.</p><p><strong>The other Environment Variables</strong></p><ul><li>KAFKA_LISTENERS:</li></ul><pre>KAFKA_LISTENERS: &#39;PLAINTEXT://:19092,CONTROLLER://:9093,PLAINTEXT_HOST://:9092&#39;</pre><p>This config configures the listeners for network connections. In this case, there are three: one for internal communication between brokers (PLAINTEXT), one for controller communication (CONTROLLER), and one exposed to the host (PLAINTEXT_HOST).</p><ul><li>KAFKA_INTER_BROKER_LISTENER_NAME:</li></ul><pre>KAFKA_INTER_BROKER_LISTENER_NAME: &#39;PLAINTEXT&#39;</pre><p>This config sets the listener name for inter-broker communication. Here, it’s set to PLAINTEXT, indicating that brokers communicate with each other using the PLAINTEXT listener.</p><ul><li>KAFKA_TRANSACTION_STATE_LOG_MIN_ISR:</li></ul><pre>KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1</pre><p>This is the minimum number of in-sync replicas (ISRs) that must be available for the transaction state log to be writable. Set to 1 in this configuration.</p><ul><li>KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR:</li></ul><pre>KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1</pre><p>This config sets the replication factor for the transaction state log. Like the offsets topic, it’s set to 1 for this simple setup.</p><ul><li>KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR</li></ul><pre>KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1</pre><p>This is the replication factor for the offsets topic. Since this is a minimal setup, it&#39;s set to 1.</p><ul><li>KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS</li></ul><pre>KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0</pre><p>Sets the initial delay before the first rebalance. It&#39;s set to 0 here, meaning rebalances start immediately without delay.</p><ul><li>KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:</li></ul><pre>KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: &#39;CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT&#39;</pre><p>This config maps listener names to security protocols. This setup indicates that the CONTROLLER and PLAINTEXT listeners use the PLAINTEXT protocol.</p><h3>Le’ts Deploy the Cluster</h3><p>Let’s deploy our cluster by running this command:</p><pre>docker-compose up -d</pre><p>If we go check in <a href="http://localhost:9000/">Portainer</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eu1roUGglr1g30XmLV10GQ.png" /><figcaption>image by <a href="https://medium.com/@mengineer">Author</a></figcaption></figure><p>We will find the 3 nodes running.</p><h4>Let’s check our Cluster Health</h4><p>Choose one broker, and go inside the container and run this command:</p><pre>./kafka-topics.sh --create --topic &quot;NewTopic&quot; --bootstrap-server kafka-1:9092</pre><p>If all is good, this will create a new topic named “NewTopic”</p><p>Let’s check if the topic is created with this command:</p><pre>./kafka-topics.sh --list --bootstrap-server kafka-1:9092</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5aVNTGztlW9DenvLYUq9vQ.png" /><figcaption>Inside node kafka-1</figcaption></figure><p>And Voila, our topic is successfully created !</p><h3>Cleaning up</h3><p>Don’t forget to remove the running containers:</p><pre>docker-compose down</pre><p>If you have any questions, feedback, or would like to share your experiences, please feel free to reach out in the comments section.</p><p><strong>Clap </strong>my article 50 times 👏, that will really help me out and boost this article to others ✍🏻❤️.</p><p>Thank you 🫶!</p><h3>References</h3><ul><li><a href="https://kafka.apache.org/documentation/">Apache Kafka</a></li><li><a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka?source=post_page-----b93d903aeb95--------------------------------">KIP-975: Docker Image for Apache Kafka</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a65192e618b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/unlock-the-power-of-apache-kafka-with-the-official-docker-image-5a65192e618b">Unlock the Power of Apache Kafka with The Official Docker Image</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>