<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nelsonjoseph on Medium]]></title>
        <description><![CDATA[Stories by Nelsonjoseph on Medium]]></description>
        <link>https://medium.com/@nelsonjoseph123?source=rss-27304eff0f39------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UoHfqvIRBGBQWR7KCIjn6w.png</url>
            <title>Stories by Nelsonjoseph on Medium</title>
            <link>https://medium.com/@nelsonjoseph123?source=rss-27304eff0f39------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:03:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nelsonjoseph123/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Real-Time Twitter Trend Analysis with Kafka and AWS]]></title>
            <link>https://towardsdev.com/real-time-twitter-trend-analysis-with-kafka-and-aws-4a4e04682e3a?source=rss-27304eff0f39------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a4e04682e3a</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Nelsonjoseph]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 21:52:37 GMT</pubDate>
            <atom:updated>2024-06-26T16:05:49.917Z</atom:updated>
            <content:encoded><![CDATA[<h4>Creating a Robust Data Streaming Infrastructure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wEfM5tz4OUPkVVFLLck1Rw.png" /></figure><h4><strong>About this Project</strong></h4><p>The aim of this project goes beyond analyzing Twitter tweet trends, which can be achieved through direct data scraping. Instead, our goal is to establish a robust infrastructure for real-time data collection. This setup involves transferring data from a producer to a consumer using a Kafka server. By demonstrating the construction of an efficient and scalable data pipeline, we illustrate the process of real-time data processing.</p><p>Explore the code for this project on GitHub: <a href="https://github.com/nelson123-lab/Twitter-trends-real-time-data-analysis-using-Kafka-and-AWS"><strong><em>Github Repository</em></strong></a></p><h3>Architecture Overview</h3><p>The architecture of this project is designed to handle the entire data lifecycle, from ingestion to analysis and visualization. Here’s a high-level overview:</p><ol><li><strong>Twitter Data Collection</strong>:</li></ol><ul><li>We use the Twitter API to stream tweets in real time.</li><li>Tweets are produced for Apache Kafka topics.</li></ul><p><strong>2. Real-Time Data Streaming</strong>:</p><ul><li>Apache Kafka, running on AWS EC2 instances, is used to handle the real-time data stream.</li><li>Kafka brokers manage the data flow, ensuring scalability and fault tolerance.</li></ul><p><strong>3. Data Storage</strong>:</p><ul><li>The incoming tweet data from Kafka is stored in Amazon S3 buckets.</li><li>Data is organized in a structured manner for efficient retrieval and processing.</li></ul><p><strong>4. Data Crawling with AWS Glue</strong>:</p><ul><li>AWS Glue crawlers are configured to scan the data stored in S3.</li><li>Glue automatically catalogs the data, creating a unified metadata repository.</li></ul><p><strong>5. Data Analysis with AWS Athena</strong>:</p><ul><li>Using AWS Athena, we can query the cataloged data directly from S3.</li><li>Athena’s serverless architecture allows for interactive and fast analysis using standard SQL queries.</li></ul><p><strong>6. Data Visualization using AWS QuickSight</strong>:</p><ul><li><strong>AWS QuickSight</strong>: Provides interactive dashboards and visualizations for the analyzed data.</li></ul><p><strong>7</strong>. <strong>Real-Time Monitoring</strong>:</p><ul><li>Continuous monitoring: Ensures the pipeline’s performance and reliability.</li></ul><p>This architecture ensures a robust and scalable solution for real-time data processing, analysis, and visualization, leveraging various AWS services to manage different stages of the data lifecycle.</p><h4>Kafka server setup</h4><p>We will first focus on the Kafka setup it’s easier to collect data using the Twitter API. The first step in the process is to sign up for an AWS account and create an AWS EC2 instance. In this instance, we will host the Kafka server to facilitate real-time data transfer between producers and consumers.</p><p>When the EC2 instance is created we have to create a folder and download the .pem key file. Now we use a similar command from your EC2 instance</p><pre>ssh -i “kafka_stock_market.pem” ec2-user@ec2–52–15–147–25.us-east-2.compute.amazonaws.com</pre><p>To download and extract Apache Kafka version 3.6.2 to your local machine, we can use the following commands:</p><pre>wget https://downloads.apache.org/kafka/3.6.2/kafka_2.12-3.6.2.tgz<br>tar -xvf kafka_2.12–3.6.2.tgz</pre><p>We need Java installed because Apache Kafka is a distributed streaming platform that is primarily written in Java and Scala.</p><pre>sudo yum install java</pre><p>Navigate to the location of the Kafka folder using the ls command, then use cd kafka_2.12-3.6.2. Now start Zookeeper using the command below:</p><pre>bin/zookeeper-server-start.sh config/zookeeper.properties</pre><p>To run the Kafka server, we need to start an EC2 instance in a new terminal while keeping Zookeeper running in another terminal using the command below.</p><pre>bin/kafka-server-start.sh config/server.properties</pre><p>This may lead to errors indicating not enough memory to run the Kafka server. We need to ensure that it has enough memory to operate efficiently without consuming excessive system resources. Now use the command below.</p><pre>export KAFKA_HEAP_OPTS=&quot;-Xmx256M -Xms128M&quot; </pre><p>This sets the initial and maximum heap memory sizes for the Kafka server&#39;s JVM, allowing you to control its memory usage. Now use the command again to start the kafka server.</p><pre>bin/kafka-server-start.sh config/server.properties</pre><p>Now we need to change the IP from private DNS to public DNS after stopping both Zookeeper and the Kafka server. We cannot access the private DNS from our local machine, so we need to make the EC2 instance public. Use the command below to view the config properties:</p><pre>sudo nano config/server.properties</pre><p>Change your hostname to the public IP as shown below:</p><pre>advertised.listeners=PLAINTEXT://your-host-name:9092<br>advertised.listeners=PLAINTEXT://52.15.147.25:9092</pre><p>Press Ctrl +X and Yes to save it. Now start Zookeeper and then start the Kafka server again. If Zookeeper is not working and raises an error indicating that a process is already running, we need to first identify the process ID of the event using the command below. The default port number for Zookeeper is 2181.</p><p>Use the following command to find the process ID for Zookeeper running on port 2181:</p><pre>sudo lsof -i :2181</pre><p>Now we need to kill the process using the PID (process ID). For example:</p><pre>sudo kill -9 28079</pre><p>Now start Zookeeper again. Then start the Kafka server. If it is refusing the connection, kill the existing process in a similar way running on port 9092, use the command:</p><pre>sudo lsof -i :9092</pre><p>Then, kill the Kafka server process using the PID. For example:</p><pre>sudo kill -9 12345</pre><p>We need to provide security access from our local machine. Go to the security settings in the EC2 instance, click on “Edit inbound rules,” and then allow the request from our machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*-JWqxIiwpiVWCpYv9Tm54w.png" /></figure><p>Click on “Add rule,” set the type to “All traffic,” and set the source to “Anywhere — IPv4.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*Te2KWdTotGEnKanLIZDcig.png" /></figure><p>Now we need to create a topic. To do this, open a new terminal, connect to the EC2 instance, and navigate to the Kafka directory.</p><p>Create the topic using the command below, replacing &lt;topic-name&gt; with your desired topic name and &lt;public-ip&gt; with the public IP of your EC2 instance:</p><pre>bin/kafka-topics.sh — create — topic demo_test — bootstrap-server 3.21.244.255:9092 — replication-factor 1 — partitions 1</pre><p>Create a producer with the same topic name.</p><pre>bin/kafka-console-producer.sh — topic demo_test — bootstrap-server 3.21.244.255:9092</pre><p>Open a new terminal for the consumer connect to EC2 open the Kafka directory and create a consumer.</p><pre>bin/kafka-console-consumer.sh — topic demo_test — bootstrap-server 3.21.244.255:9092</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*eBdJW6ZKsDH3b0stpTbUxA.png" /><figcaption>Kafka Producer on the right and Kafka Consumer on the left side</figcaption></figure><p>Now we can see real-time data transfer from the producer on the right to the consumer on the left.</p><p>Now we can try transferring data from the front end using a Google collab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*1KwYCPIQm1toSTA4MF9pJQ.png" /><figcaption>Kafka Producer</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*Dx2uh3PKVYHGQ-luLEUw9w.png" /><figcaption>Kafka Consumer</figcaption></figure><p>This is how we send data from the producer to the consumer and get the data in real-time from the consumer.</p><p>We need to set up Twitter tweet extraction by first creating a Twitter developer account using the free plan. There are free, basic, and pro versions of the API available. The free version might not be shown initially, so subscribe to the free version unless you intend to pay $100 monthly.</p><p>Now, use the tweet_data_collection.py file from the GitHub repository to extract data and integrate this code into the Kafka producer.</p><p>Now, we need to create an AWS S3 bucket and write scripts to transfer data from the consumer to the S3 bucket. First, we need to go to IAM, create a user with administrative access, and generate a secret key. Use the s3fs command below to connect to the S3 bucket.</p><pre>s3 = S3FileSystem(key=’API Key’, secret=’Secret_KEY’)</pre><p>It won’t connect if you don’t provide enough permissions for your local machine to access your S3 bucket. We need to configure the S3 bucket to allow access from our local machine if you are running locally. Download the AWS CLI to use the command line interface for Windows, open a new terminal, and use the command aws configure to set up access to the S3 bucket. This is necessary if you are using Jupyter Notebook or a Python script from your local machine.</p><p>You need to install boto3 using the command!pip install boto3when you are using Google Collab.</p><p>Now, start the consumer and let it run. Then, start the producer for a few seconds and stop it. You will see the data being updated in real-time in the S3 bucket. The reason we are stopping it is that the S3 bucket has limited space in the Free Tier, and it will get used up if it continuously scrapes all the real-time tweet data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*lev1SfAfsqckRwZU8kBGfQ.png" /><figcaption>Data is updated in real time in the S3 bucket</figcaption></figure><p>Now, create a crawler and choose our specific S3 bucket as the source, as shown below: s3://kafka-tweets-data/.</p><ul><li>Next, create a new role using IAM for the Glue crawler to access the data in S3 and all other AWS services. To do this, give administrative access by creating a new role in IAM named glue-admin-access. Assign this role to the Glue crawler.</li><li>Then, create a database by using the “Add Database” option, give it a name, and add it to the AWS Glue crawler. Run the crawler.</li></ul><p>Now, we can launch AWS Athena to query the results from S3. First, we need to specify a location to store the query results by editing the settings. Otherwise, we won’t be able to see the results. You can either create a new S3 bucket or use the same bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*hRp1z6Cqcxr_6TbHZ5lXsQ.png" /><figcaption>SQL queries in AWS Athena</figcaption></figure><p>Remember, AWS Athena is a paid service, so try it with a small set of data by running the Kafka producer for a short time. Otherwise, you will be charged depending on the usage.</p><p>Now, with the data, we can easily create a dashboard that dynamically updates according to the data using AWS QuickSight. AWS QuickSight is also a paid service and is not available with the free tier. It is a business intelligence service that allows you to create and publish interactive dashboards. QuickSight can directly connect to your data in S3, Athena, and other sources, providing a range of visualization options to help you understand your data. By setting up a data source and creating analyses in QuickSight, you can build dashboards that reflect the real-time data collected in your S3 bucket.</p><h4>Conclusion</h4><p>This project leverages a robust architecture to handle the entire data lifecycle from collection to visualization. Real-time data is collected from Twitter using the Twitter API and ingested by Apache Kafka. Data processing is managed by AWS Glue, which performs ETL tasks and dynamically updates schemas with AWS Glue Crawlers. The processed data is stored in AWS S3, providing a scalable and durable storage solution.</p><p>For data analysis, AWS Athena runs SQL queries directly on the S3 data, offering a serverless and flexible approach to querying large datasets. The insights derived from this analysis are then visualized using AWS QuickSight, which provides interactive dashboards and visualizations to make data-driven decisions.</p><p>Additionally, continuous real-time monitoring ensures the performance and reliability of the entire data pipeline. By utilizing various AWS services, this architecture offers a scalable, efficient, and comprehensive solution for real-time data processing, analysis, and visualization.</p><h4>References</h4><p>A big thanks to <a href="https://www.youtube.com/@DarshilParmar"><em>Darshil Parmar</em></a> for the data engineering project tutorials. It was great to learn from his videos.</p><ul><li><a href="https://youtu.be/KerNf0NANMo"><em>Stock Market Real-Time Data Analysis Using Kafka | End-To-End Data Engineering Project</em></a></li><li><a href="https://docs.aws.amazon.com/"><em>AWS Documentation</em></a></li></ul><h3>Before You Go</h3><p><em>Thanks for reading! If you want to get in touch with me, feel free to reach me at nelsonjoseph123@gmail.com or my </em><a href="https://www.linkedin.com/in/nj-lab/"><strong><em>LinkedIn Profile</em></strong></a><strong><em>.</em></strong><em> You can view my </em><a href="https://github.com/nelson123-lab"><strong><em>Github profile</em></strong></a><strong><em> </em></strong><em>for different data science projects.</em></p><p><a href="https://www.buymeacoffee.com/nelsonjoseX"><em>Buy me a coffee</em></a>☕</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a4e04682e3a" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/real-time-twitter-trend-analysis-with-kafka-and-aws-4a4e04682e3a">Real-Time Twitter Trend Analysis with Kafka and AWS</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a Virtual Environment and Github Repository : Part 2]]></title>
            <link>https://medium.com/@nelsonjoseph123/setting-up-a-virtual-environment-and-github-repository-part-2-bee5f77c75dc?source=rss-27304eff0f39------2</link>
            <guid isPermaLink="false">https://medium.com/p/bee5f77c75dc</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[virtual-environment]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nelsonjoseph]]></dc:creator>
            <pubDate>Tue, 21 Nov 2023 17:53:49 GMT</pubDate>
            <atom:updated>2023-11-21T17:53:49.155Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up a Virtual Environment and Github Repository : Part 2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*ly09FURnb3pHmlW03wXqrw.png" /><figcaption>Source: IStock Images</figcaption></figure><p>This blog is dedicated to the creation of a Virtual Environment for Individual Projects. If you aren’t aware of how can you create a Virtual Environment, I recommend going through my previous blog about creating a Virtual Environment and Github Repository for mulitple projects.</p><p><a href="https://medium.com/@nelsonjoseph123/setting-up-a-virtual-environment-and-github-repo-a-step-by-step-guide-107215bf0a7c">Setting up a Virtual Environment and GitHub Repo : A Step-by-Step Guide</a></p><p>Now I will explain how we can create a virtual environment for individual projects. The local virtual environment folder will serve as the local respository for your remote Github Repository. We have to follow similar steps till creating a virtual environment as shown in the first part.</p><p>There are two cases for individual projects.</p><ol><li>No Existing Github Repository for the projects</li><li>Existing Github repository.</li></ol><h4>1) No Existing Github Repository for the projects</h4><p>Let’s look at case when there is no existing Github Repository. Let’s create a Virtual Environment for Lane detection project. We have to run the command prompt as administrator to make to activate the Virtual Environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*ygojqBaVOAnwvxsDpiqU7A.png" /></figure><p>We have to navigate to the specific location where you want to create the Virtual Environment. Use the below commands.</p><pre># To navigate to the desired location.<br>cd location<br># To create a virtual environment named lane_detection<br>virtualenv lane_detection</pre><p>Now we have to open the lane_detection Virtual Environment folder on VScode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j03WZJ--vHS0oG5zab_j2A.png" /></figure><p>It will appear like the above screenshot. Now we have to edit the .gitignore file here since all the contents in Lib, Scripts, and Include gets tracked by Git.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7GwLouk_XTqih8uRkuWpPw.png" /></figure><p>We have to add these information in the .gitignore so that the library dependencies will not be added to the github. We only need information about the library dependencies not the entire files itself.</p><p>Now for this folder to serve as the github local repositoy, we need setup the Github Repositoy. We have to go to the Source Control (Ctrl+Shift+G) and initialize a repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WQzS5N9L0C83BLoiuLJLtQ.png" /></figure><p>This shows several files to be commited to the Github Repository don’t worry if you don’t a Github Repository we are yet to make a Github Repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u190vhV4OGgBB3KANNO1eA.png" /></figure><p>After clicking commit we have to type some commit message. Let’s keep it “Initial Commit” and click on the tick button on the right top corner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M26eM-TQolAd2lCwvefg0g.png" /></figure><p>After commiting the branch it will show the option to Publish Branch. The folder can be published as a private or public Github Repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FHU_Xpcdpa26SEnyOOdhvA.png" /></figure><p>After we publish the folder you can see the option to open the github repository as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*DOSQzxgjFwTUOhXSBSkJDg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BR8LkKtrykYSS3aJqsuYgA.png" /></figure><p>Now the repository is created and whatever changes you make in the local Virtual Environment folder will be shown as modification that you can commit to the remote Github Repository. Instead of initializing, committing and publishing to the Github, we can also use the Publish to Github button which automatically do altogether.</p><p>We can install the project related libraries by activating the Virtual Environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2F0ukzIdaa2xIjoIjN6MFw.png" /></figure><p>The above screenshot shows the result when I tried to activate the Virtual Environment from the VScode terminal. As mentioned earlier we won’t be able to activate the Virtual Environment unless we are running command prompt as an administrator.</p><p>Use the below commands to navigate to the virtual Environment folder and activate it.</p><pre># To navigate to the Virtual Environment directory.<br>cd folder_path_of_VE</pre><pre># To activate the Virtual Environment.<br>Scripts\activate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*MYyqDMc86ZhoWaIWNkn8wg.png" /></figure><p>The Virtual Environment is now activated and we can start installing the required libraries for our project.</p><p>Let’s say we need OpenCV and tensorflow. We can use the below commands to install it and get the <strong>requirements.txt</strong>.</p><pre># To install opencv and tensorflow packages.<br>pip install opencv-python tensorflow<br><br># To get the requirements.txt<br>pip freeze &gt; requirements.txt</pre><p>When we run the above commands we get a new file named <strong>requirements.txt</strong> that includes all the libraries that are present within our active <strong>VE</strong>. <strong>Always remember to install packages when the VE is active otherwise it won’t be updated in the VE.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0cZOQ7Dpwv1IBZYogLEBig.png" /></figure><p>Now we have a working environemnet setup for a project with a virtual environment which serves as the local Repository for the remote Github Repository.</p><h4>2) Existing Github repository</h4><p>Now let’s look at the case when you already have an existing Github repository and you wish to assign a folder on your device as the local repository of the remote Git repository. Here we are also making the local Git repository as the virtual environment folder too.</p><p>Let’s start by creating a virtual environment for object detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*3-D5gNZjdWdVroI2Wrk3Ww.png" /></figure><p>Now like what we did in the previous case let’s modify the .gitignore file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WrNXflXsiDTQUVmx1ls9lA.png" /></figure><p>Suppose we already have a Github Repository for this project and we wish to connect the Github repository with the local virtual environment folder to allow version controlling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m7AKKhI59F24t3Tyxl8EzQ.png" /><figcaption>Existing Github Repository</figcaption></figure><p>Let’s start the connection process by following the steps mentioned below.</p><p>The “git init” command initializes a new Git repository. This means that Git will start tracking changes within that directory and its subdirectories.</p><pre>git init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mHoHpdzuYa4MAxYKUj8z_Q.png" /></figure><p>The above command initialized an empty Git repository. Now we have to associate the local Git repository with the existing remote Github repository. For this first copy the Git Repository url and use the below command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*gcsYCEVpzYKYhXDXzyvWQQ.png" /></figure><pre>git remote add origin &lt;remote_repository_url&gt;</pre><p>Replace the <strong>&lt;remote_repository_url&gt; with the copied Github clone url.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*FZwhA_vxEKBTq8jo9uSaLg.png" /></figure><p>Now we have to get the latest changes in the existing Github repository to the local Git repository before pushing contents from the local repository.</p><pre># To fetch the latest changes from the remote repository<br>git fetch origin</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*y3RQVXJhhAielzng7xgJvA.png" /></figure><p>Now we have to create a new local branch named “<strong>main</strong>” that tracks the remote “<strong>main</strong>” branch and switch to it. This could be done using the below command.</p><pre>git checkout main</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*ucaOk10WepmEW8eYzyObrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*350SRjD4F1W5lSCkfpBgjg.png" /></figure><p>Now we can see that the <strong>model.py</strong> and the <strong>README.md</strong> is pulled to the local Git repository. Here the <strong>.gitignore</strong> and <strong>pyvenv.cfg</strong> files are shown as modification because the git that we initialized currently tracks only the files in the remote github repository and doesn’t track the files in the local Git repository.</p><p>Next step is to<strong> </strong>stage all the changes in the local Git repository and its subdirectories for the next commit so that git initialized can track of files in the local Git repository as well.</p><pre>git add .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*tM42sxycBOPxq1TEHexzuw.png" /></figure><p>The “git add .” means that you are telling Git to stage all the modifications in current repository for the next commit.</p><pre>git commit -m “Initial Commit”</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*T2FL3vI2zV_0AbeX24USXA.png" /></figure><p>After running <strong>git commit -m “Initial Commit”</strong> command, the changes are committed to the local repository, creating a new point in the project’s history.</p><p>Now we can push all the commit to the remote Git repository.</p><pre>git push</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*I2tXxN2u9TsSwFnaZLz0Eg.png" /></figure><p>The “git push” command pushes your committed changes from your local Git repository to a remote Git repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9XGPN3vW4EAODlwjbz5BWA.png" /></figure><p>Now we can see that the files in the local Git repository are updated in the remote Git repository as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J4m-Flg74uzjdvo_VYBi1Q.png" /></figure><p>Now all the files in the local Git repository and the remote Git repository will be tracked by Git.</p><p>Let’s activate our Virtual environment and install some necessary packages required for object detection and get our requirements.txt</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*NKShvC1QxnkLW9NeQNGIfg.png" /></figure><pre># To activate the Virtual Environment.<br>Scripts\activate<br># To install the packages.<br>pip install tensorflow ultralytics</pre><p>The <strong>ultralytics</strong> package has the <strong>YOLO</strong> class, used to create neural network models.</p><pre>pip freeze &gt; requirements.txt</pre><p>When we use the above command on an active virtual environment we can get the <strong>requirements.txt</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fiPiI_VagBLLzMTUdcM1ug.png" /></figure><p>We can just commit and push the changes to the remote Git repository and we are done with the setup.</p><h4>Conclusion</h4><p>Here we discussed main two ways through which we can setup a working environment for a project that supports version controlling. The local Git repository itself servers as the virtual environment. The things discussed in this blog are some topics that are not properly known to several peoople even with years of experience working with python. The steps for the second approach is when there is already an existing Git repository. The steps can vary when there are multiple branches in the Github Repository. Overall the idea is to make a proper working environment to start with python projects.</p><p>Here we don’t need to worry about the issue of unnecessary libraries getting installed and gets added in requirements.txt. Everything will be individual to each project.</p><p><strong>Note:-</strong> All commands used in this blog are present in the <a href="https://github.com/nelson123-lab/Medium-blogs-Contents/tree/main/Setting%20up%20a%20Virtual%20Environment%20and%20GitHub%20Repo%C2%A0:%20A%20Step-by-Step%20Guide"><em>Github Repo</em>sitory</a></p><h3>Before You Go</h3><p><em>Thanks for reading! If you want to get in touch with me, feel free to reach me at nelsonjoseph123@gmail.com or my </em><a href="https://www.linkedin.com/in/nj-lab/"><strong><em>LinkedIn Profile</em></strong></a><strong><em>.</em></strong><em> You can view my </em><a href="https://github.com/nelson123-lab"><strong><em>Github profile</em></strong></a><strong><em> </em></strong><em>for different data science projects.</em></p><p><a href="https://www.buymeacoffee.com/nelsonjoseX"><em>Buy me a coffee</em></a>☕</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bee5f77c75dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a Virtual Environment and GitHub Repo : A Step-by-Step Guide]]></title>
            <link>https://medium.com/@nelsonjoseph123/setting-up-a-virtual-environment-and-github-repo-a-step-by-step-guide-107215bf0a7c?source=rss-27304eff0f39------2</link>
            <guid isPermaLink="false">https://medium.com/p/107215bf0a7c</guid>
            <category><![CDATA[virtual-environment]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nelsonjoseph]]></dc:creator>
            <pubDate>Thu, 16 Nov 2023 02:33:50 GMT</pubDate>
            <atom:updated>2023-11-16T02:38:27.265Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up a Virtual Environment and GitHub Repo : A Step-by-Step Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fCRBaCC5uUbMIs6C.jpg" /><figcaption>Source : Internet</figcaption></figure><p>Setting up a virtual environment for Python projects is indeed a crucial step in software development. It allows for the isolation of project dependencies and ensures consistency across different environments. Creating a Python virtual environment in Windows involves a series of steps to ensure a smooth and error-free setup. Additionally, integrating the local and remote repositories on GitHub is essential for version control and collaboration.</p><p>While the steps you’ve outlined are valuable, it’s important to note that the process may vary based on the specific project requirements and the tools being used. It’s also crucial to emphasize the significance of documenting the setup process for future reference and sharing with other team members.</p><p>It would be daunting to read if I am explaining the different ways we can create virtual environments and how I finally ended up using this way. So let me get to the point straight.</p><p>Pre requisites:-</p><ul><li>Basic Knowledge about why we are creating a <strong>Virtual Environment</strong>.</li><li>Basic knowledge of Github</li><li>Basic knowledge of Git Bash</li><li>VS Code ( IDE )</li><li>Python</li><li>Command line Basic Commands</li></ul><p>Let’s start</p><p>There are two approaches for setting up a <strong>Virtual Environment(VE)</strong>:-</p><ol><li>When you have mulitple projects that requires almost similar dependencies and you will be working in the same environment for all of them.</li><li>If you want to create individual virtual environment for each of your projects and create their remote Github Repository.</li></ol><p>This blog will only focus on the first approach. I will try to make this simple as possible. The first thing we need to do is to install the python library ‘<strong>virtualenv</strong>’. This is the first step towards creating a virtual environment for your project. <strong>Virtual Environment</strong> will be represented as <strong>VE throughout the blog</strong>.</p><pre>pip install virtualenv</pre><p>Now the library is installed. we have to navigate through the command line to the folder where we want to make the <strong>Virtual Environment(VE)</strong>.</p><p>Let’s say I wish to create my <strong>VE</strong> in the desktop. When you open your command line the first time it will be looking something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*fURLOSszI0UiJ0Oq0n-LgQ.png" /></figure><p>Now we have to navigate to desktop so we use the ‘<strong>cd</strong>’ command.</p><pre>cd desktop</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*eDzX7iVNCpYWm3shOG9jyg.png" /></figure><p>Now to create a <strong>Virtual Environment(VE) </strong>use the below command.</p><pre>virtualenv venv</pre><p>Here, ‘<strong>venv</strong>’ is the <strong>VE</strong> name and can be replaced with any name. If you are planning to create an <strong>VE</strong> for Machine learning projects and you wish to work on different projects within it.</p><pre>virtualenv ML</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KWInAmSSQKvVjkdZyKl_tA.png" /></figure><p>This results in the creation of a folder named ML on our desktop with several files within it. Now we have to navigate to the <strong>VE</strong> we created.</p><p><strong>Note</strong>: It’s not activate yet. I will show you how you see when it is activated.</p><p>Let’s check the list of libraries within our <strong>ML(VE)</strong> folder using the below command.</p><pre>pip list</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*hGCrW6vVGLotE4fTjCQuGQ.png" /></figure><p>Now we are seeing a lot of libraries over here. It will be different for you and a huge number of libraries will be there too.</p><p>To activate the Environment we created we have to run the activate file within the Scripts folder within the Environment folder we created ie, ML folder in this case. Run the below command.</p><pre>Scripts\activate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*BTmyxrCXBrv3202Nt3WviA.png" /></figure><p>Now the environment is activated. You can see the environment name on the left side of the command line.</p><p>Let’s check the list of the libraries present within the activated virtual environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*lLn3bBC89sBFWwL3vd7QAQ.png" /></figure><p>Here, we can see there are only three libraries when the environment is activated. These three packages provide necessary tools to easily manage and install Python packages within that environment.</p><p>Let’s try installing some packages and check if it reflects in our environment. I am going to install numpy and tensorflow, use the below command.</p><pre>pip install numpy tensorflow</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wx4T598Hl1KB4EUgoRypbA.png" /></figure><p>Now, we can see that numpy and tensorflow is installed along with all the dependencies that is required by these libraries is installed. Now if your project require any other libraries use ‘<strong>pip</strong>’ to install them when the Environment is in an activated state otherwise it will not reflect specific to the Environment.</p><p>Now we can start on a project. Let’s say I am planning to create a project on <strong>“Stock Price Prediction using LSTM”</strong>. So let’s create a Github repository for this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*-Goq1G2NXz1KNHhU5PaNOg.png" /></figure><p>Type the desired name for the repository in the Repositoy Name and create the repository.</p><p>Now, inorder to make the local copy of the remote repository in Github. We have to navigate back to the ML (Environment folder).</p><p>First, you have to deactivate the Environment. Use the below command.</p><pre>deactivate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*CmuTM1eJ0STikNGsB-9U9A.png" /></figure><p>Now to make the local copy we have to use the below command.</p><pre>git clone https://github.com/nelson123-lab/Stock-price-prediction-using-LSTM.git</pre><p>The above line of code clones my Github Repository. If you are unaware about where to get the repository hyperlink, check the below screenshot. If you click on the ‘<strong>code</strong>’ button you can see the Github repository hyperlink</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*ycf2_ht7lEMmDNMrMlLbzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OgbDDsRIM84otdpxOg7dg.png" /></figure><p>Now, we can see a local copy of <strong>Stock-price-prediction-using-LSTM </strong>Github Repository is<strong> </strong>created within our ML(Environment) folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*d-zcaRSPCnbaIpjo9EOlhw.png" /></figure><p>Now open the folder <strong>Stock-price-prediction-using-LSTM i</strong>n vs code editor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*j7lXwVhgzLfN3XzQPsh8dA.png" /></figure><p>When you open this folder it will look like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ksy4AQX-BS50eBEpdBnR4A.png" /></figure><p>Now, if you make any changes to the local copy it will show in the commit area, these are the contents that are not present in the remote repository in Github Repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RwqqMt97hUQUfuG07PrZ-g.png" /></figure><p>If you click on the <strong>source control option</strong>( 3rd icon )on the slidebar you have option to commit the file to the remote repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*82JcPAAJdZeKaNNSrY9SoA.png" /></figure><p>When you click on the commit you have the option to type in the commit message, <strong>“modelfile”</strong> as in this case shown below. The commit message can be anything depending on what you what it to be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sx9WClRjGJk8r1gvyHoxKA.png" /></figure><p>Now you have the option to confirm the commit you did by pressing the confirm button on the right side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56dH3JUhrRKHdHK_l50h7g.png" /></figure><p>When you are done with the commit you will see the option to push your changes from local copy to remote repository in Github by clicking <strong>“sync changes”</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5pxvsjZHf4gre2L0PncR6g.png" /></figure><p>When you click sync changes it will push your code from local copy to remote repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*2AUozvz6Bos-g_Uyiw4knw.png" /></figure><p>We can see the <strong>Model.py</strong> file is added over here.</p><p>Since we are done with the cloning Github repository to local now we add our project and push the neccessary changes to github.</p><p>Now we have the Environment and the local copy of the project within the Virtual Environment let’s try to write some code within the <strong>Model.py</strong> file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2jZVggmXbbgJiR4G_C9osw.png" /></figure><p>But it’s showing the <strong>ModuleNotFoundError what might be the reason ?</strong></p><p>Think about it. We have already installed tensorflow earlier within the Environment after it was created.</p><p><strong>You might think it’s because the Python Environment is not activated</strong>. Even if you activate the virtual environment it can still show that the module is not found. Why you did all these works if you can’t get your desired packages installed.</p><p>The issue here is that you are not opening the right folder in the VS code. When you load a folder within the Virtual Environment folder you won’t be able to change your environment that you are working on, you won’t be having the option to choose the right interpreter for your virtual environment. You have to choose the Python Environment folder. Let’s see examples of both the cases.</p><p>When you are opening the project folder within the Environment and check for the active interpreter you won’t be able to see the activated Virtual Environment.</p><p>Use Ctrl+Shift+P to choose your interpreter in VS Code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mgOVT88_BRF1aEXoKR3cQw.png" /></figure><p>In the above figure you are not seeing any <strong>VE</strong> associated with our project. But if you are loading the <strong>VE</strong> folder directly, you can see there will an active <strong>VE</strong> that you can choose from the python interpreter menu as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LoQZQe-60lDcmVPn58NlCQ.png" /></figure><p>After selecting the <strong>Python 3.11.6(‘ML’: venv)</strong> you won’t be seeing any errors for the dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YPcZV07qrDVBIIZnaTiJpw.png" /></figure><p>Now you can make your project and push your updates to the github. We usually don’t add the <strong>VE files</strong> that includes the <strong>Lib, Scripts, Include</strong> within the Github Repository, instead add <strong>requirements.txt</strong> inside the Github Repository. In this case we don’t need to add the <strong>VE files</strong> in the <strong>.gitignore</strong> since these will not be tracked by the Github Repository.</p><p>You can use the below command when you have the <strong>VE</strong> in an activated state to get the <strong>requirements.txt</strong>.</p><pre>pip freeze &gt; requirements.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ZpGo2U32HiHR3uXfKSkdw.png" /></figure><p>Now we can see a <strong>requirements.txt</strong> file in our ML Environment folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nY30YyJhuLDBSk2FGKuDhw.png" /></figure><p>We have to move it to the projects folder and push it to the project repository. This allows anyone to try out your work just by installing all the dependencies from the requirements.txt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yuxVo2RqkMWvknZjrswMUg.png" /></figure><p>Now you are ready with the project setup to work further. If you are installing newer packages in future get the latest <strong>requirements.txt</strong> and push it into the Github Repository.</p><p>Now anyone forking the repository can get all the dependencies by running the following command. This will install all the packages neccessary for your project that were present in the Virtual Environment.</p><pre>pip install -r requirements.txt</pre><h3>Conclusion</h3><p>This is the case when you have multiple projects that requires similar dependencies and you want to create a <strong>Virtual Environment</strong> in general for them. There is an issue in using such a case. Suppose, you were working with a deep learning project that requires pytorch initially and currently you are working on a machine learning project that uses only sklearn, numpy and scipy. The <strong>requirements.txt</strong> will be having pytorch as well since the virtual environment have it and the project is not using it.</p><p>This will install all the unnecessary libraries mentioned in the <strong>requirements.txt</strong> along with the required libraries. We will explore creating GitHub repositories and virtual environments customized for individual projects in the next blog.</p><p><strong>Note:-</strong> All commands used in this blog are present in the <a href="https://github.com/nelson123-lab/Medium-blogs-Contents/tree/main/Setting%20up%20a%20Virtual%20Environment%20and%20GitHub%20Repo%C2%A0:%20A%20Step-by-Step%20Guide"><em>Github Repo</em>sitory</a></p><h3>Before You Go</h3><p><em>Thanks for reading! If you want to get in touch with me, feel free to reach me at nelsonjoseph123@gmail.com or my </em><a href="https://www.linkedin.com/in/nj-lab/"><strong><em>LinkedIn Profile</em></strong></a><strong><em>.</em></strong><em> You can view my </em><a href="https://github.com/nelson123-lab"><strong><em>Github profile</em></strong></a><strong><em> </em></strong><em>for different data science projects.</em></p><p><a href="https://www.buymeacoffee.com/nelsonjoseX"><em>Buy me a coffee</em></a>☕</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=107215bf0a7c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Face Restoration using GFPGAN, Lama Cleaner, and Real-ESRGAN]]></title>
            <link>https://medium.com/@nelsonjoseph123/face-restoration-using-gfpgan-lama-cleaner-and-real-esrgan-ad9bcf13efff?source=rss-27304eff0f39------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad9bcf13efff</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Nelsonjoseph]]></dc:creator>
            <pubDate>Sun, 06 Aug 2023 07:17:59 GMT</pubDate>
            <atom:updated>2023-08-06T07:17:59.573Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TgSIt7tpP1j2ELbWTsiXdA.png" /><figcaption>Noisy image to a restored image.</figcaption></figure><p>This blog demonstrates the process of performing face restoration on vintage photographs, resulting in a refined image, without incurring any cost, by utilizing open-source projects.</p><h4>Overview</h4><p>The blog comprises the utilization of various tools aimed at achieving optimal outcomes. Specifically, it addresses the restoration of a highly damaged image that poses significant challenges to the restoration process.</p><h4>Introduction</h4><p>Let’s get some insights into each of the algorithms and their respective applications.</p><blockquote><strong>GFPGAN</strong></blockquote><p>GFPGAN is a practical algorithm developed for real-world face restoration. It is designed to restore old photos or AI-generated faces. The goal of GFPGAN is to enhance the quality and appearance of faces in images. It leverages a pretrained face GAN (Generative Adversarial Network), such as StyleGAN2, to restore faces in a blind manner, meaning it can restore faces without any prior knowledge about the specific image.</p><p>GFPGAN works by utilizing the rich and diverse priors learned by the pretrained face GAN. It takes an input image with a face that needs restoration and applies a series of transformations and optimizations to enhance the face. These transformations are guided by the knowledge and patterns learned by the pretrained GAN during its training process.</p><p>The algorithm harnesses advanced deep learning techniques to thoroughly analyze and comprehend facial features, textures, and structures within the input image. Through this process, it generates a rejuvenated rendition of the face by applying acquired patterns and features derived from the pretrained GAN. Consequently, this approach significantly enhances the overall quality, sharpness, and authenticity of the restored face, resulting in a more visually appealing and realistic outcome.</p><p><strong><em>Official documentation GFPGAN : </em></strong><a href="https://github.com/TencentARC/GFPGAN"><strong><em>https://github.com/TencentARC/GFPGAN</em></strong></a></p><blockquote><strong>Lama Cleaner</strong></blockquote><p>Lama Cleaner is an image inpainting tool that harnesses the power of cutting-edge AI models. Its primary objective is to seamlessly eliminate undesirable objects, defects, or individuals from images, while also offering the capability to erase and substitute elements within the pictures. By leveraging advanced AI algorithms, this tool excels at accomplishing these tasks with exceptional accuracy and efficiency.</p><p>The operational foundation of Lama Cleaner revolves around the process known as inpainting. Inpainting is a sophisticated technique aimed at inferring and filling in missing or damaged portions of an image in a visually convincing manner. Lama Cleaner capitalizes on state-of-the-art AI models to effectively execute inpainting tasks, ensuring seamless and realistic restoration of images.</p><p>When you use Lama Cleaner, you provide an input image that contains the object or area you want to remove or replace. The AI model analyzes the image and identifies the object or area to be inpainted. It then generates a plausible replacement for the inpainted region based on the surrounding context and patterns learned from the training data.</p><p>The AI model used in Lama Cleaner has been trained on a large dataset of images, allowing it to learn various patterns, textures, and structures. This enables the model to generate realistic and visually appealing inpainted results.</p><p>Lama Cleaner offers multiple inpainting strategies and supports various AI models, giving you flexibility in choosing the approach that best suits your needs. It can be run as a desktop application, providing a user-friendly interface for inpainting tasks.</p><p>It’s important to note that while Lama Cleaner can produce impressive results, it may not always be perfect. The quality of the inpainted output depends on factors such as the complexity of the image, the size of the inpainted region, and the specific AI model and strategy used.</p><p><strong><em>Official documentation Lama-Cleaner : </em></strong><a href="https://github.com/Sanster/lama-cleaner"><strong><em>https://github.com/Sanster/lama-cleaner</em></strong></a></p><blockquote><strong>Real-ESRGAN</strong></blockquote><p>Real-ESRGAN is an image super-resolution algorithm that aims to enhance the resolution and quality of low-resolution images. It is an extension of the ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) algorithm, specifically designed for real-world blind super-resolution tasks.</p><p>The key idea behind Real-ESRGAN is to train the model using pure synthetic data to simulate complex real-world degradations. This approach allows the algorithm to learn and generalize better to real-world scenarios where the degradation factors are often unknown and complex.</p><p>Real-ESRGAN employs a high-order degradation modeling process to simulate various real-world degradations, such as noise, blur, and compression artifacts. It also takes into account common artifacts like ringing and overshoot during the synthesis process.</p><p>The training process of Real-ESRGAN involves a generator network and a discriminator network. The generator network takes a low-resolution image as input and generates a high-resolution image. The discriminator network aims to distinguish between the generated high-resolution images and real high-resolution images. The generator network is trained to fool the discriminator network by producing high-quality and visually pleasing images.</p><p>To improve the training stability and discriminator capability, Real-ESRGAN uses a U-Net discriminator with spectral normalization. Spectral normalization helps to control the Lipschitz constant of the discriminator, which leads to more stable training dynamics.</p><p><strong><em>Official documentation REAL-ESRGAN : </em></strong><a href="https://github.com/xinntao/Real-ESRGAN"><strong><em>https://github.com/xinntao/Real-ESRGAN</em></strong></a></p><h4>Implementation</h4><p>GFPGAN’s usage involves following its documentation for seamless real-world face restoration, improving the quality and appearance of images with faces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*DCS4bemFPC9GhvpKAthz4Q.png" /><figcaption>Installation instructions from documentation.</figcaption></figure><p>To initiate the usage of GFPGAN, the first step entails cloning the GitHub repository dedicated to the GFPGAN project.</p><p>Link to Repository : <a href="https://github.com/TencentARC/GFPGAN">https://github.com/TencentARC/GFPGAN</a></p><p>The process begins by opening the terminal and pasting the cloning command provided under the “<strong>Clone repo</strong>” heading. After successful cloning, navigate into the <strong>GFPGAN</strong> folder within the terminal using the command “<strong>cd GFPGAN</strong>”</p><pre>git clone https://github.com/TencentARC/GFPGAN.git<br>cd GFPGAN</pre><p>Following the previous step, the next action involves installing all the necessary dependencies within the GFPGAN folder. Simply copy each dependency command from the repository and paste them one by one into the terminal. Then, hit “Enter” to execute the installation process for each dependency.</p><pre># Install basicsr — https://github.com/xinntao/BasicSR<br># We use BasicSR for both training and inference<br>pip install basicsr<br><br># Install facexlib — https://github.com/xinntao/facexlib<br># We use face detection and face restoration helper in the facexlib package<br>pip install facexlib<br><br>pip install -r requirements.txt<br>python setup.py develop<br><br># If you want to enhance the background (non-face) regions with Real-ESRGAN,<br># you also need to install the realesrgan package<br>pip install realesrgan</pre><p>Subsequently, the next step is to execute the provided inference command, exactly as demonstrated in the repository, to begin the process of utilizing GFPGAN for face restoration.</p><pre>python inference_gfpgan.py -i inputs/whole_imgs -o results -v 1.3 -s 2</pre><p>The current state of the GFPGAN folder resembles the following structure, exemplified as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*1uEQXqd0CJYOXaFnb0hkPQ.png" /></figure><p>To initiate the process, the initial step involves executing the <strong>setup.py</strong> file. Following that, to utilize sample images, navigate to the <strong>inputs</strong> folder, specifically the <strong>whole_imgs</strong> subfolder, and insert your images. Then, execute the <strong>inference_gfpgan.py</strong> file to generate the desired output.The output will be in the <strong>results</strong> folder -&gt; <strong>restored_imgs</strong></p><p>Here is the output obtained with the our image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQG4FNLwPCCn4j_Fsf9AUw.png" /></figure><p>In this instance, it is noteworthy that despite an overall improvement in the quality of the input image, certain creases persist. To address this issue, an effective resolution entails employing either <strong>lama cleaner</strong> or <strong>Vance AI</strong> for a thorough cleaning process.</p><p>Based on the findings, it has been observed that Vance AI yields superior results compared to Lama Cleaner in terms of crease removal. Notably, Vance AI offers a premium photo enhancement service, requiring payment for its utilization. By simply uploading the picture and signing in, users can access the results effortlessly. Furthermore, a trial option is available, allowing users to evaluate the efficacy of the service before committing to a full subscription.</p><p><strong><em>Vance AI website : </em></strong><a href="https://vanceai.com/"><strong><em>https://vanceai.com/</em></strong></a></p><p>Here is the result from Vance AI when GFPGAN output is used as the input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TSiIDCECgEftyrs3vQ9rzA.png" /><figcaption>Output generated with Vance AI when GFPGAN output is given as input.</figcaption></figure><p>When the original image is utilized as the input for Vance AI, the results obtained are unsatisfactory in quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cuowkOD03JcAPUNM5NyDag.png" /><figcaption>Output generated with Vance AI when original image is given as input.</figcaption></figure><p>I have experimented with alternative AI tools such as Topaz AI, but unfortunately, it does not deliver satisfactory results, particularly in the context of creases. Topaz Photo AI functions as a paid service; however, it does offer a trial version for users to assess its performance before making a commitment.</p><p><strong><em>Topaz Photo AI Website : </em></strong><a href="https://www.topazlabs.com/topaz-photo-ai"><strong><em>https://www.topazlabs.com/topaz-photo-ai</em></strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WpcOGxieRYUz7Q1g661f7g.png" /><figcaption>Output generated with Topaz Photo AI when original image is given as input.</figcaption></figure><p>Now, let’s proceed with crease cleaning using Lama Cleaner, an open-source application known for its user-friendly nature. The process is notably straightforward, consisting of just two steps. Simply refer to the provided documentation for seamless utilization.</p><pre># In order to use the GPU, install cuda version of pytorch first.<br># pip install torch==1.13.1+cu117 torchvision==0.14.1 — extra-index-url https://download.pytorch.org/whl/cu117<br>pip install lama-cleaner<br>lama-cleaner — model=lama — device=cpu — port=8080</pre><p>To initiate the process, open the terminal and install the required package. Following that, execute the next command, which runs the lama-cleaner program on the CPU, while also configuring it to listen on port 8080 for further instructions and image processing tasks.</p><p>Now, the lama Cleaner will be running in the local host :-</p><p><a href="http://localhost:8080/"><strong><em>http://localhost:8080</em></strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*riPlf7mA2sKDsJdskDrieg.png" /><figcaption>Lama Output when GFPGAN output is given as input.</figcaption></figure><p>Here, it is noticeable that the quality of the picture declined following the application of Lama Cleaner. In contrast, Vance AI exhibited superior performance compared to Lama Cleaner, providing more favorable outcomes. The primary distinction between the two lies in the approach to crease removal: Lama Cleaner necessitates manual cleaning through the use of a brush, whereas Vance AI streamlines the process with a single click, efficiently eliminating creases.</p><p>Now, to enhance the image resolution, we are utilizing Real-ESRGAN, a powerful tool for super-resolution. The process is made straightforward through the utilization of the online demo workshop. By simply uploading the image, we can obtain the super-resolutioned version with improved clarity and sharpness.</p><p><strong><em>Real- ESRGAN website : </em></strong><a href="https://arc.tencent.com/en/ai-demos/imgRestore"><strong><em>https://arc.tencent.com/en/ai-demos/imgRestore</em></strong></a></p><p>Let’s see the results :-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_kbzHgRAgYk1O4Q8g_ecWA.png" /><figcaption>Real-ESRGAN Ouput when Lama Output is given as input.</figcaption></figure><h4>Conclusion</h4><p>In conclusion, this overview provides insights into the overall image restoration process, highlighting the potential of GFPGAN as a reliable starting point for further advancements in restoration techniques. The demonstration presented here showcases a cost-effective approach to face restoration, where no payment is required. While the process employs pre-trained models without fine-tuning, it is important to note that other paid photo restoration websites might offer more refined results.</p><p>Furthermore, the significance of Generative Adversarial Networks (GANs) in the real world cannot be understated, as they continue to revolutionize various applications, enabling innovations in art, design, healthcare, and many other domains, thereby shaping the future of AI-driven creativity and problem-solving.</p><h4>Before You Go</h4><p><em>Thanks for reading! If you want to get in touch with me, feel free to reach me at nelsonjoseph123@gmail.com or my </em><a href="https://www.linkedin.com/in/nelsonjoseph123/"><strong><em>LinkedIn Profile</em></strong></a><em>. You can view my </em><a href="https://github.com/nelson123-lab"><strong><em>Github profile</em></strong></a><strong><em> </em></strong><em>for different data science projects.</em></p><p><a href="https://www.buymeacoffee.com/nelsonjoseX"><em>Buy me a coffee</em></a>☕</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad9bcf13efff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chat GPT blog written by itself]]></title>
            <link>https://medium.com/@nelsonjoseph123/chat-gpt-blog-written-by-itself-9034f7f7bf85?source=rss-27304eff0f39------2</link>
            <guid isPermaLink="false">https://medium.com/p/9034f7f7bf85</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nelsonjoseph]]></dc:creator>
            <pubDate>Wed, 04 Jan 2023 05:44:19 GMT</pubDate>
            <atom:updated>2023-01-06T17:44:30.155Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*Utuwx-78IENGFy45S_0NHw.jpeg" /><figcaption>Photo by Open AI</figcaption></figure><p>I was experimenting with Chat GPT’s features because the internet was abuzz with talk of its potential applications. I’ve read several blogs about using Chat GPT as a chatbot, to learn to program, to answer frequently asked questions, and even as a way to combat depression.</p><p>Asking a few questions to Chat GPT about itself led to the creation of the information found on this blog.</p><p><strong>Prompt 1</strong></p><blockquote>Write a blog about how to use Chat GPT and what are its applications?</blockquote><p>Sure, here is a blog post on Chat GPT:</p><p>Chat GPT is a chatbot that uses the GPT (Generative Pre-trained Transformer) language model to generate human-like responses to user input. It is trained on a large dataset of human conversation, allowing it to understand and respond to a wide range of topics and situations.</p><p>To use Chat GPT, you simply need to enter your message into the chat interface and the chatbot will generate a response. The chatbot uses natural language processing (NLP) techniques to understand the meaning of your message and generate an appropriate response.</p><p>One of the main advantages of Chat GPT is that it is able to generate responses that are coherent and human-like, making it a useful tool for simulating conversation with a human. It can be used for a variety of applications, including customer service, language learning, and entertainment.</p><p>There are a few things to keep in mind when using Chat GPT:</p><ol><li>The chatbot is trained on a large dataset of human conversation, but it is still a machine and may not always generate responses that are appropriate or accurate.</li><li>Chat GPT is designed to generate responses based on the input it receives, so it is important to be clear and concise when communicating with the chatbot.</li><li>The chatbot may not always understand context or follow-up questions, so it is important to be patient and provide additional information as needed.</li></ol><p>Overall, Chat GPT is a useful tool for simulating conversation with a human and has many potential applications in a variety of fields.</p><p><strong>Prompt 2</strong></p><blockquote>What are the real-life applications where we can use chat gpt?</blockquote><p>There are several real-life applications where Chat GPT can be used:</p><ol><li>Customer service: Chat GPT can be used to provide assistance and answer questions for customers, allowing businesses to scale their customer service efforts without having to hire additional staff.</li><li>Language learning: Chat GPT can be used to simulate conversation with a native speaker, providing a way for language learners to practice their skills in a natural, conversational setting.</li><li>Entertainment: Chat GPT can be used to create interactive chatbots that can engage users in conversation and provide entertainment, such as through trivia games or chat-based stories.</li><li>Virtual assistants: Chat GPT can be used to create virtual assistants that can perform tasks and provide information to users, such as setting reminders or answering questions about the weather.</li><li>Social media: Chat GPT can be used to create chatbots that can interact with users on social media platforms and provide assistance or entertainment.</li><li>Online education: Chat GPT can be used to create chatbots that can provide interactive lessons and exercises for students, allowing for a more personalized learning experience.</li></ol><p>Overall, Chat GPT has many potential applications in a variety of fields and can be used to create chatbots that can assist, entertain, or educate users in real-life situations.</p><p><strong>Prompt 3</strong></p><blockquote>How to connect to chat gpt api and use it for some projects?</blockquote><p>To connect to the Chat GPT API and use it for a project, you will need to follow these steps:</p><ol><li>Sign up for an API key: First, you will need to sign up for an API key on the Chat GPT website. This will typically require creating an account and agreeing to the terms of service.</li><li>Install the API client: Next, you will need to install the API client for your programming language of choice. The API client is a library that allows you to access the API and use it in your project.</li><li>Authenticate your API key: After installing the API client, you will need to authenticate your API key by adding it to your code. This will allow you to access the API and use it in your project.</li><li>Make API requests: Once you have authenticated your API key, you can start making API requests to the Chat GPT API. This will typically involve specifying the input message and receiving the output response from the API.</li><li>Handle the API response: After making an API request, you will need to handle the API response in your code. This will typically involve processing the output from the API and using it in your project.</li></ol><p>Overall, connecting to the Chat GPT API and using it for a project involves installing the API client, authenticating your API key, making API requests, and handling the API response. With these steps, you can use the Chat GPT API to create chatbots or other projects that involve natural language processing and conversation simulation.</p><p><strong>Prompt 4</strong></p><blockquote>Resources to learn about transformers by which chat gpt is created?</blockquote><p>Here are some resources to learn about transformers, which are used to create Chat GPT:</p><ol><li>“Attention Is All You Need” (Vaswani et al., 2017) — This is the original research paper that introduced the transformer model. It provides a detailed explanation of how transformers work and why they are effective.</li><li>“The Illustrated Transformer” (J. Alber, 2019) — This is a blog post that provides a visual explanation of how transformers work. It includes diagrams and examples to help illustrate the concepts.</li><li>“Transformers: State-of-the-Art Natural Language Processing” (C. Olah et al., 2020) — This is a research paper that provides an overview of the transformer model and its applications in natural language processing.</li><li>“How Do You Train a GPT-3?” (S. Merity et al., 2020) — This is a blog post that provides a high-level overview of how GPT-3, a large transformer-based language model, is trained.</li><li>“The Transformer Architecture” (TensorFlow, 2021) — This is a tutorial from the TensorFlow website that provides an in-depth explanation of transformer architecture and how it is used in natural language processing.</li></ol><p>These resources should provide a good starting point for learning about transformers.</p><p>At first, I thought that Chat GPT failed to offer any website links or citation links in response to our inquiries. We confirm the validity of the information depending on the references from which the information is provided on the internet.</p><p>The Chat GPT can also offer reference hyperlinks upon request, but they do not direct users to the relevant page in most cases.</p><p><strong>Prompt 5</strong></p><blockquote>Could you share me the reference hyperlinks for learning about transformers used in chat GPT?</blockquote><p>Here are a few references that you may find helpful for learning about transformers, which are a type of neural network architecture that has been used in natural language processing tasks such as chatbots:</p><ul><li>“Attention Is All You Need” (<a href="https://arxiv.org/abs/1706.03762"><strong>https://arxiv.org/abs/1706.03762</strong></a>) is a research paper by Vaswani et al. that introduced the transformer architecture.</li><li>“The Illustrated Transformer” (<a href="http://jalammar.github.io/illustrated-transformer/"><strong>http://jalammar.github.io/illustrated-transformer/</strong></a>) is a blog post that provides a visual explanation of how transformers work.</li><li>“Transformers: State-of-the-art Natural Language Processing” (<a href="https://ai.googleblog.com/2017/08/transformers-state-of-art-natural.html"><strong>https://ai.googleblog.com/2017/08/transformers-state-of-art-natural.html</strong></a>) is a blog post by Google AI that discusses the use of transformers in natural language processing.</li><li>“Transformer Networks: A Comprehensive Overview” (<a href="https://towardsdatascience.com/transformer-networks-a-comprehensive-overview-c13c5f5bce71"><strong>https://towardsdatascience.com/transformer-networks-a-comprehensive-overview-c13c5f5bce71</strong></a>) is a tutorial that provides an overview of transformers and their applications.</li></ul><p>I hope these resources are helpful! If you have any further questions, don’t hesitate to ask.</p><p>Here are some applications that I found to be useful when using chat GPT:</p><ol><li>Chat GPT can be helpful for job seekers by providing customized cover letters based on the information from their resumes. Simply copy and paste the information from the resume into the chat GPT, and it will generate a cover letter tailored to the specific job requirements.</li><li>Chat GPT can assist in writing blogs by providing concise information on a specific subject, saving the user the time and effort of searching the internet for information. This can be particularly useful when working on a subtopic within a larger blog.</li><li>Chat GPT can be used as a study aid by allowing you to request explanations of certain topics in simpler language. This can be especially helpful if you are struggling to understand a concept or if you are learning at a more advanced level. For example, you could ask Chat GPT to explain a complex topic in simpler terms to help you better understand the material.</li></ol><p><strong>Prompt</strong></p><blockquote>“Explain me machine learning as you are explaining a 5 year old child?”</blockquote><p>Sure, here is an explanation of machine learning as if you were explaining it to a 5 year old child:</p><p>Machine learning is a way for computers to learn and make decisions without being explicitly programmed. Imagine you have a bunch of colored balls, and you want the computer to figure out which balls are red and which are not. You could try to write a list of rules for the computer to follow, like “if the ball is red, then it is a red ball” and “if the ball is not red, then it is not a red ball.” But this would be really hard and time-consuming, especially if you have a lot of balls.</p><p>Instead, you could use machine learning. You would show the computer a bunch of balls and tell it which ones are red and which ones are not. The computer would then look for patterns and try to figure out the rules on its own. After seeing enough balls, the computer should be able to accurately predict whether a new ball is red or not, even if it has never seen that ball before. This is how machine learning works.</p><p>Machine learning is used in a lot of different ways, such as helping computers understand speech, recognize faces, and make recommendations. It is a very powerful tool that can make computers smarter and more helpful.</p><p><strong>Prompt</strong></p><blockquote>“Explain me machine learning as you are explaining a graduate student?”</blockquote><p>Certainly! At a high level, machine learning is a method of teaching computers to learn from data, without explicitly programming them to perform a certain task. It involves training a model on a dataset and using that trained model to make predictions or decisions.</p><p>There are many different types of machine learning, including supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.</p><p>In supervised learning, the model is trained on labeled data, meaning that the data includes both input features and the corresponding correct output. For example, a model might be trained on a dataset of images and their associated labels (e.g., “dog”, “cat”, “car”). The goal is for the model to make predictions for new, unseen data based on the patterns it learned from the training data.</p><p>In unsupervised learning, the model is not given any labeled training data and must find patterns in the data on its own. This can be used for tasks such as clustering, in which the goal is to group similar data points together.</p><p>Semi-supervised learning is a combination of supervised and unsupervised learning, in which the model is given some labeled data and some unlabeled data. This can be useful when there is a lot of data available, but only a small portion of it is labeled.</p><p>In reinforcement learning, the model learns by interacting with its environment and receiving rewards or punishments for certain actions. This can be used to train an agent to make decisions in a game or real-world setting.</p><p>Overall, the goal of machine learning is to enable computers to learn and make decisions or predictions based on data, without being explicitly programmed for a specific task.</p><p>The output from chat GPT has a word limit for replies that is somewhere in the range of 500 words. The chat GPT will continue when we type “<strong>continue</strong>” in the chat window even though it occasionally stops mid-sentence and fails to finish what it is trying to say.</p><p>By clicking the <a href="https://openai.com/blog/chatgpt/">link</a>, you can play with the GPT chat. Create an account by selecting the <strong>TRY CHATGPT</strong> button.</p><p>Thank you for your time. I hope you find this information useful, and please do not hesitate to connect with me on <a href="https://www.linkedin.com/in/nelsonjoseph123/">Linkedin</a>.</p><p><a href="https://www.buymeacoffee.com/nelsonjoseX"><em>Buy me a coffee</em></a>☕</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9034f7f7bf85" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graph Visualization using Python]]></title>
            <link>https://medium.com/@nelsonjoseph123/graph-visualization-using-python-bbd9a593c533?source=rss-27304eff0f39------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbd9a593c533</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[networkx]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nelsonjoseph]]></dc:creator>
            <pubDate>Thu, 10 Nov 2022 22:48:21 GMT</pubDate>
            <atom:updated>2022-11-10T22:48:21.167Z</atom:updated>
            <content:encoded><![CDATA[<p>Data visualization is critical for the human mind to better understand information. They are commonly used to describe structural relationships, with nodes serving as building blocks and edges serving as reinforcements between them. This data structure’s uniqueness and possibilities allow it to be used in a wide range of fields, from molecular biology to social sciences.</p><p>We will simply look at the graph’s visualization in this blog. If you are new to this topic, it is crucial for readers to read my first blog post about graphs as a data structure. I have included the link down below.</p><p><a href="https://medium.com/@nelsonjoseph123/graph-as-a-data-structure-d04db591a0e5">Graph as a Data Structure</a></p><p>Python has several graph data visualization libraries that include <a href="https://networkx.org/"><strong><em>Networkx</em></strong></a>, <a href="https://snap.stanford.edu/snappy/"><strong><em>SNAP</em></strong></a>, <a href="https://github.com/imohitmayank/jaal"><strong><em>Jaal</em></strong></a>, <a href="https://graph-tool.skewed.de/static/doc/index.html"><strong><em>graph-tool</em></strong></a>, <a href="https://pyvis.readthedocs.io/en/latest/index.html"><strong><em>pyvis</em></strong></a>, and <a href="https://igraph.org/"><strong><em>igraph</em></strong></a> which can be used according to different scenarios. In most cases, the user interface and functionality are the same, but the customization options differ.</p><p>In this blog, we will concentrate on the most popular <strong>Networkx</strong> Python library. Networkx installation is a breeze. Simply enter the following command into the terminal.</p><pre>pip install networkx </pre><p>We can import the networkx library into Python in the following way</p><pre>import networkx as nx</pre><p>Now is the part where we dive into visualization of graphs. <strong>Networkx provides the option to create an empty graph with no nodes and no edges</strong> using the below line code.</p><pre>G = nx.Graph<strong>()</strong></pre><p>Now the graph can be expanded in a variety of ways. Let us try a few different approaches to implementing the graph below.</p><h4>Undirected graph</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pPJlYg8EaA2rlN7M9urfjw.png" /><figcaption>Undirected sample graph <strong>(</strong>Fig 1.1<strong>)</strong></figcaption></figure><p>There are two ways to add nodes to the empty graph:-</p><ol><li>Adding nodes one at a time, we must add all of the nodes one by one to create the desired graph.</li></ol><pre>G.add_node<strong>(</strong>&#39;A&#39;<strong>)</strong><br>G.add_node<strong>(</strong>&#39;B&#39;<strong>)</strong><br>G.add_node<strong>(</strong>&#39;C&#39;<strong>)</strong><br>G.add_node<strong>(</strong>&#39;D&#39;<strong>)</strong><br>G.add_node<strong>(</strong>&#39;E&#39;<strong>)</strong><br>G.add_node<strong>(</strong>&#39;F&#39;<strong>)</strong><br>G.add_node<strong>(</strong>&#39;G&#39;<strong>)</strong></pre><p>2. Adding all the nodes to the empty graph as a list.</p><pre>G.add_nodes_from<strong>(</strong>[&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;, &#39;E&#39;, &#39;F&#39;, &#39;G&#39;]<strong>)</strong></pre><p>Similarly, there are several methods for adding edges to the graph:-</p><ol><li>Adding edges one at a time, we must add all of the edges one by one to create the desired graph.</li></ol><pre>G.add_edge<strong>(</strong>&#39;A&#39;, &#39;B&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;A&#39;, &#39;G&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;A&#39;, &#39;D&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;B&#39;, &#39;G&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;B&#39;, &#39;C&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;B&#39;, &#39;E&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;C&#39;, &#39;E&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;C&#39;, &#39;F&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;D&#39;, &#39;F&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;E&#39;, &#39;F&#39;<strong>)</strong><br>G.add_edge<strong>(</strong>&#39;F&#39;, &#39;G&#39;<strong>)</strong></pre><p>2. Adding all the edges to the empty graph as a list.</p><pre>G.add_edges_from<strong>(</strong>[(&#39;A&#39;, &#39;B&#39;), (&#39;A&#39;, &#39;G&#39;), (&#39;A&#39;, &#39;D&#39;), (&#39;B&#39;, &#39;G&#39;),  (&#39;B&#39;, &#39;C&#39;), (&#39;B&#39;, &#39;E&#39;), (&#39;C&#39;, &#39;E&#39;), (&#39;C&#39;, &#39;F&#39;), (&#39;D&#39;, &#39;F&#39;), (&#39;E&#39;, &#39;F&#39;), (&#39;F&#39;, &#39;G&#39;)]<strong>)</strong></pre><p>The value of an adjacency list comes into play here. It can simplify our work by eliminating the need to add nodes and edges separately. Let us see what we can do.</p><pre>Adjacency list = <strong>{</strong>&#39;A&#39;: [&#39;B&#39;, &#39;G&#39;, &#39;D&#39;],&#39;B&#39;: [&#39;A&#39;, &#39;G&#39;, &#39;C&#39;, &#39;E&#39;],&#39;C&#39;: [&#39;B&#39;, &#39;E&#39;, &#39;F&#39;],&#39;D&#39;: [&#39;A&#39;, &#39;F&#39;],&#39;E&#39;: [&#39;B&#39;, &#39;C&#39;, &#39;F&#39;],&#39;F&#39;: [&#39;G&#39;, &#39;D&#39;, &#39;C&#39;, &#39;E&#39;],&#39;G&#39;: [&#39;A&#39;, &#39;B&#39;, &#39;F&#39;]<strong>}</strong></pre><pre>G = nx.Graph<strong>(</strong>Adjacency list<strong>)</strong></pre><p>The Networkx library supports weighted edges graphs; for more information, see the <a href="https://networkx.org/documentation/stable/auto_examples/drawing/plot_weighted_graph.html"><em>official documentation</em></a>.</p><p>We have so far created a graph with nodes and edges. We can use the following command to see if our graph has the correct edges.</p><pre>G.edges<strong>()</strong></pre><pre>#Output</pre><pre>EdgeView<strong>(</strong>[(&#39;A&#39;, &#39;B&#39;), (&#39;A&#39;, &#39;D&#39;), (&#39;A&#39;, &#39;G&#39;), (&#39;B&#39;, &#39;G&#39;), (&#39;B&#39;, &#39;C&#39;), (&#39;B&#39;, &#39;E&#39;), (&#39;C&#39;, &#39;F&#39;), (&#39;C&#39;, &#39;E&#39;), (&#39;D&#39;, &#39;F&#39;), (&#39;E&#39;, &#39;F&#39;), (&#39;F&#39;, &#39;G&#39;)]<strong>)</strong></pre><p>The visualization option is built with the networkx graph package and is quite simple to use; however, it is highly unstable and only suitable for small networks. For large networks, we must use a visualization library such as matplotlib in conjunction with networkx.</p><pre>nx.draw<strong>(</strong>G<strong>)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*psYunyROaWaGEKDIgfEZ5g.png" /><figcaption>Output of Graph G (Fig 1.2)</figcaption></figure><p>The graph does not provide us with any useful information. We can only examine the degrees (the number of edges connected to a single node) of each node.</p><p>To improve our visualization, Networkx provides several parameters within <strong>nx.draw()</strong>.</p><pre>nx.draw<strong>(</strong>G, pos = None, ax = None, with_labels = True,font_size = 20, node_size = 2000, node_color = &#39;lightgreen&#39;<strong>)</strong></pre><p><strong>nx.draw(<em>G</em>, <em>pos=None</em>, <em>ax=None</em>, <em>**kwds</em>)</strong> is the general syntax of <strong>nx.draw()</strong></p><h4>Parameters</h4><blockquote><strong>G </strong>— A Networkx graph created with <strong>G = nx.Graph ()</strong></blockquote><blockquote><strong>pos </strong>— Nodes serve as keys, and positions serve as values in a dictionary. It is a purely optional parameter.</blockquote><blockquote><strong>ax </strong>— To draw the graph in matplotlib specified axis. It is an optional parameter too.</blockquote><blockquote>All of the remaining keywords are classified as <strong>**kwds</strong>, which are optional keywords that can be used to improve the graph.</blockquote><blockquote><strong>with_labels = True</strong>, adds the labels to graph ie, the nodes labels</blockquote><blockquote><strong>font_size = 20</strong>, font size of the node labels.</blockquote><blockquote><strong>node_size = 2000</strong>, size of the nodes in the graph</blockquote><blockquote><strong>node_color = &#39;lightgreen&#39;</strong>, the color of nodes can be modified using this parameter</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*sZqUmzkKK39_QqgaqKsCSQ.png" /><figcaption>Output of Graph G <strong>(</strong>Fig 1.3<strong>)</strong></figcaption></figure><p>There is a problem that you may have noticed: this <strong>Fig 1.3</strong> differs from <strong>Fig 1.2</strong>, which does not include the parameter. This is due to the fact that when we run the script each time, the pattern of the graph changes even though it remains structurally unchanged.</p><p>To avoid the aforementioned issue, we can add a <strong>pos</strong> parameter with a <strong>seed</strong> value, so that as long as the seed remains constant, the graph will remain constant.</p><pre>pos =nx.spring_layout<strong>(</strong>G, seed = 50<strong>)</strong><br>nx.draw<strong>(</strong>G, pos, ax = None, with_labels = True,font_size = 20, node_size = 2000, node_color = &#39;lightgreen&#39;<strong>)</strong></pre><p>Check out the <a href="https://networkx.org/documentation/stable/reference/generated/networkx.drawing.layout.spring_layout.html"><em>official documentation</em></a> if you’re interested in learning more about nx.spring layout.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*pdE2zQRr2U1aMVzidSvCqg.png" /><figcaption>Output of Graph G <strong>(</strong>Fig 1.4<strong>)</strong></figcaption></figure><p>As long as we keep the seed value at 50, the graph in <strong>Fig. 1.4</strong> remains the same. If we change the seed number, the graph will appear differently.</p><h4>Directed Graph</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1r7dfJ3OOIidhvSstHAnCA.png" /><figcaption>Directed Sample Graph G <strong>(</strong>Fig 2.1<strong>)</strong></figcaption></figure><p>By using the commands listed below, we may directly plot a directed graph using an adjacency list.</p><pre>Adjacency list = <strong>{</strong>&#39;A&#39;: [&#39;B&#39;, &#39;D&#39;], &#39;B&#39;: [&#39;G&#39;, &#39;C&#39;, &#39;E&#39;], &#39;C&#39;: [&#39;F&#39;],</pre><pre>&#39;D&#39;: [&#39;A&#39;, &#39;F&#39;], &#39;E&#39;: [&#39;C&#39;, &#39;F&#39;], &#39;F&#39;: [], &#39;G&#39;: [&#39;A&#39;, &#39;F&#39;]<strong>}</strong></pre><pre>G = nx.DiGraph<strong>(</strong>Adjacency list<strong>)</strong></pre><pre>pos =nx.spring_layout<strong>(</strong>G, seed = 50<strong>)</strong></pre><pre>nx.draw<strong>(</strong>G, pos, ax = None, with_labels = True,font_size = 20, node_size = 2000, node_color = &#39;lightgreen&#39;<strong>)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*rbwmUveDry8-Idpug6wqUQ.png" /><figcaption>Output of Graph G <strong>(</strong>Fig 2.1<strong>)</strong></figcaption></figure><p>Graph, DiGraph, MultiGraph, and MultiDiGraph are some of the numerous types of graphs that Networkx supports, depending on the use cases we can utilize it. Check the <a href="https://networkx.org/documentation/networkx-2.0/reference/classes/index.html"><em>official documentaion</em></a> for more details.</p><h3>Conclusion</h3><p>Although <strong>Networkx</strong> graphs provide numerous benefits, they also have a number of drawbacks, including fixed graphs that lack interactivity. There are numerous interactive graphs for network visualization available. One such package that supports auto-layout and offers manual interactivity is <strong>PyVis</strong> (zoom, drag, select, etc). Another dash package, <strong>Visdcc</strong>, makes programmed adjustments automatically when data or property changes. Last but not least, <strong>Jaal</strong> is an interactive network visualization tool created in Python utilizing Dash and Visdcc. The network data may be played with in many different ways with Jaal, which is more like a dashboard than a network plot. Options include searching, filtering, and even coloring nodes and edges in the graph.</p><p>We’ll look at how to customize networkx graphs in forthcoming stories.</p><p>Thank you for your time. I hope you find this information useful, and please do not hesitate to connect with me on <a href="https://www.linkedin.com/in/nelsonjoseph123/">Linkedin</a>.</p><p><a href="https://www.buymeacoffee.com/nelsonjoseX"><em>Buy me a coffee</em></a>☕</p><p><strong>References</strong></p><p><a href="https://networkx.org/"><em>Networkx</em></a>, <a href="https://www.toptal.com/data-science/graph-data-science-python-networkx#:~:text=Python%20graph%20data%20visualization%20libraries,nodes%20and%20edges%20of%20networks."><em>Graph Data Science With Python/NetworkX</em></a>, <a href="https://algorithms.discrete.ma.tum.de/"><em>Visualizations of Graph Algorithms</em></a>, <a href="https://towardsdatascience.com/visualizing-networks-in-python-d70f4cbeb259"><em>Visualizing Networks in Python</em></a>, <a href="https://towardsdatascience.com/introducing-jaal-interacting-with-network-made-easy-124173bb4fa"><em>Introducing Jaal — Interacting with Network Made Easy</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbd9a593c533" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graph as a Data Structure]]></title>
            <link>https://medium.com/@nelsonjoseph123/graph-as-a-data-structure-d04db591a0e5?source=rss-27304eff0f39------2</link>
            <guid isPermaLink="false">https://medium.com/p/d04db591a0e5</guid>
            <category><![CDATA[data-structures]]></category>
            <category><![CDATA[graph-theory]]></category>
            <category><![CDATA[data-structures-course]]></category>
            <dc:creator><![CDATA[Nelsonjoseph]]></dc:creator>
            <pubDate>Sun, 06 Nov 2022 19:49:31 GMT</pubDate>
            <atom:updated>2022-11-06T19:49:31.057Z</atom:updated>
            <content:encoded><![CDATA[<p>Graphs are used to represent elements that share connections between them. It is considered as a non-linear data structure that has Node/Vertices <strong>(V)</strong> as the elements and the edges <strong>(E)</strong> as the connection link between the elements. A graph <strong>G</strong> with a set of vertices <strong>(V)</strong> with a set of edges <strong>(E)</strong> is represented as <strong>G (V, E)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABwaFOSIVFlGbJ4Kcaeu8g.jpeg" /><figcaption>An Undirected Graph G with 7 nodes</figcaption></figure><p><strong>A, B, C, D, E, F,</strong> and <strong>G</strong> are the vertices in the graph G.</p><p><strong>A-B</strong>, <strong>A-G</strong>, <strong>A-D</strong>, <strong>B-A</strong>, <strong>B-G</strong>, <strong>B-C</strong>, <strong>B-E</strong>, <strong>C-B</strong>, <strong>C-E</strong>, <strong>C-F</strong>, <strong>D-A</strong>, <strong>D-F</strong>, <strong>E-B</strong>, <strong>E-C</strong>, <strong>E-F</strong>, <strong>F-G</strong>, <strong>F-D, F-C</strong>, <strong>F-E</strong>, <strong>G-A</strong>, <strong>G-B, </strong>and<strong> G-F</strong> are the edges of Graph <strong>G</strong>. Since they are undirected edges, the <strong>A-D</strong> edge is the same as the <strong>D-A</strong> edge and similarly for others.</p><p>Graphs can be divided into <strong>Undirected</strong> and <strong>Directed</strong> graphs. The Facebook friends group is a real-life example of an undirected graph. When a person adds a new friend on Facebook, both of them have the same access to each other’s profile. The undirected can also mean bi-directional so that all nodes can interact both ways, so the direction does not come into consideration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X0M8GWMRUZT8yjyJDzPbZQ.png" /><figcaption>A Directed Graph G with 7 nodes</figcaption></figure><p><strong>A, B, C, D, E, F,</strong> and <strong>G</strong> are the vertices in the graph G.</p><p><strong>A-B</strong>, <strong>A-D</strong>, <strong>B-G</strong>, <strong>B-C</strong>, <strong>B-E</strong>, <strong>C-F</strong>, <strong>D-A</strong>, <strong>D-F</strong>, <strong>E-C</strong>, <strong>E-F</strong>, <strong>G-A</strong>, and <strong>G-F</strong> are the edges of Graph <strong>G</strong>. Here, the direction of edges is important, so the <strong>A-D</strong> edge is not the same as the <strong>D-A</strong> edge.</p><p>In a directed graph, the edges are not necessarily mutual. Suppose the edges are roads between 2 cities. Here, a person in <strong>City A</strong> can travel directly to <strong>City B</strong> but, a person in <strong>City B</strong> cannot travel to <strong>City A</strong> directly. The person has to take a different route, ie, <strong>B-G-A</strong>. In this diagram, a person in <strong>City A</strong> and <strong>City D</strong> can travel back and forth.</p><p>Another real-life example of a directed graph is the Instagram followers network. Consider directed edges representing a person following another from the above diagram. <strong>Person C</strong> is being followed by <strong>B</strong> and <strong>E</strong>, but <strong>C</strong> only follows <strong>F</strong>. If both are following each other then both of them will be having full access to each other profiles. Influencers on Instagram will be having many directed edges toward them. Here, <strong>Person F</strong> is such a case.</p><p>Both graphs can be with <strong>weights</strong>, and they can be anything that we assume. If we are taking the nodes as different cities and the edges as the roads connecting them, then the weight could be the distance between two cities, or the gasoline required for a car to travel the distance, and even the time of travel. The weight changes according to the situation in which the graph is constructed.</p><p>The main question that arises for someone who is new to graphs is how we represent these graphs. The most commonly used representations for graphs are :-</p><ol><li>Adjacency Lists</li><li>Adjacency Matrix</li></ol><p><strong>Adjacency Lists</strong></p><p>It is a collection of unordered lists, each describing the set of neighbors of a particular node in the graph. Let us check the above graphs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1tmF9T7CubXwlYChlecnww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*853v_UrxnXzJW0KQPoVHXA.png" /></figure><p><strong>Adjacency Matrix</strong></p><p>It is a 2D matrix of shape <strong>(VXV)</strong> with 1 and 0 indicating a connected and disconnected edge between two vertices respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1tmF9T7CubXwlYChlecnww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RrTBs4ZTY7CckdL8_XnDLg.png" /></figure><p>In the upcoming blogs, we will see how to visualize graphs using Python.</p><p>Thanks for reading. Hope you find this information useful and feel free to connect with me on <a href="https://www.linkedin.com/in/nelsonjoseph123/">Linkedin</a>.</p><p><a href="https://www.buymeacoffee.com/nelsonjoseX"><em>Buy me a coffee</em></a>☕</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d04db591a0e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Tree in Machine Learning]]></title>
            <link>https://medium.com/@nelsonjoseph123/decision-tree-in-machine-learning-a21b4b2ee316?source=rss-27304eff0f39------2</link>
            <guid isPermaLink="false">https://medium.com/p/a21b4b2ee316</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nelsonjoseph]]></dc:creator>
            <pubDate>Fri, 28 Oct 2022 22:24:24 GMT</pubDate>
            <atom:updated>2022-10-29T04:49:41.490Z</atom:updated>
            <content:encoded><![CDATA[<p>A decision tree algorithm is a supervised Machine learning algorithm, it satisfies the conditions of a supervised algorithm and takes training sets with correct answers to train in order to get desired outputs from it. It is considered non-parametric as it requires any predefined set of parameters before training. The decision tree is a discriminative model like logistic regression, SVM, and Neural networks as it draws boundaries in the data space.</p><p>Now let us go into more depth about how a decision tree works and how this name originated. How is the decision tree different from a real tree?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*M-cvD129lLae77DPsFmb-A.jpeg" /><figcaption>An Inverted Real Tree similar to a Decision Tree</figcaption></figure><p>The decision tree is similar to a real tree, but it starts from its root and grows to branches and leaves. Each branch in a decision tree represents the outcome of a test. The leaves in the decision tree represent the decision made, ie, the output of the class.</p><p>Decision trees are generally used for handling non-linear data sets effectively where the data cannot be represented on a linear relationship between the input variable (X) and the output variable (Y). The simple way of structuring makes it popular. This concept has been in use long before machine learning came into action for solving problems, as in a flowchart.</p><h4>Structure</h4><p>The learning starts by first deciding the root node, ie, which feature among the features should be placed on the top as the root. The Root node represents the entire sample of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6xAoBTdmTSGkAS9_QcuRjQ.jpeg" /><figcaption>The decision tree diagram of an ideal case</figcaption></figure><p>The decision nodes are the features that are taken into account for the further split of the tree. The process of splitting a single node into multiple nodes is known as splitting. Splitting is done recursively in a decision tree by splitting the data into sub-populations based on several dichotomous independent variables. <strong>The main question here is what decides the splitting and what determines the root node.</strong></p><p>In order to answer the above question, one needs to understand the different types of classification techniques. <strong>CART</strong> (Classification and Regression Trees), <strong>ID3</strong> (Iterative Dichotomiser 3), and <strong>C4.5</strong> are the most commonly used training algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F5E1_NO1UFFLU61wfBMkuw.jpeg" /><figcaption>Comparison Chart of Training Algorithms</figcaption></figure><p>From the above chart, we can get an idea about different algorithms used for making decision trees. The cost function for the splitting is different in each case. <strong>The data attribute with the minimum Gini Index, maximum Information Gain, and maximum Gain ratio is used by CART, ID3, and C4.5 respectively for the Root node and for the further split.</strong></p><p>In the upcoming blogs, we can see how to calculate the Gini Index, Information Gain, and Gain Ratio for splitting the data.</p><p>That’s all I collected for you and thanks for reading! Hope you find this information useful. Feel free to connect with me on <a href="https://www.linkedin.com/in/nelsonjoseph123/">Linkedin</a>.</p><p><a href="https://www.buymeacoffee.com/nelsonjoseX"><em>Buy me a coffee</em></a>☕</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a21b4b2ee316" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>